A Longitudinal Randomized Trial of a Sustained Content Literacy Intervention From First to Second Grade: Transfer Effects on Students' Reading Comprehension

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomization was conducted at the school level (30 schools), which is stronger than the class-level requirement, so the class-level RCT criterion is met.
    • "A total of 30 elementary schools (N = 2,952 students; N = 144 teachers) were randomly assigned to a treatment or control group." (p. 2)
    • Relevant Quotes: 1) "We conducted a longitudinal cluster (school) randomized controlled trial (RCT) to test the effectiveness of sustaining the MORE content literacy intervention from Grade 1 to Grade 2 on students' reading comprehension outcomes." (p. 19) 2) "A total of 30 elementary schools (N = 2,952 students; N = 144 teachers) were randomly assigned to a treatment or control group." (p. 2, Abstract) 3) "In particular, schools were blocked by three levels of school size (i.e., small, medium, large schools based on total student enrollment), two levels of prior reading proficiency (i.e., above and below the median on end-of-grade 3 reading performance), and prior experience with MORE and then randomized to treatment or control conditions for the 12-month study period." (p. 19) Detailed Analysis: Quotes checked verbatim against the PDF and confirmed exact. The unit of randomization is clearly described as the school: 30 elementary schools in one urban district were blocked on school size, prior reading proficiency, and prior MORE experience, and then randomly assigned to treatment or control conditions. The ERCT standard states that if a study was done as a school-level RCT, the weaker class-level criterion is automatically considered met. School-level assignment prevents contamination between treatment and control students within the same classroom, satisfying the intent of criterion C. Criterion C is met because randomization was performed at the school level, which exceeds the class-level requirement.
  • E

    Exam-based Assessment

    • The study used the MAP reading assessment (NWEA), a widely recognized standardized computer-adaptive test, as an outcome measure for domain-general reading comprehension.
    • "The MAP reading assessment (Northwest Evaluation Association, 2011) was used to assess students' domain-general reading comprehension and vocabulary." (p. 31)
    • Relevant Quotes: 1) "The MAP reading assessment (Northwest Evaluation Association, 2011) was used to assess students' domain-general reading comprehension and vocabulary. The MAP is a vertically scaled and computer-adaptive reading assessment that measures student growth in reading comprehension from kindergarten to second grade using the Rasch unit (RIT) scale (Northwest Evaluation Association, 2011)." (p. 31) 2) "The test-retest reliabilities of an overall RIT total reading and subtests of reading comprehension and vocabulary ranged from .89 to .96 (Brown & Coughlin, 2007). The MAP reading assessments were administered at three-time points: winter of Grade 1 (January), spring of Grade 1 (April), and fall of Grade 2 (September)." (p. 31) 3) "We created a transfer measure of science content reading comprehension to assess Grade 2 students' ability to comprehend texts about schema-related topics." (p. 31) 4) "We used the semantic association task (Read, 1998; 2004) to assess students' vocabulary knowledge depth of taught words in science thematic units..." (p. 33) Detailed Analysis: Quotes checked verbatim and confirmed exact. The study used a mix of assessments. The primary outcome for research question 1 (domain-general reading comprehension and vocabulary growth) was the MAP (Measure of Academic Progress) from NWEA, a nationally normed, vertically scaled, computer-adaptive standardized assessment with documented reliability (.89-.96), administered at three time points as an outcome measure, not merely as a baseline covariate. The Grade 2 science content reading comprehension measure and the vocabulary knowledge depth measure were researcher-created instruments and would not on their own satisfy criterion E. However, because a widely recognized standardized exam (MAP) was used to measure a key educational outcome of the intervention, the criterion of using a standardized exam-based assessment is satisfied. Criterion E is met because the study used the MAP, a widely recognized standardized reading assessment, as an outcome measure, even though supplementary custom measures were also used.
  • T

    Term Duration

    • The intervention began in February 2019 and outcomes were measured through fall of Grade 2 (September 2019) and the end of the Grade 2 science unit, an interval far exceeding one academic term.
    • "Teachers were given substantial flexibility in their pacing calendar to teach the 20 lessons over a 5- to 10-week window from February to March 2019." (p. 22)
    • Relevant Quotes: 1) "Teachers were given substantial flexibility in their pacing calendar to teach the 20 lessons over a 5- to 10-week window from February to March 2019." (p. 22) 2) "The MAP reading assessments were administered at three-time points: winter of Grade 1 (January), spring of Grade 1 (April), and fall of Grade 2 (September)." (p. 31) 3) "A total of 30 elementary schools in one urban school district located in the southeastern United States participated in the 12-month program implementation." (p. 19) 4) "The unit was designed to be taught over a period of 10 weeks, starting in late October and ending in late January, and teachers were given a pacing calendar to support their planning." (p. 23) Detailed Analysis: Quotes checked verbatim and confirmed exact. The intervention started with Grade 1 thematic lessons in February 2019. Outcomes were measured at multiple later points: MAP in April 2019 (about 2-3 months after start), MAP in September 2019 (about 7 months after start), and the Grade 2 science content reading comprehension assessment after the Grade 2 unit ended in late January (about 12 months after the intervention began). The interval from intervention start (February 2019) to the fall Grade 2 MAP measurement (September 2019) alone is roughly seven months, which exceeds one academic term (3-4 months), and the final domain-specific outcome was measured about a year after the start. Because the stronger year-duration criterion is also met (see Y), this weaker term-duration criterion is considered met as well. Criterion T is met because the interval from intervention start to outcome measurement substantially exceeds one full academic term.
  • D

    Documented Control Group

    • The control group is documented in detail, including sample sizes, demographics, baseline test scores, the business-as- usual curriculum it received, and its instructional time.
    • "Table 2 displays descriptive characteristics of participating teachers (N = 144) and students (N = 2,275) at the beginning of the study in Grade 1." (p. 20)
    • Relevant Quotes: 1) "Table 2 displays descriptive characteristics of participating teachers (N = 144) and students (N = 2,275) at the beginning of the study in Grade 1." (p. 20) 2) "Table 2 ... Characteristics Overall Treatment Control ... Students ... N 2,275 1,260 1,015 ... Baseline MAP reading, M (SD) 169.27 (15.87) 167.35 (15.70) 171.64 (15.76)" (p. 73, Table 2) 3) "Overall, the analytic sample of students who completed the final science content comprehension outcome included 72% of the treatment group (n = 1,176) and 74% of the control group (n = 980)." (p. 20) 4) "In Grade 1, the Tier I English language arts (ELA) curriculum was based on a balanced literacy program involving word study, guided reading and writing activities in small groups, teacher-directed lessons, and independent reading. In Grade 2, the district adopted a new Tier I ELA curriculum, Expeditionary Learning (EL)..." (p. 21) 5) "For both analytic samples, the baseline difference between treatment and control conditions on pretest scores was below the threshold of 0.25 SDs established by What Works Clearinghouse (WWC) Standards (2021) and all statistical models included pretest scores to adjust for these baseline differences." (p. 21) Detailed Analysis: Quotes checked verbatim and confirmed exact. The paper documents the control group thoroughly. Table 2 reports control-group sample sizes (1,015 students, 66 teachers) and demographics (gender, race/ethnicity, English proficiency, SES, IEP status) alongside baseline MAP reading and mathematics scores by condition. Baseline equivalence was formally tested against WWC standards. The conditions experienced by control classrooms are described (the district Tier I ELA curriculum, control read-aloud book Lexile levels, and control teachers' weekly instructional time in ELA, science, and social studies in Table 4), and the consort flow diagram documents attrition by condition. Criterion D is met because the control group's composition, baseline performance, and conditions are documented in detail.

• Level 2 Criteria

  • S

    School-level RCT

    • Thirty elementary schools were the unit of randomization in a blocked cluster design, satisfying the school-level RCT criterion.
    • "Prior to program implementation, we pre-registered the study design, conducted a power analysis to determine the minimum detectable effect size, and implemented a blocked, cluster (school-level) randomized controlled trial (Kim et al., 2021b)." (p. 19)
    • Relevant Quotes: 1) "Prior to program implementation, we pre-registered the study design, conducted a power analysis to determine the minimum detectable effect size, and implemented a blocked, cluster (school-level) randomized controlled trial (Kim et al., 2021b)." (p. 19) 2) "A total of 30 elementary schools (N = 2,952 students; N = 144 teachers) were randomly assigned to a treatment or control group." (p. 2, Abstract) 3) "In particular, schools were blocked by three levels of school size ... two levels of prior reading proficiency ... and prior experience with MORE and then randomized to treatment or control conditions for the 12-month study period." (p. 19) Detailed Analysis: Quotes checked verbatim and confirmed exact. The design is explicitly described as a blocked, cluster (school-level) randomized controlled trial in which 30 elementary schools were the randomized units. The number of schools, the blocking variables, and the assignment procedure are all described. This directly satisfies the requirement that randomization occur among schools rather than among classes or students within schools. Criterion S is met because entire schools (30 elementary schools) were randomly assigned to treatment or control conditions.
  • I

    Independent Conduct

    • The same Harvard-led research team that developed the MORE intervention also implemented the trial and analyzed the outcomes, with no independent third-party evaluator reported.
    • "We developed a sustained content literacy intervention that emphasized building domain and topic knowledge from Grade 1 to Grade 2 and evaluated transfer effects on students' reading comprehension outcomes." (p. 2)
    • Relevant Quotes: 1) "We developed a sustained content literacy intervention that emphasized building domain and topic knowledge from Grade 1 to Grade 2 and evaluated transfer effects on students' reading comprehension outcomes." (p. 2, Abstract) 2) "In developing the Model of Reading Engagement (MORE) intervention, we identified research-based practices to build and deepen students' domain and topic knowledge schemas..." (p. 12) 3) "The research team hosted monthly meetings for literacy facilitators to communicate information about the MORE lesson components and to address teachers' questions." (p. 22) 4) "During the implementation period, the research team provided treatment teachers with ongoing support through daily video clips (5-10 minutes) that were anchored to that day's lesson components." (pp. 24-25) 5) "We created a transfer measure of science content reading comprehension to assess Grade 2 students' ability to comprehend texts about schema-related topics." (p. 31) Detailed Analysis: Quotes checked verbatim and confirmed exact. The MORE intervention was designed by the author team (led by James S. Kim at Harvard), and the same team delivered teacher professional development, supported implementation, developed the primary Grade 2 outcome instruments, and conducted all analyses. There is no statement anywhere in the paper that an external or third-party evaluation team conducted data collection or analysis, and no disclosure of independent oversight. Under criterion I, when the intervention designers also conduct and analyze the evaluation without documented third-party involvement, the criterion fails. Criterion I is not met because the intervention developers themselves conducted the trial and analysis without documented independent evaluation.
  • Y

    Year Duration

    • The study was an explicitly 12-month intervention (February 2019 through the Grade 2 unit ending late January 2020) with final outcomes measured about a year after the start, exceeding 75% of an academic year of tracking.
    • "A total of 30 elementary schools in one urban school district located in the southeastern United States participated in the 12-month program implementation." (p. 19)
    • Relevant Quotes: 1) "A total of 30 elementary schools in one urban school district located in the southeastern United States participated in the 12-month program implementation." (p. 19) 2) "Over 12 months, the treatment group students participated in (a) spring Grade 1 thematic content literacy lessons in science and social studies followed by wide reading of thematically related informational texts during summer, and (b) fall to spring Grade 2 thematic content literacy lessons in science." (p. 2, Abstract) 3) "Teachers were given substantial flexibility in their pacing calendar to teach the 20 lessons over a 5- to 10-week window from February to March 2019." (p. 22) 4) "The unit was designed to be taught over a period of 10 weeks, starting in late October and ending in late January, and teachers were given a pacing calendar to support their planning." (p. 23) 5) "At the second follow-up, 27% of students were lost to attrition and included in the second follow-up (N = 2,156) analyses of domain-specific reading comprehension at the end of grade 2." (p. 20) Detailed Analysis: Quotes checked verbatim and confirmed exact. The intervention began with Grade 1 lessons in February 2019 and continued through summer reading and the Grade 2 science unit that ran from late October to late January (2020), with the domain-specific science content reading comprehension outcome administered after that unit. The interval from intervention start (February 2019) to the final outcome measurement (roughly January-February 2020) is approximately 12 months, spanning a full academic year of tracking. Although COVID-19 school closures cancelled additional planned spring 2020 and Grade 3 assessments, the analyzed outcomes still cover the full 12-month intervention period, which clearly exceeds the 75%-of-an-academic-year threshold. Because Y is met, the weaker T criterion is also met. Criterion Y is met because students were tracked from the February 2019 intervention start through outcome measurement about 12 months later.
  • B

    Balanced Control Group

    • The treatment package (thematic lessons, teacher PD, and free summer books) was itself the treatment variable tested against business-as-usual instruction, and in Grade 2 instructional time was empirically equivalent between conditions.
    • "Because there was no difference in the amount of time teachers spent on content literacy instruction, we can also rule out additional time as a factor that explains the impact on domain-specific reading comprehension." (p. 45)
    • Relevant Quotes: 1) "Overall, treatment group teachers (M = 177.34, SD = 130.46) reported spending approximately 68 more minutes per week on science instruction than control teachers (M = 109.05, SD = 99.37). Likewise, treatment group teachers (M = 181.82, SD = 133.05) devoted nearly 72 more minutes per week to social studies instruction relative to control teachers (M = 108.90, SD = 100.60). However, the amount of time on ELA/reading instruction was similar for treatment and control groups." (p. 28) 2) "As shown in Table 4, treatment and control teachers devoted an average of 678.04 and 673.74 minutes per week, respectively, on ELA/reading instruction, and 104.53 and 108.04 minutes per week, respectively, on science instruction, though neither difference was statistically significant (p > .05). This result suggests that the treatment teachers, though asked to take on 45 additional lessons, did not do so at the expense of instructional time dedicated to the district's ELA curriculum." (pp. 29-30) 3) "Treatment group students received their print books during the last week of school during first grade. As an incentive for participation in the study, both treatment and control group parents were given access to an app where their children could choose six digital books to read on contemporary and historical figures who overcame obstacles." (p. 23) 4) "To implement the MORE Grade 1 lessons, teachers participated in a two-hour in-person professional development meeting and received ongoing support during the implemented MORE content literacy lessons..." (p. 22) 5) "The MORE Grade 1 lessons and procedures for promoting wide reading of thematically related texts and the MORE Grade 2 lessons were implemented during the supplemental science and social studies content block. In Grade 1 and Grade 2, there was no district-wide science and social studies curriculum and schools had considerable flexibility in the scope and sequence of content instruction." (p. 22) 6) "Because there was no difference in the amount of time teachers spent on content literacy instruction, we can also rule out additional time as a factor that explains the impact on domain-specific reading comprehension." (p. 45) Detailed Analysis: Quotes checked verbatim and confirmed exact. Re-applying the updated criterion B decision tree with particular care, as instructed. Step 1 (EXTRA_RESOURCES_PRESENT): Yes. The treatment condition received teacher PD (two hours in Grade 1, three hours in Grade 2, plus daily video clips and facilitator meetings), MORE lesson materials, ten free print books per treatment student for summer reading, and roughly 68-72 more minutes per week of Grade 1 science/social studies instruction. Step 1a (NEGLIGIBLE_DIFF): No, the Grade 1 time gap (~140 min/week combined) and the treatment-only print books are not negligible. Step 2 (RESOURCES_ARE_TREATMENT, i.e., are the extra resources integral to and inseparable from delivering the tested intervention rather than a separable add-on): the MORE intervention is explicitly defined as the package of thematic content-literacy lessons, teacher training to deliver them, and aligned wide-reading materials; this package cannot be delivered without spending instructional time on the lessons themselves or providing the associated books. The lessons were placed in a supplemental content block where "there was no district-wide science and social studies curriculum" (p. 22), so control classrooms had no equivalent structured use of that time by default. This closely parallels the ERCT standard's own worked exception (education-technology tool requiring ~40 extra minutes/week of usage that is integral to the intervention, met against a business-as-usual control) rather than a separable, optional add-on to an otherwise-identical lesson. Step 3 (CONTROL_MATCHES_RESOURCES): No for Grade 1 (control got neither the extra content time nor the free books), but for Grade 2 -- the basis for the near/mid/far-transfer science comprehension effects that are the paper's primary domain-specific findings -- treatment and control teachers' ELA and science instructional time were empirically equivalent (678.04 vs. 673.74 min/week ELA; 104.53 vs. 108.04 min/week science, both ns), and the authors explicitly rule out extra time as an explanation for the Grade 2 effect. Not a within-subjects design (Step 4 does not apply). Following the "integral, not separable" branch of the decision tree for the Grade 1 package elements and the "control matches resources" branch for Grade 2, criterion B is satisfied, though the Grade 1 imbalance in content-area instructional time and treatment-only print books is a genuine, non-trivial confound for the domain-general reading outcome (RQ1) and should be flagged as the main caveat on this criterion. Criterion B is met because the additional Grade 1 lessons, books, and training are integral, inseparable components of the MORE treatment package explicitly tested against business-as-usual instruction, and the Grade 2 instructional time underlying the primary science-transfer findings was empirically balanced; the Grade 1 time/book imbalance remains a noteworthy limitation on the domain-general reading finding.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this study by a different research team is reported or found; prior and subsequent MORE studies located online were conducted by the same author team.
    • "This study builds on previous research evaluating the design, implementation, and effectiveness of the core components of the MORE intervention. Previous research highlights the unique contributions of each component of MORE (e.g., Kim et al., 2021a; 2021b)." (p. 17)
    • Relevant Quotes: 1) "This study builds on previous research evaluating the design, implementation, and effectiveness of the core components of the MORE intervention. Previous research highlights the unique contributions of each component of MORE (e.g., Kim et al., 2021a; 2021b). In early efficacy studies involving Grade 1 science thematic lessons, MORE improved reading, vocabulary, and writing outcomes relative to control students in Tier I literacy instruction." (p. 17) 2) "Keeping these limitations in mind, we encourage future replications to determine whether the full implementation of Grade 1 and Grade 2 science and social studies lessons promotes gains in reading comprehension (both domain-general reading and domain-specific reading in social studies) across a 12-month time period." (p. 48) Detailed Analysis: Quotes checked verbatim and confirmed exact. The only prior evaluations of the MORE intervention cited in the paper (Kim et al., 2021a, 2021b) were conducted by the same research team led by James S. Kim, so they do not constitute independent replication. The authors themselves explicitly call for future replications, indicating none of this sustained Grade 1 to Grade 2 design existed at time of writing. An internet search (Semantic Scholar citation graph for DOI 10.1037/edu0000751, 36 citing works reviewed as of this check) found no independent replication of this specific 12-month sustained MORE trial by a different research team; the vast majority of citing works are conceptual or methodological papers (e.g., item-response modeling papers by Gilbert et al., a same-research-group multilingual tutoring study by Relyea et al., and general content-literacy commentary), not replications, and none report an independent re-implementation of this study's design in a different context by a different team, published in a peer-reviewed outlet. Criterion R is not met because no independent replication of this study by a different research team is documented in the paper or found via internet search.
  • A

    All-subject Exams

    • Standardized outcome testing covered only reading (MAP reading); mathematics and other core subjects were not assessed as outcomes, and the science measure was a custom researcher-made test.
    • "The MAP reading assessments were administered at three-time points: winter of Grade 1 (January), spring of Grade 1 (April), and fall of Grade 2 (September)." (p. 31)
    • Relevant Quotes: 1) "The MAP reading assessment (Northwest Evaluation Association, 2011) was used to assess students' domain-general reading comprehension and vocabulary." (p. 31) 2) "We created a transfer measure of science content reading comprehension to assess Grade 2 students' ability to comprehend texts about schema-related topics." (p. 31) 3) "we fit multi-level models in which the pretest MAP total reading and MAP total mathematics scores were predicted by the treatment indicator, randomization block, and random effects for schools." (p. 21) Detailed Analysis: Quotes checked verbatim and confirmed exact. Criterion A requires that impact be measured on all main subjects using standardized exam-based assessments. Here, the only standardized outcome measure was the MAP reading assessment (comprehension and vocabulary subtests). MAP mathematics scores were used solely as a baseline covariate, not as an outcome, so no impact on mathematics was measured. The Grade 2 science outcome was a researcher-created transfer measure rather than a standardized exam, and no other core subjects (mathematics, science via standardized tests, etc.) were assessed as outcomes. Elementary students' main subjects include at least reading and mathematics, and the study provides no justification qualifying for the specialized intervention exception. Criterion A is not met because outcomes were measured with a standardized exam only in reading, not across all main subjects.
  • G

    Graduation Tracking

    • Tracking ended in Grade 2 after COVID-19 school closures cancelled planned Grade 3 assessments; a possible same-author Grade 3 follow-up was located online but does not reach graduation and could not be verified with quotes.
    • "The planned assessments in Grade 3 MORE were not implemented due to school closures in spring 2020 (see Supplementary Materials S-A)." (p. 19)
    • Relevant Quotes: 1) "The planned assessments in Grade 3 MORE were not implemented due to school closures in spring 2020 (see Supplementary Materials S-A)." (p. 19) 2) "Thus, we were able to examine the intervention treatment effects for students who started the study in the spring of Grade 1 and continued through the end of the study in the spring of Grade 2." (p. 19) 3) "Given the impact of COVID-19 on our original research plan, we were unable to continue the longitudinal growth model analysis with the domain-general reading comprehension measure or to assess social studies reading comprehension in the spring of Grade 2." (p. 47) Detailed Analysis: Quotes checked verbatim and confirmed exact. Criterion G requires that participants be tracked until graduation from their educational stage (here, the end of elementary school, typically Grade 5 or 6 in the US). The study followed students only from the spring of Grade 1 through Grade 2, and even the planned Grade 3 follow-up assessments were cancelled due to COVID-19 school closures. An internet search for follow-up publications by the same author team found a citation record (Semantic Scholar, citing DOI 10.1037/edu0000751) for a paper titled "Long-Term Effects of a Sustained Content Literacy Intervention on Third Graders' Reading Comprehension Outcomes" by James S. Kim et al. This candidate paper could plausibly extend tracking of the same cohort to Grade 3. However, repeated attempts to retrieve its venue, publication year, DOI, or full text (via Semantic Scholar's paper API, OpenAlex, ERIC, and Harvard-affiliated sources) were unsuccessful, so no verbatim quotes could be obtained and its existence/content could not be independently confirmed; per instructions, no quote is fabricated for it. Regardless, even if this Grade 3 follow-up were confirmed, tracking through Grade 3 would still fall well short of graduation from elementary school (Grade 5 or 6), so it would not by itself satisfy criterion G. Criterion G is not met because measurement stopped in Grade 2, long before students graduated from elementary school, and no confirmed graduation-tracking follow-up publication was found.
  • P

    Pre-Registered

    • The study design was pre-registered in the AEA RCT Registry (trial AEARCTR-0003489) on October 24, 2018, confirmed via the registry itself to predate the Grade 1 intervention that began in February 2019.
    • "Prior to program implementation, we pre-registered the study design, conducted a power analysis to determine the minimum detectable effect size, and implemented a blocked, cluster (school-level) randomized controlled trial (Kim et al., 2021b)." (p. 19)
    • Relevant Quotes: 1) "Prior to program implementation, we pre-registered the study design, conducted a power analysis to determine the minimum detectable effect size, and implemented a blocked, cluster (school-level) randomized controlled trial (Kim et al., 2021b)." (p. 19) 2) "Details on the original study are in our preregistration plan (Kim, 2021) and the online supplementary materials." (p. 18) 3) "Kim, J. S. (2021). 'A Randomized Controlled Trial to Replicate and Personalize with Technology the Effects of a Model of Reading Engagement (MORE) on First- and Second-Graders' Science and Social Studies Domain Knowledge, Reading Engagement, Reading Comprehension, and Basic Literacy Skills.' AEA RCT Registry. February 06. https://doi.org/10.1257/rct.3489-1.2000000000000002" (References) Detailed Analysis: Quotes checked verbatim and confirmed exact. The reference list cites the registry entry with a "February 06" date, which could be misread as a 2021 registration date occurring after the Grade 1 intervention (Feb-Mar 2019). This was verified directly against the AEA Social Science Registry (trial AEARCTR-0003489, resolved via the paper's DOI 10.1257/rct.3489-1.2000000000000002): the trial's initial registration date is October 24, 2018, with a listed study start date of October 25, 2018; "February 06, 2021" is only the date of the most recent update/version to an already-existing registry entry, not the original registration date. This confirms that pre-registration genuinely preceded both the Grade 1 intervention (February 2019) and all data collection, satisfying the timing requirement of criterion P. Criterion P is met because the study was pre-registered in the AEA RCT Registry on October 24, 2018, confirmed via the registry to predate program implementation and data collection.