Time to Transfer: Long-Term Effects of a Sustained and Spiraled Content Literacy Intervention in the Elementary Grades

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomization occurred at the school level (30 schools), which is stronger than and therefore satisfies the class-level RCT requirement.
    • "A total of 30 elementary schools (N = 2,870 students) were randomized to a treatment or control condition."
    • Relevant Quotes: 1) "A total of 30 elementary schools (N = 2,870 students) were randomized to a treatment or control condition." (p. 1279, Abstract) 2) "For this longitudinal cluster (school) RCT study, 30 elementary schools in one urban school district located in the southeastern United States were recruited." (p. 1284) 3) "Schools were blocked by three levels of school size (i.e., small, medium, and large schools based on total student enrollment), two levels of prior reading proficiency ... and prior experience with MORE and then randomized to a treatment (full MORE spiral curriculum from Grades 1 to 3) or a control condition." (pp. 1284-1285) Detailed Analysis: The paper explicitly and repeatedly states that the unit of randomization was the entire school, not individual classes or students. Thirty elementary schools were recruited, blocked on size, prior reading proficiency, and prior MORE experience, and then randomized as whole units to treatment or control. School-level randomization is the stronger design required for Level 2 (criterion S), and per the ERCT standard, meeting the stronger school-level criterion automatically satisfies the weaker class-level criterion. There is no indication of within-school or within-class randomization anywhere in the methods. Criterion C is met because randomization occurred at the school level, which exceeds the class-level requirement.
  • E

    Exam-based Assessment

    • The study used state-wide End-of-Grade standardized assessments for its primary domain-general reading and mathematics outcomes.
    • "We measured students' domain-general reading comprehension and mathematics ability with posttests in Grades 3 and 4, using state-wide end-of-grade (EOG) standardized assessments, during the last week of the school year."
    • Relevant Quotes: 1) "We measured students' domain-general reading comprehension and mathematics ability with posttests in Grades 3 and 4, using state-wide end-of-grade (EOG) standardized assessments, during the last week of the school year." (p. 1289) 2) "For each of the domains, we used the item response theory-scaled EOG test score to estimate the overall ability in respective domains. Internal consistencies are approximately .90 for Grades 3 and 4 across demographic subgroups (North Carolina Department of Public Instruction, 2020)." (p. 1289) 3) "We used the MAP (NWEA, 2019) reading and mathematics assessment scores (test-retest reliabilities = .79-.86) obtained in Grade 1 to control for students' baseline reading and mathematics abilities. The MAP assessment is a vertically scaled and computer-adaptive assessment..." (p. 1289) 4) By contrast, the domain-specific outcomes were researcher-developed: "We used a semantic association task ... to assess students' ability to identify semantically related words..." (p. 1288) and "We developed near-, mid-, and far-transfer domain-specific (science) reading comprehension passages and 29 multiple-choice questions..." (p. 1289) Detailed Analysis: The study's key domain-general reading and mathematics outcomes, which anchor the headline effects reported in the abstract (ES = .11-.16), were measured using statewide End-of-Grade (EOG) assessments, which are standard, government-administered, widely recognized standardized exams, not instruments created by the researchers. The Grade 1 baseline measures (MAP) are likewise a nationally normed, standardized, computer- adaptive assessment (NWEA). While the study also used custom, researcher-designed domain-specific vocabulary and reading-comprehension instruments (which are appropriate as supplementary transfer measures but would not by themselves satisfy this criterion), the presence of a genuine standardized exam-based assessment (EOG) as a primary outcome is sufficient to meet this criterion. Criterion E is met because the study used state-wide EOG standardized assessments as a primary outcome measure for reading and mathematics.
  • T

    Term Duration

    • Outcomes were measured years after intervention start, which greatly exceeds the one-term minimum.
    • "In the current study, there were nearly 36 months between the beginning and end of the full MORE spiral curriculum."
    • Relevant Quotes: 1) "In the treatment condition (i.e., full spiral curriculum), students participated in content literacy lessons from Grades 1 to 3 during the school year and wide reading of thematically related informational texts in the summer following Grades 1 and 2." (p. 1279, Abstract) 2) "Treatment impacts were sustained at 14-month follow-up on Grade 4 reading comprehension (ES = .12) and mathematics achievement (ES = .16)." (p. 1279, Abstract) 3) "In the current study, there were nearly 36 months between the beginning and end of the full MORE spiral curriculum." (p. 1293) Detailed Analysis: The intervention began in Grade 1 (spring 2019) and outcomes were measured as late as spring of Grade 4 (roughly 2022), a 14-month follow-up after the Grade 3 posttest and nearly three years after the intervention began. This interval vastly exceeds the minimum one-term requirement, and it also satisfies the stronger Year Duration criterion (see Y below), which per the standard automatically satisfies this weaker Term Duration criterion. Criterion T is met because outcomes were tracked for multiple years after the intervention began, far exceeding one academic term.
  • D

    Documented Control Group

    • The control group's size, demographics, and baseline characteristics are documented in detail (Tables 2-3) and compared to the treatment group.
    • "In the control condition (i.e., partial spiral curriculum), students participated in lessons in only Grade 3."
    • Relevant Quotes: 1) "In the control condition (i.e., partial spiral curriculum), students participated in lessons in only Grade 3." (p. 1279, Abstract) 2) "Table 2 compares the demographic and baseline achievement characteristics of the students in the RCT sample to students in the non-RCT sample. ... more students received individual education plan (IEP; p < .01) and attained lower reading and mathematics scores at Grade 1 baseline (ps < .05) compared to the non-RCT sample." (p. 1284) 3) "Table 3, Balance Checks for Analytic Sample of Students Remaining in the Long-Term Impact Analysis (School-Level Averages)" lists control-school n = 15 and detailed means/SDs for White, Black, Hispanic, Asian, Male, Limited English proficiency, IEP, SES bands, and baseline MAP reading/mathematics scores for control schools versus treatment schools. (p. 1287) 4) "Follow-up 1 (G3 Spring) ... CONTROL: Analyzed (n = 871), Lost to follow-up (missing demographic or outcome data) (n = 412)." (Figure 3, p. 1286) Detailed Analysis: The paper documents the control condition in substantial detail: what the control group received ("lessons in only Grade 3," i.e., the partial spiral), its sample size at each wave (1,283 randomized, 871-902 analyzed at follow-ups), and its demographic and baseline-achievement characteristics via Tables 2 and 3, including race/ ethnicity, sex, English proficiency, IEP status, SES, and baseline MAP reading/mathematics scores, directly compared to the treatment group. A formal baseline-equivalence analysis (Table 3) is also reported. This level of detail allows a reader to assess whether the control group was comparable to the treatment group at baseline. Criterion D is met because the control group's composition, size, and characteristics are thoroughly documented and compared to the treatment group.

• Level 2 Criteria

  • S

    School-level RCT

    • Thirty entire schools were randomized to treatment or control, satisfying the school-level RCT requirement.
    • "For this longitudinal cluster (school) RCT study, 30 elementary schools in one urban school district located in the southeastern United States were recruited."
    • Relevant Quotes: 1) "A total of 30 elementary schools (N = 2,870 students) were randomized to a treatment or control condition." (p. 1279, Abstract) 2) "For this longitudinal cluster (school) RCT study, 30 elementary schools in one urban school district located in the southeastern United States were recruited." (p. 1284) 3) "Schools were blocked by three levels of school size ... and then randomized to a treatment (full MORE spiral curriculum from Grades 1 to 3) or a control condition." (pp. 1284-1285) 4) "Due to the cluster-randomized nature of the design, we applied clustered SEs at the school level using the 'cluster' option in Stata 17 (StataCorp, 2021)." (p. 1289) Detailed Analysis: The study is explicitly and repeatedly described as a "cluster (school) randomized controlled trial" in which entire elementary schools (the educational institution implementing the intervention) were the unit of randomization, assignment, and statistical clustering. Thirty schools (15 treatment, 15 control) were randomized after blocking on school size, prior reading proficiency, and prior MORE experience. This is a clear, unambiguous school-level RCT. Criterion S is met because randomization, allocation, and the analytic clustering were all conducted at the school level.
  • I

    Independent Conduct

    • The same research team that designed the MORE intervention also conducted and analyzed the trial, with no documented independent third-party evaluator.
    • "The MORE intervention was developed in the context of a research-practice partnership ... we collaborated with practitioners ... to improve Grade 3 reading comprehension."
    • Relevant Quotes: 1) "The MORE intervention was developed in the context of a research-practice partnership that focused on the twin goals of problem solving and knowledge generation (Donovan et al., 2021). As part of a 10-year partnership, we collaborated with practitioners in an urban school district located in the southeastern United States to improve Grade 3 reading comprehension..." (p. 1283) 2) "James S. Kim served as lead for conceptualization, funding acquisition, investigation, methodology, project administration, supervision, writing-original draft, and writing-review and editing." (p. 1279, Author Contributions) 3) "Joshua B. Gilbert and Ethan Scherer contributed equally to data curation, methodology, software, validation, and formal analysis." (p. 1279, Author Contributions) 4) "This research was funded by the Chan Zuckerberg Initiative. The opinions expressed are those of the authors and do not represent the views of the funding agency." (p. 1279) Detailed Analysis: The paper does not report that data collection, outcome administration, or analysis was conducted by a third party independent of the intervention designers. On the contrary, the author-contribution statement shows the same research team (led by James S. Kim, based at Harvard) that conceptualized, designed, developed (through a decade-long research-practice partnership), implemented, and funded-acquisition-managed the MORE intervention also performed the data curation, formal analysis, and reporting of the RCT. There is no mention of an external evaluation agency, blinded outside data collectors, or any third-party oversight of data collection or analysis; the only external party mentioned is the funder (Chan Zuckerberg Initiative), and the district's own EOG/MAP administrative assessments were used, but their scoring/analysis was performed in-house by the intervention's developers. This does not meet the standard's requirement for independent conduct. Criterion I is not met because the same research team that designed the MORE intervention also implemented, collected data for, and analyzed the RCT, with no documented independent or third-party evaluation.
  • Y

    Year Duration

    • The study tracked outcomes across roughly three years, well beyond the required 75% of one academic year.
    • "In the current study, there were nearly 36 months between the beginning and end of the full MORE spiral curriculum."
    • Relevant Quotes: 1) "In the current study, there were nearly 36 months between the beginning and end of the full MORE spiral curriculum." (p. 1293) 2) "Treatment impacts were sustained at 14-month follow-up on Grade 4 reading comprehension (ES = .12) and mathematics achievement (ES = .16)." (p. 1279, Abstract) 3) Figure 2 timeline shows intervention activities spanning "Grade 1 MORE Science/Social Studies + Wide Reading," "Grade 2 MORE Science + Wide Reading," "Grade 3 MORE Science (online)," and outcome measurement extending to "Time 4: Spring Grade 4 DGR, DGM." (p. 1285) Detailed Analysis: The intervention itself spans three full academic years (Grades 1 through 3) plus two intervening summers, and outcomes were tracked even further, into Grade 4 (a 14-month follow-up after the Grade 3 posttest). This tracking window (approximately 36 months from intervention start to final follow-up) vastly exceeds the 75%-of-one-academic-year threshold required by this criterion. Criterion Y is met because the intervention and its outcome tracking span multiple academic years, far exceeding 75% of one academic year.
  • B

    Balanced Control Group

    • The difference in dosage (full 3-year spiral vs. partial 1-year spiral) is the explicit treatment variable under test, and resources were equalized during the shared Grade 3 year, so the control condition is not an unbalanced confound.
    • "Thus, the experimental contrast was between the full spiral curriculum relative to a counterfactual condition involving the partial spiral curriculum."
    • Relevant Quotes: 1) "In the treatment condition (i.e., full spiral curriculum), students participated in content literacy lessons from Grades 1 to 3 during the school year and wide reading of thematically related informational texts in the summer following Grades 1 and 2. In the control condition (i.e., partial spiral curriculum), students participated in lessons in only Grade 3." (p. 1279, Abstract) 2) "This intervention study was designed to test the long-term impact and costs of the full Grades 1 to 3 spiral using a cluster (school) randomized controlled trial (RCT)..." (p. 1283) 3) "Thus, the experimental contrast was between the full spiral curriculum relative to a counterfactual condition involving the partial spiral curriculum." (p. 1284) 4) "In the spring of Grade 3 (2021)... we provided both treatment and control conditions with the MORE intervention lessons and associated professional development for teachers to ensure all students had equal access to learning opportunities." (p. 1285) Detailed Analysis: Applying the criterion B decision procedure: extra instructional resources are present, since the treatment group received two additional years (Grades 1-2) of MORE lessons and summer wide-reading that the control group did not receive during those years. The key question is whether this dosage difference is the treatment variable itself. It clearly is: the paper's central research question is whether a "full" 3-year spiral (Grades 1-3) produces larger and longer-lasting transfer effects than a "partial" 1-year spiral (Grade 3 only); the authors explicitly frame the "experimental contrast" as "full spiral curriculum relative to ... partial spiral curriculum." This is analogous to a dosage/duration study where the extra years of exposure are the primary manipulation under test, not an incidental confound. In the one grade where both conditions were active simultaneously (Grade 3), the authors explicitly equalized resources, giving the MORE curriculum and professional development to both treatment and control teachers "to ensure all students had equal access to learning opportunities." Thus, the design isolates the dosage (years of exposure) as the tested variable while removing unrelated confounds during the shared Grade 3 year. Criterion B is met because the additional Grades 1-2 MORE exposure is the explicit treatment variable being tested (full vs. partial spiral dosage), and the authors explicitly equalized instructional resources between conditions during the shared Grade 3 implementation.

• Level 3 Criteria

  • R

    Reproduced

    • The only replications of MORE referenced in the paper, or found in follow-up internet search, were conducted by the same research team, not by an independent team in a peer-reviewed outlet.
    • "In our most recent study (Kim et al., 2023), we reported findings from the Grades 1 to 2 implementation of the MORE spiral, in which we conducted a conceptual replication of the MORE spiral in Grade 1 science and social studies in spring 2019..."
    • Relevant Quotes: 1) "In our most recent study (Kim et al., 2023), we reported findings from the Grades 1 to 2 implementation of the MORE spiral, in which we conducted a conceptual replication of the MORE spiral in Grade 1 science and social studies in spring 2019..." (p. 1283) 2) "Previous research provides validating evidence that the MORE intervention can produce short-term impacts following 1- and 2-year program implementations. In an early efficacy study of the MORE 10-day thematic science lesson on the topic of animal survival in Grade 1..." (p. 1283) 3) Reference list: "Kim, J. S., Burkhauser, M. A., Relyea, J. E., Gilbert, J. B., Scherer, E., Fitzgerald, J., Mosher, D., & McIntyre, S. (2023). A longitudinal randomized trial of a sustained content literacy intervention from first to second grade..." (p. 1295); "Kim, J. S., Burkhauser, M. A., Mesite, L. M., Asher, C. A., Relyea, J. E., Fitzgerald, J., & Elmore, J. (2021). Improving reading comprehension, science domain knowledge, and reading engagement through a first-grade content literacy intervention..." (p. 1295); "Kim, J. S., Relyea, J. E., Burkhauser, M. A., Scherer, E., & Rich, P. (2021). Improving elementary grade students' science and social studies vocabulary knowledge depth, reading comprehension, and argumentative writing: A conceptual replication..." (p. 1296) 4) Internet search follow-up (2025): "Relyea, J. E., Gilbert, J. B., Burkhauser, M. A., Scherer, E., Mosher, D. M., Wei, Z., Tvedt, J. N., & Kim, J. S. (2025). Asset-based implementation of structured adaptations in an online third-grade content literacy intervention. Reading Research Quarterly." This paper describes a "conceptually replicating prior research" design testing structured teacher adaptations to MORE, but is authored by an overlapping/identical set of researchers (Relyea, Gilbert, Burkhauser, Scherer, Tvedt, Kim) as the present study, not an independent team. Detailed Analysis: The paper explicitly frames its own prior related studies as "conceptual replications" of the MORE intervention. However, every one of these replications (Kim, Burkhauser, et al., 2021; Kim, Relyea, et al., 2021; Kim et al., 2023) was conducted by the same lead author (James S. Kim) and largely overlapping co-authors as the present study, using the same research-practice partnership and, in some cases, the same longitudinal cohort. A targeted internet search for post-publication replications identified only one additional related study (Relyea et al., 2025, Reading Research Quarterly), which again shares the same core author team (Relyea, Gilbert, Burkhauser, Scherer, Tvedt, Kim) as this paper and is therefore not an independent replication. The ERCT standard requires that replication be conducted by a different, independent research team, ideally in a different context, and published in a peer-reviewed outlet. No evidence was found in the paper, in its author/citation network, or via internet search, of an independent research team outside of the Kim/READS Lab research group replicating this specific full Grades 1-3 spiral-curriculum design (or the earlier single/two-year MORE efficacy studies) in a different context. Criterion R is not met because the only replications identified, both referenced in the paper and found via subsequent internet search, are internal, conducted by the same research team as the original studies, not by an independent team.
  • A

    All-subject Exams

    • The study measured both EOG reading and EOG mathematics, the two core state-tested subjects at this grade level, not only the intervention's target subject.
    • "We examined cross-domain transfer on Grade 3 mathematics, which primarily assessed conceptual word problems requiring strong comprehension ability."
    • Relevant Quotes: 1) "We measured students' domain-general reading comprehension and mathematics ability with posttests in Grades 3 and 4, using state-wide end-of-grade (EOG) standardized assessments..." (p. 1289) 2) "We examined cross-domain transfer on Grade 3 mathematics, which primarily assessed conceptual word problems requiring strong comprehension ability." (p. 1284) 3) "As shown in Table 8, there were statistically significant ITT effects (ps < .001) on Grade 4 domain-general reading comprehension (ES = .12) and mathematics (ES = .16)." (p. 1290) Detailed Analysis: Because criterion E is met on the strength of the EOG standardized reading and mathematics assessments, this criterion (which requires E as a prerequisite) can be evaluated. The study assessed both domain-general reading (ELA) and mathematics achievement using state-wide EOG standardized tests in both Grades 3 and 4, in addition to domain-specific science outcomes. For this grade band (elementary Grades 1-4) in the study's U.S. state context, reading and mathematics are the core subjects with state-wide standardized End-of-Grade assessments (science EOG testing in North Carolina begins later, at Grade 5, and is not referenced as available for this sample). Because the study measured both of the state-tested core academic subjects available at this grade level, rather than reporting only the subject targeted by the literacy-focused intervention, it avoids the core problem this criterion targets (masking negative effects on untested subjects). Criterion A is met because the study assessed outcomes in both core standardized-exam subjects available at this grade level (reading and mathematics), not only the intervention's target subject.
  • G

    Graduation Tracking

    • Tracking ended at Grade 4, well short of graduation, with no evidence of continued follow-up to graduation found in this paper or in subsequent publications.
    • "Our third aim was to examine long-term impacts at a 14-month follow-up at the end of Grade 4."
    • Relevant Quotes: 1) "Our third aim was to examine long-term impacts at a 14-month follow-up at the end of Grade 4." (p. 1284) 2) "RQ 3: Are there long-term effects on Grade 4 reading comprehension and mathematics outcomes?" (p. 1284) 3) Figure 2 shows the final measurement point as "Time 4: Spring Grade 4 DGR, DGM," with "Grade 4 (No Program Implementation)" and no further waves depicted. (p. 1285) 4) "To examine the external validity of our findings, it will be critical to study the implementation of the full MORE spiral curriculum in new school districts to determine if the treatment effects replicate in other contexts." (p. 1294, Limitations) 5) Internet search follow-up: no subsequent publication by this author team (James S. Kim, Joshua B. Gilbert, Jackie Eunjung Relyea, and colleagues at Harvard's READS Lab) tracking this specific 2018-enrolled cohort of students beyond spring Grade 4 (2022) was located. The 2025 Relyea et al. paper on structured adaptations to MORE concerns a different, later cohort of third-grade classrooms and does not extend follow-up of this cohort. Detailed Analysis: Since Y is met, this criterion can be evaluated on its own merits. The longest follow-up reported is a 14-month follow-up at the end of Grade 4, which is still within elementary school and far short of graduation from elementary school (typically Grade 5 in the U.S.) let alone secondary school. The paper's Discussion and Limitations sections propose future directions (e.g., replicating the full spiral in new districts, examining the role of vocabulary versus background knowledge) but do not describe any planned or completed follow-up tracking students through graduation. A search of the authors' subsequent publications and the READS Lab's public research listings did not identify any companion follow-up publication tracking this specific cohort beyond Grade 4. Criterion G is not met because outcome tracking ended at Grade 4, well before students' graduation from elementary school, with no evidence of continued follow-up in this paper or in subsequent publications located via internet search.
  • P

    Pre-Registered

    • The study protocol was pre-registered on the AEA RCT Registry on October 24, 2018 (AEARCTR-0003489), before intervention and data collection began in spring 2019, as confirmed by independent verification of the registry.
    • "Details on the original study design and analysis are in our preregistration plan (Kim, 2018)."
    • Relevant Quotes: 1) "Details on the original study design and analysis are in our preregistration plan (Kim, 2018)." (p. 1287) 2) Reference: "Kim, J. S. (2018). A randomized controlled trial to replicate and personalize with technology the effects of a model of reading engagement (MORE) on first- and second-graders' science and social studies domain knowledge, reading engagement, reading comprehension, and basic literacy skills. AEA RCT Registry. https://doi.org/10.1257/rct.3489-1.2000000000000002" (p. 1295) 3) "In the spring of Grade 1 (2019) and 2 (2020), students in the treatment condition participated in the full MORE spiral curriculum..." (p. 1285) 4) AEA RCT Registry (AEARCTR-0003489), verified via internet search: the trial "A Randomized Controlled Trial to Replicate and Personalize with Technology the Effects of a Model of Reading Engagement (MORE)" was first registered October 24, 2018, with a planned intervention start date of October 25, 2018. Detailed Analysis: The paper cites a formal pre-registration of the study design and analysis plan on the AEA RCT Registry (Kim, 2018), a widely used, publicly accessible trial registry. Independent verification of the registry entry (AEARCTR-0003489) confirms the trial title matches the cited reference and that it was first registered on October 24, 2018, well before the earliest intervention and data collection activities described in the paper, which began in spring of Grade 1, 2019. This confirms that pre-registration occurred before data collection began, satisfying the timing requirement in addition to the existence requirement. Criterion P is met because the study protocol was publicly pre-registered (AEA RCT Registry, AEARCTR-0003489, first registered October 24, 2018) prior to the start of data collection in spring 2019, as confirmed by both the paper's citation and independent verification of the registry record.