Effectiveness of Tier 1 Content-Integrated Literacy Intervention on Early Elementary English Learners' Reading Comprehension and Writing: Evidence from Randomized Controlled Trial

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • The study randomized entire schools (a stronger unit than class-level), which automatically satisfies the class-level RCT requirement.
    • "This cluster randomized controlled trial (RCT) study was conducted in 30 elementary schools in a large, urban school district in the southeastern U.S." (p. 14)
    • Relevant Quotes: 1) "This cluster randomized controlled trial (RCT) study was conducted in 30 elementary schools in a large, urban school district in the southeastern U.S." (p. 14) 2) "We used a randomized block design, in which 30 schools were stratified into seven blocks based on demographic characteristics and academic performance. Within each block, half the schools were randomly assigned to treatment in first-grade treatment lessons (control condition in second grade) and half were assigned to second-grade treatment lessons (control condition in first grade)." (p. 14) Detailed Analysis: The unit of randomization was the entire school, not individual classes or students within a single classroom. The ERCT Standard treats school-level randomization as automatically satisfying the weaker class-level criterion, since randomizing whole schools removes any risk of within-school or within-class contamination between treatment and control conditions. Criterion C is met because the study's unit of randomization (school) is stronger than and subsumes the class-level requirement.
  • E

    Exam-based Assessment

    • Reading comprehension and oral proficiency were measured with MAP and WIDA, both widely used, standardized assessments, not custom-built tests.
    • "The MAP Reading is a computerized adaptive assessment to measure K-2 students' abilities or growth in reading comprehension using the Rasch unit (RIT) scale." (p. 18-19)
    • Relevant Quotes: 1) "The MAP Reading test was used to assess students' reading comprehension before (pretest) and after (posttest) the intervention implementation. The MAP Reading is a computerized adaptive assessment to measure K-2 students' abilities or growth in reading comprehension using the Rasch unit (RIT) scale." (p. 18-19) 2) "The WIDA screener was designed to assess English language proficiency of ELs based on WIDA's English language development standards (WIDA Consortium, 2015) that reflect the social and academic language needed and used in a K-12 school context." (p. 19) 3) "Test-retest reliabilities ranged from .89 to .96 (Brown & Coughlin 2007)." (p. 19, re: MAP Reading) Detailed Analysis: The study's primary reading, listening, speaking, and writing outcomes were measured using MAP (Measure of Academic Progress) and WIDA ACCESS for ELLs, both nationally recognized, widely used, standardized assessments with established psychometric properties, administered and scored by external testing organizations (NWEA and DRC respectively) rather than being purpose-built by the research team. Although some secondary outcomes (domain-specific vocabulary, argumentative writing) used researcher-designed instruments, the core reading/language outcomes relied on established standardized tests. Criterion E is met because MAP and WIDA are widely recognized standardized assessments used to measure the study's primary reading and language outcomes.
  • T

    Term Duration

    • The intervention and its outcome measurement spanned only about five weeks, far short of the one-term minimum required by the standard.
    • "...was implemented during the content-area block in classrooms over the period of five weeks." (p. 15)
    • Relevant Quotes: 1) "The MORE treatment curriculum was designed in a science and social studies thematic units in first and second grade that consisted of 10 lessons in each unit and was implemented during the content-area block in classrooms over the period of five weeks." (p. 15) 2) "...we did not find significant improvement in reading compression measured by MAP reading after the 20-lesson intervention presumably because reading comprehension is considered an unconstrained and complex reading skill that develops gradually and incrementally over time..." (p. 32) 3) "The MAP Reading test was used to assess students' reading comprehension before (pretest) and after (posttest) the intervention implementation." (p. 18-19) Detailed Analysis: The paper does not provide explicit calendar start and end dates, but it clearly and repeatedly characterizes the entire intervention as a 20-lesson program delivered "over the period of five weeks," with posttests administered immediately following this implementation window. There is no mention of a delayed or follow-up measurement occurring a term or more after the intervention began; the "posttest" appears to directly follow the five-week implementation period. Five weeks is far shorter than the one academic term (roughly 3-4 months) required by the ERCT Standard, and the paper gives no indication that outcome measurement was separated from intervention start by anything approaching a term. Criterion T is not met because the interval from intervention start to outcome measurement was only about five weeks, well under the one-term minimum.
  • D

    Documented Control Group

    • The control group's demographics, baseline scores, and business-as-usual instructional condition are documented in detail, including a dedicated table.
    • "Table 1 ... Baseline Demographic Characteristics of English Learners by Intervention Treatment Condition" (p. 50)
    • Relevant Quotes: 1) "Table 1 ... Baseline Demographic Characteristics of English Learners by Intervention Treatment Condition" listing N, grade, gender, race/ethnicity, SES status, individualized education program rate, school-wide EL proportion, and MAP/DIBELS pretest scores separately for treatment (n=681) and control (n=633) groups. (p. 50, Table 1) 2) "We found a statistically significant difference between the two groups in baseline MAP reading (treatment: M = 166.48, SD =16.95; control: M = 168.79, SD = 16.96) (p = .02); thus, the baseline MAP reading scores were included in the statistical models to reduce bias in the estimate of the treatment effects." (p. 24-25) 3) "Control schools" received "control condition in second grade" or "control condition in first grade" as part of the crossed randomized block design, i.e., business-as-usual instruction in the non-treated grade. (p. 14) Detailed Analysis: Table 1 provides a full breakdown of demographic and baseline characteristics (gender, race/ethnicity, SES status, IEP status, school-wide EL proportion, MAP and DIBELS pretest scores) separately for the treatment and control groups, including sample sizes. The authors also explicitly note and control for a statistically significant baseline difference in MAP reading between groups, demonstrating close attention to control group comparability. The nature of control-condition instruction (regular classroom instruction in the non-treatment grade at the same school) is also described. Criterion D is met because the control group's demographics, size, and baseline characteristics are documented in detail via Table 1 and accompanying text.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomization occurred at the school level (via randomized blocks of schools), satisfying the stronger school-level RCT criterion directly.
    • "This cluster randomized controlled trial (RCT) study was conducted in 30 elementary schools ..." (p. 14)
    • Relevant Quotes: 1) "This cluster randomized controlled trial (RCT) study was conducted in 30 elementary schools in a large, urban school district in the southeastern U.S." (p. 14) 2) "We used a randomized block design, in which 30 schools were stratified into seven blocks based on demographic characteristics and academic performance. Within each block, half the schools were randomly assigned to treatment in first-grade treatment lessons (control condition in second grade) and half were assigned to second-grade treatment lessons (control condition in first grade)." (p. 14) Detailed Analysis: The randomization unit was explicitly the school: 30 schools were stratified into blocks and randomly assigned within each block to determine which grade (first or second) received treatment versus control lessons. This is a direct school-level cluster randomized design, not classroom- or student-level randomization. Criterion S is met because the study is explicitly a school-level cluster RCT.
  • I

    Independent Conduct

    • The same research team that designed the MORE intervention and its predecessor studies also implemented, supported, and scored the current trial, with no independent evaluator described.
    • "The research team continuously provided treatment teachers with ongoing support and assistance along with more detailed guidance on lesson materials and implementation." (p. 16)
    • Relevant Quotes: 1) "The present study was designed to replicate findings from the previous studies of Tier 1 content-integrated literacy intervention for first- and second-grade students (Authors, 2021a; 2021b) and extend understanding of the effects on the EL population." (p. 13) 2) "The research team continuously provided treatment teachers with ongoing support and assistance along with more detailed guidance on lesson materials and implementation. Literacy facilitators at each school collaborated with the research team to address teachers' logistical and instructional questions." (p. 16) 3) "Two raters or argumentative writing participated in extensive training and practice process using the scoring manual and anchor texts ... The final scores were determined by consensus reached by the raters and one of the authors after discussion to resolve discrepancies." (p. 20-21) Detailed Analysis: The paper identifies itself as a direct replication and extension of the authors' own prior intervention studies ("Authors, 2021a; 2021b"), indicating that the same research team both designed the MORE curriculum and conducted the present evaluation of it. Furthermore, the research team is described as directly and continuously supporting treatment teachers' implementation, and one of the study's own authors participated in resolving scoring discrepancies for the writing outcome measure. There is no mention anywhere in the paper of an independent, third-party organization conducting data collection, scoring, or analysis separate from the intervention's designers. Criterion I is not met because the same research team that designed the intervention (and its predecessor versions) also implemented, supported, and helped score the outcome measures for this trial, with no independent evaluator identified.
  • Y

    Year Duration

    • Because the Term Duration (T) criterion was not met, the stronger Year Duration criterion cannot be met either, per the standard's cascade rule.
    • "...was implemented during the content-area block in classrooms over the period of five weeks." (p. 15)
    • Relevant Quotes: 1) "The MORE treatment curriculum was designed in a science and social studies thematic units in first and second grade that consisted of 10 lessons in each unit and was implemented during the content-area block in classrooms over the period of five weeks." (p. 15) Detailed Analysis: Per the ERCT Standard, if the weaker Term Duration (T) criterion is not met, the stronger Year Duration (Y) criterion, which requires outcomes to be tracked across at least 75% of an academic year, also cannot be met. The intervention and measurement window here was approximately five weeks, dramatically short of even the term-length minimum, let alone a year-long tracking period. Criterion Y is not met because criterion T was not met, and the study's five-week window falls far short of the required academic-year duration in any case.
  • B

    Balanced Control Group

    • Treatment classrooms spent more time on science and social studies content, but this extra content-area time is integral to the content-integrated literacy design being tested, not an unrelated add-on.
    • "...treatment teachers reported spending nearly 60 minutes more on science and social studies instruction relative to their counterparts." (p. 17)
    • Relevant Quotes: 1) "Treatment teachers, on average, spent approximately 20 minutes/week less on English Language Arts (ELA)/reading instruction than control teachers. However, treatment teachers reported spending nearly 60 minutes more on science and social studies instruction relative to their counterparts." (p. 17) 2) "The current intervention consisted of the following evidence-based instructional components: (a) the use of conceptually related complex informational texts in a thematic unit in interactive read-aloud activities, (b) domain- specific vocabulary network building, (c) argumentative writing, and (d) research collaboration." (p. 9) 3) "Control schools" received business-as-usual instruction, i.e., the non-treatment grade at each school continued with standard reading/ELA instruction rather than the MORE curriculum. (p. 14) Detailed Analysis: Following the criterion B decision procedure: extra resources are present (treatment classrooms spent roughly 60 more minutes/week on science and social studies content, and 20 fewer minutes/week on generic ELA instruction). However, the additional science/social-studies instructional time is not a separable "bonus" resource added on top of an otherwise-identical curriculum; it is the very substance of the content-integrated literacy intervention being tested, i.e., the study's central hypothesis is that embedding literacy instruction in content-rich science/social-studies material (rather than generic ELA time) improves outcomes. The control classrooms received the standard, business-as-usual ELA/content instruction typical of the district. This matches the standard's guidance that when additional resources are integral to the treatment package being tested, business-as-usual conditions for the control group are acceptable. Criterion B is met because the additional science/social-studies instructional time is integral to the content-integrated literacy design being evaluated, and the control group received standard business-as-usual instruction.

• Level 3 Criteria

  • R

    Reproduced

    • The paper is a self-replication and extension by the same research team of its own prior work; no independent replication by a separate team was found in the paper or via internet search.
    • "The present study was designed to replicate findings from the previous studies of Tier 1 content-integrated literacy intervention for first- and second-grade students (Authors, 2021a; 2021b) and extend understanding of the effects on the EL population." (p. 13)
    • Relevant Quotes: 1) "The current study replicated and extended the previous findings of content-integrated literacy intervention focusing on its effectiveness on first- and second-grade English learners' ... reading comprehension, writing, vocabulary knowledge, and oral proficiency." (Abstract, p. 2) 2) "The present study was designed to replicate findings from the previous studies of Tier 1 content-integrated literacy intervention for first- and second-grade students (Authors, 2021a; 2021b) and extend understanding of the effects on the EL population." (p. 13) 3) "Findings replicate main effects on vocabulary knowledge and writing, while also extending previous research by highlighting mechanisms underlying improved reading comprehension and writing." (Abstract, p. 2) 4) Internet search (OpenAlex citation records and related-work search) identified the peer-reviewed version of this same study, "Effects of tier 1 content literacy intervention on early-grade English learners' reading and writing" (Relyea, Kim, Rich, & Fitzgerald, 2024, Journal of Educational Psychology, 116(7), 1172-1195, DOI 10.1037/edu0000882), which is the same underlying trial and dataset carried to publication by the same authors (with an added co-author), not an independent replication. A search of the 14 works citing this 2024 publication (via OpenAlex) found only thematically related reviews and further studies by overlapping members of the same research team (e.g., Relyea as lead author on 2025 follow-on intervention papers); no independently authored study by a separate research team replicating this specific EL cluster RCT was identified. Detailed Analysis: The paper explicitly frames itself as a replication, but it is a replication conducted by the same research team as the original studies (cited as "Authors, 2021a; 2021b" - the very authors of the current paper), extended to a new subpopulation (ELs) rather than an independent team repeating the study in a different context. The ERCT Standard's Reproduced criterion requires independent replication by a different research team, published in a peer-reviewed outlet. Internet searching for subsequent studies and citation records did not surface any independent, differently-authored replication of this specific study; the only closely related subsequent publication is the same team's own peer-reviewed write-up of this trial. Criterion R is not met because the replication described is a self-replication by the same author team, not an independent reproduction by a different research group, and no independent reproduction was located via internet search.
  • A

    All-subject Exams

    • Outcomes were limited to reading, writing, vocabulary, and oral proficiency in science and social studies content; no other core subjects (e.g., mathematics) were assessed.
    • "...focusing on its effectiveness on first- and second-grade English learners' (N = 1,314) reading comprehension, writing, vocabulary knowledge, and oral proficiency." (Abstract, p. 2)
    • Relevant Quotes: 1) "The current study replicated and extended the previous findings of content-integrated literacy intervention focusing on its effectiveness on first- and second-grade English learners' (N = 1,314) reading comprehension, writing, vocabulary knowledge, and oral proficiency." (Abstract, p. 2) 2) "Statistically significant findings were replicated on science and social studies vocabulary knowledge (ES = .51 and .53, respectively) and argumentative writing (ES = .27 and .41, respectively)." (Abstract, p. 2) 3) There is no mention anywhere in the Measures section of a mathematics assessment or any other core subject beyond reading/language arts (via science and social studies content vehicles). Detailed Analysis: Criterion E is met (see criterion e), so criterion A can be evaluated on its own merits. The study measured reading comprehension, writing, vocabulary knowledge, and oral proficiency, using science and social studies texts as the content vehicle for literacy instruction, but it did not assess mathematics or any other core academic subject. Although this is fundamentally a literacy intervention (not a math intervention), the ERCT Standard's All-subject criterion asks whether the study measured impact across all main subjects to rule out unintended negative effects elsewhere in the curriculum; here, because treatment classrooms spent less time on generic ELA/reading instruction and more time on science/social studies content, a plausible concern is whether other academic areas (e.g., math) were affected, but this was never examined. No exception rationale (e.g., a specialized upper- secondary/vocational focus) is provided that would justify omitting other subjects. Criterion A is not met because the study did not assess outcomes in mathematics or other core subjects beyond the literacy/language and content-area vocabulary outcomes it targeted.
  • G

    Graduation Tracking

    • Because the Year Duration (Y) criterion was not met, Graduation Tracking cannot be met either, per the standard's cascade rule; internet search of subsequent publications also found no evidence of graduation-level follow-up of this EL cohort.
    • "...was implemented during the content-area block in classrooms over the period of five weeks." (p. 15)
    • Relevant Quotes: 1) "The MORE treatment curriculum was designed in a science and social studies thematic units in first and second grade that consisted of 10 lessons in each unit and was implemented during the content-area block in classrooms over the period of five weeks." (p. 15) 2) "...future research on sustained, multi-year content-integrated literacy intervention is warranted." (p. 35, Limitations) 3) Internet search identified a related follow-up study by overlapping members of the same research team, "A longitudinal randomized trial of a sustained content literacy intervention from first to second grade" (Kim, Burkhauser, Relyea, Gilbert, Scherer, Fitzgerald, Mosher, & McIntyre, 2023, Journal of Educational Psychology, 115(1)), but this study extends the general-population MORE intervention only from first to second grade, not through graduation, and does not report graduation-level tracking. The 2024 peer-reviewed version of the present EL study itself (Relyea, Kim, Rich, & Fitzgerald, 2024, Journal of Educational Psychology, 116(7)) likewise reports only immediate posttest outcomes for this EL cohort, with no longer-term or graduation-level follow-up located. Detailed Analysis: Per the ERCT Standard, if criterion Y (Year Duration) is not met, criterion G (Graduation Tracking) is automatically not met as well. In addition, the paper's own Limitations section explicitly calls for future "sustained, multi-year" intervention research, confirming that no long-term or graduation-level follow-up was conducted in the current study; outcomes were measured immediately after the five-week, 20-lesson intervention. Internet search for subsequent publications tracking this same EL cohort toward graduation did not locate any such follow-up study. Criterion G is not met because criterion Y was not met, and the study includes no follow-up beyond the immediate posttest, a conclusion corroborated by the absence of any graduation-tracking follow-up publication found via internet search.
  • P

    Pre-Registered

    • The paper states the study design was preregistered but provides no registry name, ID, link, or date to verify that registration preceded data collection, and internet search could not locate the registration record.
    • "We preregistered the study design and conducted a power analysis to determine the minimum detectable effect size." (Study Design and Participant Selection)
    • Relevant Quotes: 1) "This cluster randomized controlled trial (RCT) study was conducted in 30 elementary schools in a large, urban school district in the southeastern U.S. We preregistered the study design and conducted a power analysis to determine the minimum detectable effect size." (Methods, Study Design and Participant Selection) 2) No other passage in the paper, including the references list, provides a registry name (e.g., OSF, AEA RCT Registry, ClinicalTrials.gov), a registration ID, a link, or a registration date. Detailed Analysis: Both the quote and the absence-of-detail finding were re-checked against the PDF and confirmed. The ERCT standard requires quoted evidence of a pre-registration reference together with a date confirming that registration occurred before data collection began. While the authors do state, in passing, that the study design was preregistered, no registry platform, ID, link, or date is provided anywhere in the manuscript to substantiate this claim or to verify its timing relative to data collection. Internet-based follow-up: web searches and direct queries against OpenAlex, Crossref, Semantic Scholar, and the 2024 peer-reviewed Journal of Educational Psychology version of this study did not surface a registry name, ID, link, or registration date for this trial. Per the standard's strict evidentiary requirement, an unverifiable claim of preregistration without a link or date does not satisfy the criterion. Criterion P is not met because no registry link, ID, or date is provided or could be located to verify the claimed preregistration occurred before data collection.