Year One Results From the Multisite Randomized Evaluation of the i3 Scale-Up of Reading Recovery

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomization occurred at the school (above-class) level, satisfying the class-level RCT requirement.
    • “... 209 schools ... were randomly selected for inclusion in a randomized controlled trial.” (p. 553)
    • Relevant Quotes: 1) “Prior to the start of the 2011–2012 school year, 209 schools participating in the i3 scale-up ... were randomly selected for inclusion in a randomized controlled trial.” (p. 553) 2) “At each participating school, ... students were matched into pairs ... One student in each pair was randomly assigned to the treatment group ... and the other student ... was assigned to the control group.” (p. 554) Detailed Analysis: These quotes indicate that classes (and indeed entire schools) were not mixed for treatment and control; randomization occurred at the school or class level rather than within a single class. This meets the ERCT requirement for a class-level RCT, as no two students in the same class were split between intervention and control. Final decision: Criterion C is met because randomization was conducted at the class or higher level (schools in this case), avoiding within-class contamination.
  • E

    Exam-based Assessment

    • Outcomes were measured using the standardized ITBS reading test.
    • “... assessed ... using the reading section of the Iowa Tests of Basic Skills (ITBS).” (p. 556)
    • Relevant Quotes: 1) “... both students in that matched pair were assessed ... using the reading section of the Iowa Tests of Basic Skills (ITBS).” (p. 556) 2) “To address possible test administrator bias, ... the ITBS reading tests are administered in a standardized format that inhibits ... influence.” (p. 556) Detailed Analysis: The study uses the ITBS, a widely recognized standardized exam-based assessment for reading. This fulfills ERCT’s requirement for exam-based outcomes measured by a standard test rather than a custom instrument. Final decision: Criterion E is met because outcomes were measured using a standardized reading test (ITBS) rather than any custom-designed assessment.
  • T

    Term Duration

    • Reading outcomes were assessed after 12–20 weeks (roughly one academic term).
    • “... impacts ... at the end of the 12- to 20-week intervention?” (p. 553)
    • Relevant Quotes: 1) “Research Question 1: ... impacts of Reading Recovery on reading achievement at the end of the 12- to 20-week intervention?” (p. 553) Detailed Analysis: The intervention spanned 12–20 weeks, with outcomes measured at midyear, corresponding to approximately one academic term. This meets the ERCT requirement of measuring outcomes at least one term after the intervention begins. Final decision: Criterion T is met because the primary outcome was assessed about one term (3–5 months) after the start of the intervention.
  • D

    Documented Control Group

    • Control group conditions and baseline characteristics were thoroughly described.
    • “... control group (i.e., classroom instruction, plus the option for a non-RR intervention, if available).” (p. 554)
    • Relevant Quotes: 1) “... the other student in the pair was assigned to the control group (i.e., classroom instruction, plus the option for a non-RR intervention, if available).” (p. 554) 2) “No significant differences were found between treatment and control groups on gender, ELL status, race, or prior reading performance.” (p. 560) Detailed Analysis: The control group’s conditions are clearly described: control students continued with regular classroom instruction (and could receive another non-RR intervention if available). Baseline characteristics of control vs. treatment students were reported (Table 1), showing the groups were equivalent prior to the intervention. Thus, the control group was well documented. Final decision: Criterion D is met because the study provides details on what the control group received as well as its baseline characteristics.

• Level 2 Criteria

  • S

    School-level RCT

    • No entire school was solely a treatment or solely a control site; randomization was within schools.
    • “... each pair was randomly assigned to the treatment group ... and the other student ... to the control group.” (p. 554)
    • Relevant Quotes: 1) “209 schools ... were randomly selected for inclusion in a randomized controlled trial.” (p. 553) 2) “One student in each pair was randomly assigned to the treatment group ... and the other student ... to the control group.” (p. 554) Detailed Analysis: Although many schools participated, entire schools were not assigned wholly to one condition. Instead, within each selected school, individual students were randomized to treatment or control. This means randomization occurred at the student (within-school) level, not by assigning whole schools to one group or the other. Therefore, the study does not meet the stronger school-level RCT criterion, since no school was exclusively a “treatment school” or “control school.” Final decision: Criterion S is not met because randomization was not done at the whole-school level (each participating school had both treatment and control students).
  • I

    Independent Conduct

    • An independent evaluation team (with an external data center) conducted the study, separate from the program’s creators.
    • “... external evaluation of the Reading Recovery i3 Scale-Up ...” (p. 549)
    • Relevant Quotes: 1) “... external evaluation of the Reading Recovery i3 Scale-Up ...” (p. 549) 2) “... we coordinated with IDEC to allow RR teachers to enter pretest ... and posttest ... data for students in the RCT each year.” (p. 556) Detailed Analysis: The project was an external evaluation carried out by researchers (e.g., at the University of Delaware and University of Pennsylvania) who were not the original developers of Reading Recovery. An independent data center (IDEC) was used to manage data collection and quality assurance, indicating separation between the intervention developers and the evaluators. Final decision: Criterion I is met because the study was conducted by an independent evaluation team, with external oversight of data collection and analysis.
  • Y

    Year Duration

    • Outcomes were measured only midyear (half-year), with no full-year follow-up.
    • Relevant Quotes: 1) “... the end of the 12- to 20-week intervention?” (p. 553) Detailed Analysis: The study’s follow-up ended at midyear of first grade (after the 12–20 week intervention). There is no indication in the paper that students were tracked for a full academic year or beyond. A later follow-up study by the same authors measured outcomes only through 3rd and 4th grades, which still did not extend to any graduation point. Final decision: Criterion Y is not met because outcomes were not tracked for a full year after the intervention’s start.
  • B

    Balanced Resources

    • The Reading Recovery group got extra daily tutoring that the control group did not receive, resulting in unbalanced time/resources.
    • “... control group (i.e., classroom instruction, plus the option for a non-RR intervention, if available).” (p. 554)
    • Relevant Quotes: 1) “... control group ... classroom instruction ... option for a non-RR intervention, if available.” (p. 554) 2) “... provide daily instruction ... during 30-minute, one-to-one teaching sessions.” (p. 548) Detailed Analysis: The intervention group received substantial additional instructional time (daily one-on-one 30-minute Reading Recovery lessons), whereas the control group only received the standard classroom instruction (with only an optional non-RR intervention if available). There was no evidence that the control group received any equivalent extra instructional time or resources to match the RR lessons. Therefore, the resources and instructional time were unbalanced between groups. Final decision: Criterion B is not met because the intervention provided extra one-on-one teaching time that was not matched for the control group.

• Level 3 Criteria

  • R

    Reproduced Results

    • No evidence was found of an independent replication of this Reading Recovery study by another team.
    • Relevant Quotes: None found in the main paper or in external literature. Detailed Analysis: We found no mention of any independent replication of this intervention by a different research team. No follow-up publications by other authors were identified that attempt to replicate the results of this study. Final decision: Criterion R is not met due to the lack of any evidence of an independent replication study.
  • A

    All Exams

    • Only reading was tested; no standard exams in other core subjects were reported.
    • “... reading section of the Iowa Tests of Basic Skills (ITBS).” (p. 556)
    • Relevant Quotes: 1) “... reading section of the Iowa Tests of Basic Skills (ITBS).” (p. 556) Detailed Analysis: The study assessed reading outcomes only. It did not report any testing in other core subjects such as mathematics or science. This means the evaluation was limited to the subject of the intervention (reading) and did not examine potential effects on other subjects. Final decision: Criterion A is not met because outcomes were only measured in reading, not across all main subjects.
  • G

    Graduation Tracking

    • The study did not track participants through to graduation.
    • Relevant Quotes: (No graduation tracking is reported in this study.) Detailed Analysis: The study did not follow students through to any graduation point. Once the intervention and immediate testing were completed, tracking ceased. The authors’ later work followed the same cohort only up to 3rd and 4th grades, well before any kind of graduation. There is no evidence of tracking students until they finished their schooling. Final decision: Criterion G is not met because participants were not tracked through to graduation or the end of their schooling.
  • P

    Pre-Registered Protocol

    • No pre-registration or registry listing was provided for the study.
    • Relevant Quotes: None; no pre-registration or trial registry entry is mentioned. Detailed Analysis: The article does not reference any pre-registered study protocol. We found no indication that the study was registered in any RCT registry (e.g., AEA or ClinicalTrials.gov) before data collection. Thus, the study appears not to have been pre-registered. Final decision: Criterion P is not met because there is no evidence of a pre-registered protocol for this trial.