Academic performance of subsequent schools and impacts of early interventions: Evidence from a randomized controlled trial in Head Start settings

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomization was done at the Head Start site (center) level, which satisfies or exceeds class-level randomization.
    • "One site in each matched pair was then randomly assigned to the intervention group and the other to the control group." (p. 947)
    • The Chicago School Readiness Project (CSRP) used a clustered randomized controlled trial (RCT) design. Two classrooms were randomly selected from each site. After the randomized assignment, one site in each matched pair was then randomly assigned to the intervention group and the other to the control group. The random assignment took place at the site (center) level, meaning entire sites were assigned to treatment or control. This method ensured that contamination between conditions within the same building was avoided.
  • E

    Exam-based Assessment

    • The study measured child-level academic outcomes via teacher ratings, not a standardized, exam-based assessment of each child.
    • "Academic outcomes were measured using a modified version of the Academic Rating Scale (ARS; Rock, Pollack, & Hausken, 2002)." (p. 949)
    • The study measured child-level academic outcomes using teacher ratings rather than a standardized, exam-based assessment. While the study referenced a standardized measure (the ISAT), this was only used as a school-level aggregate, not at the individual student level. Since the core academic outcome measures were based on teacher-reported rating scales, the Exam-based Assessment criterion is not met.
  • T

    Term Duration

    • The paper reports that the intervention was conducted over an entire Head Start academic term (fall to spring), meeting the term duration criterion.
    • "Cohort One participated from fall to spring in 2004–05; Cohort Two participated from fall to spring in 2005–06." (p. 947)
    • The intervention lasted from early fall until late spring, covering an entire Head Start academic cycle. This meets the required minimum duration of a single academic term (approximately 3–4 months).
  • D

    Documented Control Group

    • The control group’s business-as-usual setting is clearly described, including their staffing support and how it differed from the intervention.
    • "Teachers in the control group were given staffing support by a teacher’s aide..." (p. 947)
    • The study describes the control group’s conditions, including demographic characteristics and the additional support they received. Control classrooms maintained the regular Head Start program but with an extra aide to keep student–staff ratios comparable. This documentation of the control group setup satisfies the criterion.

• Level 2 Criteria

  • S

    School-level RCT

    • Whole Head Start sites (equivalent to schools) were the unit of randomization, fulfilling the school-level RCT requirement.
    • "One site in each matched pair was then randomly assigned to the intervention group and the other to the control group." (p. 947)
    • The randomization was conducted at the site (school) level: one site in each matched pair was assigned to the intervention and the other to control. By randomizing entire centers (schools) rather than smaller units, the study meets the School-level RCT criterion.
  • I

    Independent Conduct

    • The study does not mention any external evaluators. The intervention appears to have been evaluated by its own designers, lacking independent oversight.
    • There is no indication that a third-party team conducted the evaluation. The paper references prior studies by the same authors (Raver et al., 2009, 2011) for design and methods, implying the original researchers implemented and evaluated the program themselves. This absence of an independent evaluation team means the Independent Conduct criterion is not met.
  • Y

    Year Duration

    • The intervention spanned an entire preschool year (approximately 9 months), satisfying the one-year duration criterion.
    • "...at the end of the 9-month intervention during the Head Start year..." (p. 952)
    • The CSRP intervention was implemented over a full Head Start academic year (fall through spring). This year-long duration meets the requirement for an intervention lasting at least one academic year.
  • B

    Balanced Resources

    • The intervention group received extra training and mental health consultation services, whereas the control group did not receive comparable resources or attention.
    • "...who only provided additional staffing support ... for the same amount of time per week as the MHCs in the intervention group." (p. 947)
    • Teachers in the intervention group received extensive training and on-site mental health consultant support. In contrast, control group teachers only received help from an aide for an equivalent time, without any training or mental health services. Because the control condition did not get an equivalent level of resources or attention, the Balanced Resources criterion is not met.

• Level 3 Criteria

  • R

    Reproduced Results

    • No independent replication by other researchers is reported; this was a single-site study carried out by one team.
    • "The findings in our study should be interpreted with care. The CSRP intervention was conducted among a small sample of children ... in Chicago." (p. 953)
    • The authors caution that their findings stem from a single, small cluster-RCT in Chicago. We found no evidence of any other research team conducting an independent replication of the CSRP in a different context. Therefore, the Reproduced criterion is not met.
  • A

    All Exams

    • Academic performance was only assessed in language/literacy and math (via teacher-rated scales), rather than covering all core subjects with standardized exams.
    • "...including language and literacy ... and mathematical thinking..." (p. 949)
    • The study focused on language, literacy, and math outcomes (through teacher rating scales), and did not evaluate other core subjects like science or social studies using standardized tests. Thus, it did not administer exam-based assessments across all main academic subjects, and the All-subject Exams criterion is not satisfied.
  • G

    Graduation Tracking

    • The original CSRP participants were followed up in later years. A subsequent study by the same research team collected data on these students’ outcomes in high school, fulfilling the graduation tracking criterion.
    • "...taken toward the end of students’ high school careers." (2023 follow-up)
    • Although the 2012 paper itself only reports outcomes through kindergarten, the CSRP study did not end there. The researchers continued to track the children in later years. By approximately 11–14 years after the intervention, the original team gathered follow-up data as the participants reached late high school (Watts et al., 2023). This long-term tracking demonstrates that the study ultimately met the Graduation Tracking criterion.
  • P

    Pre-Registered Protocol

    • No pre-registered analysis plan or study registration is mentioned. There is no evidence that the trial was registered before data collection.
    • The authors did not reference any pre-registration or trial registry entry for this study. As the research appears to have been conducted without a publicly registered protocol, the Pre-registered Protocol criterion is not met.