Large-scale randomized controlled trial with 4th graders using intelligent tutoring of the structure strategy to improve nonfiction reading comprehension

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomization occurred at the classroom level, satisfying the class-level RCT requirement.
    • "One hundred and thirty-one teachers’ classrooms were randomly assigned to experimental conditions (ITSS and control) within schools.";
    • Entire classrooms (at the teacher level) were the unit of randomization, with 131 fourth-grade classes randomly assigned to either ITSS or control within their schools​. This class-level clustering prevents cross-group contamination and satisfies the criterion for a class-level RCT.
  • E

    Exam-based Assessment

    • A standardized reading test (Gray Silent Reading Test) was used for outcome measurement.
    • "The GSRT (Wiederholt and Blalock 2000) form B was administered at pretest and Form A was administered at post-test.";
    • The study employed the Gray Silent Reading Test (GSRT), a standardized reading comprehension exam, at both pretest and post-test​. The GSRT’s reliability was high (Cronbach’s α = 0.88);, indicating it is a well-established assessment. Using a recognized standardized test for outcome measurement meets the exam-based assessment criterion.
  • T

    Term Duration

    • Post-tests were administered after about 6–7 months of intervention, exceeding a single term.
    • "Students used the ITSS software for 6–7 months and then completed the post tests."​
    • The intervention duration spanned roughly 6–7 months of the school year before outcomes were measured​. This means post-tests were administered more than one academic term after the start of the intervention, fulfilling the term-length requirement.
  • D

    Documented Control Group

    • The control group’s composition and baseline performance were clearly documented and comparable to the intervention group.
    • "ITSS and control classrooms were comparable in their reading level before the implementation of the experiment."​
    • The control condition was clearly documented and equivalent to the intervention group at baseline. All students in control classes followed the standard language arts curriculum (business-as-usual) with no ITSS. Pretest data show no significant differences between ITSS and control groups; they “were comparable in their reading level” prior to the intervention​. Thus, the study provides detailed control group information and baseline equivalence, meeting this criterion.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomization was done at the class level within schools, not at the school level (no whole-school assignment).
    • "The multisite CRT design that uses teacher random assignment, within each school, was selected over other designs that use school- or student-level random assignment."​
    • This was a class-level trial conducted within schools rather than a whole-school randomization. The authors explicitly chose classroom (teacher-level) random assignment instead of randomizing entire schools​. Because schools themselves were not the unit of assignment, the study does not meet the school-level RCT criterion.
  • I

    Independent Conduct

    • The researchers who developed the intervention also implemented the study (no independent evaluators were involved).
    • "After random assignment the research team visited each school and conducted professional development sessions for teachers assigned to the ITSS group.";
    • The trial was carried out by the program’s own researchers without an independent party. For example, the ITSS developers (study authors) themselves provided teacher training and oversaw implementation​. There is no mention of any external evaluator or independent monitoring. Consequently, the study was not conducted independently of the intervention developers, and this criterion is not met.
  • Y

    Year Duration

    • The study spanned about 7 months of one school year, with no outcomes tracked for a full year or longer.
    • (No long-term follow-up beyond the intervention year was conducted.)
    • The active intervention and data collection lasted for most of one school year (approximately 6–7 months) but did not continue for a full academic year or beyond​. Outcomes were assessed at the end of the 4th-grade year, with no further follow-up into the next grade. Therefore, the study falls short of the one-year duration criterion.
  • B

    Balanced Resources

    • Both groups had equal instructional time and curricular resources; ITSS replaced part of the normal class time rather than adding extra time.
    • "Total time for daily and weekly language arts instruction was to be identical for both the intervention and control classrooms."​
    • The intervention did not receive extra instructional time or materials beyond what the control group received. In fact, the ITSS lessons were integrated as a partial substitute for regular language arts, not an addition. The authors state that the total allotted time for language arts was the same in both ITSS and control classes​. By holding instructional time and resources constant across groups, the study meets the balanced resources criterion.

• Level 3 Criteria

  • R

    Reproduced Results

    • The study has not been independently replicated by an unrelated research team.
    • "These results replicate findings from previous studies on the ITSS. The positive results provide additional evidence of the impact of ITSS on reading comprehension for students attending high-poverty schools who need support the most."​
    • No independent replication by a different research team has been documented for this 2012 study. The original authors themselves later conducted a follow-up efficacy trial in high-poverty schools, which reported that it “replicate findings from previous studies on the ITSS”. However, this 2024 replication was carried out by the same team that developed ITSS. As of this review, we found no evidence of an independent replication by outsiders. Thus, criterion R is not met.
  • A

    All Exams

    • Outcomes were limited to reading comprehension; no other core subjects were tested.
    • "Reading comprehension was measured using a standardized reading comprehension test and an experimenter-designed recall and main idea tests... (No other subjects were assessed.)";
    • The trial evaluated only reading-related outcomes. All outcome measures—GSRT and researcher-designed comprehension tasks—pertained to reading comprehension. It did not include assessments in other subject areas (such as math or science achievement). Therefore, it fails to meet the criterion of testing across all core subjects.
  • G

    Graduation Tracking

    • Participants were not tracked beyond the immediate post-test in 4th grade (no long-term follow-up through graduation).
    • "Control teachers were offered the same professional development for the following year once the study was completed, along with the option to use ITSS.";
    • The study tracked student outcomes only through the end of 4th grade, with post-tests administered at that point. There was no longitudinal follow-up of these students in subsequent years. In fact, after the study ended, control schools were given the ITSS intervention​, indicating the experiment concluded in that year. Because students were not followed to later grades or graduation, criterion G is not met.
  • P

    Pre-Registered Protocol

    • No pre-registered study protocol was identified for this trial.
    • (No pre-registration of the study was mentioned in the paper or found in registries.)
    • The authors did not indicate that the trial was pre-registered in any public registry before data collection. We found no registration information (e.g., a trial ID on ClinicalTrials.gov or an education RCT registry) in the paper. Without a documented pre-registered protocol, the study does not fulfill the transparency criterion for pre-registration.