Abstract
We investigate the efficacy of a reading intervention integrated with Engaged Learners, a program that applies behavioral and cognitive principles to increase student behavioral attention and reduce distractions during instruction. Using a three-arm randomized controlled trial, we randomized 159 Grade 3-5 students with co-occurring reading and behavioral attention challenges to a researcher-implemented small group reading intervention with Engaged Learners (READ +ENGAGE), an identical researcher-implemented reading intervention without Engaged Learners (READ), or a Business-as-Usual (BaU) condition. The READ +ENGAGE condition students demonstrated statistically significant greater behavioral attention according to direct observations and interventionist reports than READ condition students. The pattern of reading effect sizes contrasting the conditions suggests that the READ +ENGAGE and READ conditions were associated with higher performance on reading outcomes than the BaU condition and students in the READ condition significantly outperformed the BaU condition on measures of word reading, fluency and reading comprehension, and mid-transfer vocabulary and reading comprehension. We also explored whether integrating the Engaged Learners program in a reading intervention would be associated with improved reading outcomes by contrasting the READ + ENGAGE to the READ condition; however, the findings did not support this hypothesis.
Full
Article
ERCT Criteria Breakdown
-
Level 1 Criteria
-
C
Class-level RCT
- Student-level randomization is acceptable here because the intervention is delivered as small-group tutoring (2-4 students).
- "This process resulted in 159 students randomized 1:1:1, blocked on school and grade." (p. 6)
Relevant Quotes:
1) "This process resulted in 159 students randomized 1:1:1, blocked on school and grade." (p. 6)
2) "Intervention group sizes ranged from two to four students and took place outside of the core reading block." (p. 8)
Detailed Analysis:
Criterion C requires class-level (or stronger) randomization to avoid contamination, but ERCT allows an exception when the intervention is personal teaching like tutoring. The paper states students were randomized (not classes) and that intervention groups were small (two to four students) and conducted outside the core reading block, which matches a tutoring-like delivery model.
Therefore, although randomization is at the student level, the tutoring exception applies and Criterion C is met.
Final sentence explaining if criterion C is met/not met because the intervention is delivered as small-group tutoring, so student-level randomization is acceptable under the ERCT exception.
-
E
Exam-based Assessment
- The study used widely recognized standardized reading assessments (TOWRE-2, TOSREC, GMRT) with standard scores and reported reliability.
- "Results are reported in standard scores." (p. 6)
Relevant Quotes:
1) "Results are reported in standard scores." (p. 6)
2) "The TOSREC (Wagner et al., 2010) is a 3-minute group-administered test of reading fluency and comprehension." (p. 7)
3) "Internal consistency reliability ranges from .91 to .93, and the Kuder-Richardson reliability statistic ranges from .92 to .93." (p. 7)
Detailed Analysis:
Criterion E requires standardized exam-based assessments (not custom tests aligned to the intervention). The paper uses multiple established standardized instruments: TOWRE-2 and TOSREC report standard scores, and GMRT reports high internal consistency and Kuder-Richardson reliability. Although the paper also includes "mid-transfer" measures created from curriculum items, the presence of multiple standardized tests as outcomes satisfies Criterion E.
Final sentence explaining if criterion E is met/not met because the primary outcomes include established standardized reading exams.
-
T
Term Duration
- Outcomes were measured after an intervention period running from October to February, which exceeds one academic term.
- "researcher-hired interventionists delivered 30-40-minute intervention lessons for 3-5 days per week from October to February." (p. 8)
Relevant Quotes:
1) "researcher-hired interventionists delivered 30-40-minute intervention lessons for 3-5 days per week from October to February." (p. 8)
Detailed Analysis:
Criterion T requires outcome measurement at least one academic term after the intervention begins (typically about 3-4 months). The intervention ran "from October to February", which is roughly 4-5 months, and posttest data were collected at the end of this period. This exceeds one term.
Final sentence explaining if criterion T is met/not met because the October-to-February timeline spans more than a full academic term.
-
D
Documented Control Group
- The Business-as-Usual control is described and the paper reports control-group demographics and details of services received.
- "The BaU condition students received instruction as determined by their schools without input or support from the research team." (p. 4)
Relevant Quotes:
1) "The BaU condition students received instruction as determined by their schools without input or support from the research team." (p. 4)
2) "Table 1. Demographic Characteristics of the Analytic Sample" (p. 7)
3) "Data indicated that school personnel delivered at least one reading intervention to 59.52% ... of the BaU ... students." (p. 11)
Detailed Analysis:
Criterion D requires a well-documented control group (who they are, baseline characteristics, and what they received). The paper defines the BaU condition, provides demographic characteristics by condition (Table 1), and quantifies school-provided interventions received by each condition, including BaU. This is sufficient documentation to interpret comparisons.
Final sentence explaining if criterion D is met/not met because the BaU group is clearly described and supported with demographics and services-received data.
-
Level 2 Criteria
-
S
School-level RCT
- Randomization occurred at the student level within schools rather than at the school level.
- "This process resulted in 159 students randomized 1:1:1, blocked on school and grade." (p. 6)
Relevant Quotes:
1) "This process resulted in 159 students randomized 1:1:1, blocked on school and grade." (p. 6)
Detailed Analysis:
Criterion S requires school-level randomization (schools assigned to conditions). The quote shows students were randomized and merely blocked by school and grade. That is not school-level assignment.
Final sentence explaining if criterion S is met/not met because the unit of randomization was students, not schools.
-
I
Independent Conduct
- The authors developed the Engaged Learners program and the study was researcher-implemented with coaching by the first author.
- "Our team developed the Engaged Learners program ..." (p. 3)
Relevant Quotes:
1) "Our team developed the Engaged Learners program ..." (p. 3)
2) "...randomized Grade 3-5 students ... to a researcher-delivered reading intervention ..." (p. 4)
3) "Bi-weekly-or as needed-coaching was provided ... by doctoral students or the first author." (p. 10)
Detailed Analysis:
Criterion I requires independent conduct by an external evaluator. The paper explicitly states the authors' team developed Engaged Learners, the interventions were researcher-delivered, and coaching was provided by the first author or doctoral students. No external independent evaluation team is described.
Final sentence explaining if criterion I is met/not met because the intervention developers also implemented and oversaw delivery.
-
Y
Year Duration
- The intervention and measurement window (October to February) is under a full academic year.
- "researcher-hired interventionists delivered 30-40-minute intervention lessons for 3-5 days per week from October to February." (p. 8)
Relevant Quotes:
1) "researcher-hired interventionists delivered 30-40-minute intervention lessons for 3-5 days per week from October to February." (p. 8)
Detailed Analysis:
Criterion Y requires outcomes measured at least one full academic year after the intervention begins (about 9-10 months in typical K-12 contexts). The stated October-to-February period is about 4-5 months, so it does not meet the year-duration requirement.
Final sentence explaining if criterion Y is met/not met because the stated study timeline is substantially shorter than a full academic year.
-
B
Balanced Control Group
- The intervention adds substantial instructional time and staffing, but these added resources are integral to what the study is testing against Business-as-Usual.
- "researcher-hired interventionists delivered 30-40-minute intervention lessons for 3-5 days per week from October to February." (p. 8)
Relevant Quotes:
1) "researcher-hired interventionists delivered 30-40-minute intervention lessons for 3-5 days per week from October to February." (p. 8)
2) "Intervention group sizes ranged from two to four students and took place outside of the core reading block." (p. 8)
3) "The BaU condition students received instruction as determined by their schools without input or support from the research team." (p. 4)
Detailed Analysis:
Criterion B asks whether time, budget, and staffing inputs are balanced between intervention and control, unless the added inputs are the treatment variable (or are integral to the intervention as defined and tested). Here, both active conditions receive a researcher-implemented small-group reading intervention delivered by researcher-hired interventionists outside the core reading block, which clearly adds instructional time and staffing relative to BaU.
The paper's research questions explicitly compare the packaged interventions (READ and READ+ENGAGE) against Business-as-Usual, meaning the added tutoring time and staffing are integral to the defined intervention being evaluated, rather than an incidental add-on that should be matched by a placebo activity. Between the two active arms (READ vs READ+ENGAGE), delivery format and dosage are intended to be comparable, with differences focused on the Engaged Learners components.
Final sentence explaining if criterion B is met/not met because the added resources (small-group tutoring delivered by hired staff) are an integral part of the intervention package being tested against Business-as-Usual.
-
Level 3 Criteria
-
R
Reproduced
- No peer-reviewed independent replication by other authors was found, and the paper itself calls for future replications.
- "Replications of this study should consider its use." (p. 15)
Relevant Quotes:
1) "Replications of this study should consider its use." (p. 15)
Detailed Analysis:
Criterion R requires an independent reproduction by a different research team in a different context, published in a peer-reviewed journal. The paper itself frames replication as a future need.
An internet search (as of 2025-12-30) did not identify any peer-reviewed studies by other author teams that explicitly report an independent replication of this specific three-arm RCT (READ + ENGAGE vs READ vs BaU) with the same Engaged Learners integration.
Final sentence explaining if criterion R is met/not met because no independent peer-reviewed replication was found.
-
A
All-subject Exams
- The study measured reading and attention outcomes only, not standardized outcomes across all core subjects.
- "Reading Outcomes ... Attention Outcomes ..." (pp. 13-14)
Relevant Quotes:
1) "Reading Outcomes" (p. 13)
2) "Attention Outcomes" (p. 13)
Detailed Analysis:
Criterion A requires standardized, exam-based measurement across all main subjects (not just the targeted domain) to detect possible trade-offs. This study reports reading outcomes and behavioral attention outcomes, and does not report standardized measures in other core subjects (for example mathematics or science). Therefore the all-subject requirement is not met.
Final sentence explaining if criterion A is met/not met because the outcomes cover reading and attention only, not all core subjects.
-
G
Graduation Tracking
- The study does not track students to graduation, and Criterion G cannot be met because Criterion Y is not met.
Relevant Quotes:
1) "researcher-hired interventionists delivered 30-40-minute intervention lessons for 3-5 days per week from October to February." (p. 8)
2) "Graduate assistants (GA) ... coded each student's behavioral attention from a randomly selected pretest (baseline) and posttest (last 2 weeks) video-recorded lesson." (p. 8)
Detailed Analysis:
Criterion G requires tracking participants until graduation, and ERCT specifies that if Criterion Y (year duration) is not met then Criterion G is not met. This study's described timeline runs from October to February with pretest and posttest measurement, which is far shorter than year-long tracking, and there is no description of tracking students through an educational-stage graduation.
An internet search (as of 2025-12-30) did not identify follow-up papers by the same authors reporting graduation tracking for this cohort.
Final sentence explaining if criterion G is met/not met because the study is short-term (under a year) and no graduation-tracking follow-up was found.
-
P
Pre-Registered
- The authors explicitly state that the study was not pre-registered.
- "This study's design, hypotheses, and data analytic plan were not pre-registered." (p. 12)
Relevant Quotes:
1) "This study's design, hypotheses, and data analytic plan were not pre-registered." (p. 12)
Detailed Analysis:
Criterion P requires a pre-registered protocol with registration occurring before data collection begins. The paper explicitly states the design, hypotheses, and analytic plan were not pre-registered, so the criterion is not met and no registry verification is needed.
Final sentence explaining if criterion P is met/not met because the authors state the study was not pre-registered.
Request an Update or Contact Us
Are you the author of this study? Let us know if you have any questions or updates.
