The Effectiveness of a Language Comprehension Intervention on the Language, Reading, and Writing Skills of Socioeconomically Disadvantaged Students

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomization was conducted within each classroom at the individual-student level, not at the class or school level, and the group intervention is not a personal tutoring exception.
    • "The randomization was conducted within each classroom, and the sample was stratified according to their ethnicity (Roma vs. non-Roma)."
    • Relevant Quotes: 1) "The qualifying students were randomly assigned to either experimental or control treatment. The randomization was conducted within each classroom, and the sample was stratified according to their ethnicity (Roma vs. non-Roma). The participants were randomly assigned to one of the two treatments by a researcher who was not involved in the project using the following Stata command: randtreat, generate (treatment) strata (School Roma) misfits (wstrata)." (p. 5) 2) "The students from the experimental group were removed from their classes and received the intervention in another classroom within the same school building. Each experimental group included about 6 students." (pp. 7-8) 3) "The students from the control group received instruction from their homeroom teachers." (p. 10) Detailed Analysis: Criterion C requires that randomization be performed at the class level or stronger (school level), to prevent contamination between treatment and control students who share the same environment. The paper states explicitly that "the randomization was conducted within each classroom," meaning individual students within the same class were assigned to experimental or control conditions. This is precisely the within-class, student-level randomization that criterion C is designed to catch, because treatment and control students remain in the same classroom for the rest of the school day, creating a risk of contamination. The tutoring/personal-teaching exception does not apply: the intervention was delivered to small groups of about six students, not as one-to-one tutoring. Therefore the weaker student-level randomization is not permissible here. Criterion C is not met because randomization was carried out within each classroom at the individual-student level rather than at the class or school level.
  • E

    Exam-based Assessment

    • The study measured outcomes with several widely recognized standardized tests (WISC-IV, KTEA-3, YARC, TOWRE-2, WIAT-III) adapted to Romanian, not solely custom-made instruments.
    • "We measured the generalized vocabulary knowledge with the Romanian version of the expressive vocabulary subscale of Wechsler Intelligence Scale for Children, Fourth Edition (WISC-IV) (Wechsler 2003)."
    • Relevant Quotes: 1) "We measured the generalized vocabulary knowledge with the Romanian version of the expressive vocabulary subscale of Wechsler Intelligence Scale for Children, Fourth Edition (WISC-IV) (Wechsler 2003)." (p. 9) 2) "We measured the students' ability to understand an orally conveyed message with 5 passages of increased difficulty from Kaufman Test of Educational Achievement--Third Edition (KTEA-3; Kaufman and Kaufman 2014)." (p. 9) 3) "We measured the students' reading comprehension with the Romanian adapted version of Missing Handbag passage from the York Assessment of Reading for Comprehension (Snowling et al. 2011)." (pp. 9-10) 4) "We measured the students' word reading fluency with a Romanian adapted test similar to the Test of Word Reading Efficiency--Second Edition (TOWRE-2, Torgesen et al. 2012)." (p. 10) 5) "The written composition was measured with a Romanian adapted version of the Essay Composition subtest of Wechsler Individual Achievement Test--Third edition (WIAT-III; Wechsler 2009)." (p. 10) Detailed Analysis: Criterion E requires that outcomes be measured with standardized, widely recognized tests rather than instruments created specifically for the study. Although the researchers also employed several researcher- developed measures (which included taught words), they additionally used a suite of internationally recognized standardized instruments: WISC-IV, KTEA-3, YARC, TOWRE-2, and WIAT-III. These are established, validated assessments that were adapted to the Romanian context, and the paper reports high internal consistency for each. The presence of these recognized standardized tests satisfies the requirement, independently of the custom measures. Criterion E is met because the study used widely recognized standardized tests (WISC-IV, KTEA-3, YARC, TOWRE-2, and WIAT-III) to assess outcomes.
  • T

    Term Duration

    • The intervention ran for more than 12 months, and outcomes were measured immediately after and 8 months later, far exceeding one academic term.
    • "our intervention lasted longer (>12months) than most vocabulary interventions"
    • Relevant Quotes: 1) "The students were assessed prior to the intervention (Time 1, January 2022) when they were in 3rd grade, within a 3-week window. Then, the students were assessed immediately after the intervention (Time 2, March 2023) when the students were in 4th grade, and 8months later (Time 3, November 2023) when the students were in 5th grade." (p. 7) 2) "There was a total of 132 intervention sessions, and each session lasted about 30min." (p. 7) 3) "We should also note that our intervention lasted longer (>12months) than most vocabulary interventions (Rogde et al. 2019)." (p. 14) Detailed Analysis: Criterion T requires that outcomes be measured at least one full academic term (roughly 3-4 months) after the intervention begins. Here the baseline was in January 2022 and the immediate post-test was in March 2023, giving an interval of more than 12 months from intervention start to first outcome measurement, with a further delayed measurement in November 2023. The authors themselves state the intervention "lasted longer (>12months)." This far exceeds the one-term minimum. Criterion T is met because the interval from intervention start to outcome measurement spanned more than a year, well beyond a single academic term.
  • D

    Documented Control Group

    • The control group's size, ethnic composition, baseline scores, demographics, and business-as-usual condition are documented in the text and in Table 1.
    • "There were 227 students allocated to the control group, out of which 131 were Roma (Mage=121months) and 96 were non-Roma (Mage=116months)."
    • Relevant Quotes: 1) "There were 227 students allocated to the control group, out of which 131 were Roma (Mage=121months) and 96 were non-Roma (Mage=116months)." (p. 5) 2) "The students from the control group received instruction from their homeroom teachers. Since the class sizes were small to begin with (about 12 students in each class), the control group also benefited from small group instruction (i.e., 6 students in each group). The teachers were certified to teach elementary school students and had many years of teaching experience (M=27; Mdn=28; SD=11; Range=6-45). The teachers were expected to deliver business-as-usual language lessons, and to document the activities in a mini-lesson plan." (p. 10) 3) "Table 1 | Descriptive statistics of all measures at all time points." (p. 6, with dedicated Control / Control Roma / Control Non-Roma columns and baseline Time 1 means, SDs, and ranges) Detailed Analysis: Criterion D requires detailed documentation of the control group, including size, demographics, baseline performance, and the conditions they experienced. The paper reports the control group's size (n=227), its Roma/non-Roma composition, mean ages, and baseline (Time 1) scores across all measures in Table 1, along with socioeconomic indicators (parents' education, people per room, absences). It also documents that the control group received business-as-usual language lessons in small groups from experienced homeroom teachers, with activities recorded in mini-lesson plans. Criterion D is met because the control group's characteristics, size, baseline data, and condition are clearly documented in the text and Table 1.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomization occurred within classrooms at the student level, not among whole schools, so school-level randomization was not performed.
    • "The randomization was conducted within each classroom, and the sample was stratified according to their ethnicity (Roma vs. non-Roma)."
    • Relevant Quotes: 1) "The randomization was conducted within each classroom, and the sample was stratified according to their ethnicity (Roma vs. non-Roma). The participants were randomly assigned to one of the two treatments by a researcher who was not involved in the project using the following Stata command: randtreat, generate (treatment) strata (School Roma) misfits (wstrata)." (p. 5) 2) "We have identified 40 communities that fitted our profile and invited to participate in our experiment all registered students that attended 2nd grade the year prior to the beginning of the intervention." (p. 5) Detailed Analysis: Criterion S requires that entire schools (or equivalent institutional units) be the unit of randomization. In this study, schools were used only as a stratification variable, while the actual random assignment to experimental or control conditions was performed within each classroom at the individual-student level. Because both treatment and control students came from the same classrooms and schools, the randomization was not conducted among schools. Criterion S is not met because randomization was performed within classrooms at the student level rather than at the school level.
  • I

    Independent Conduct

    • The intervention was designed by the authors, who also performed the statistical analysis and drew the conclusions; only the test administration was carried out by independent, blinded assessors.
    • "The stories, the novel, and the intervention script were written by the first author, assisted by two research assistants."
    • Relevant Quotes: 1) "The stories, the novel, and the intervention script were written by the first author, assisted by two research assistants." (p. 8) 2) "We designed the intervention to focus explicitly on training intensively new content words and developing expressive language skills (Melby-Lervåg et al. 2020)." (p. 8) 3) "The participants were assessed by trained independent assessors who were not aware of the students' treatment." (p. 7) 4) "Three researchers listened selectively to the audio recordings of the intervention sessions, rated the quality of the interventions and provided the teachers with corrective feedback via phone." (p. 8) 5) "We analyzed the data using Mplus 8.9 (Muthén and Muthén 1998-2023)." (p. 10) Detailed Analysis: Criterion I requires that the study be conducted independently from the authors who designed the intervention, to reduce bias in implementation, measurement, analysis, and reporting. In this study the authors themselves designed the intervention (the first author wrote the stories, the novel, and the scripted guide), monitored fidelity, performed all statistical analyses, and formulated the conclusions. While the test administration was handled by trained independent assessors blind to condition, and randomization was done by a researcher not involved in the project, the core analysis and conclusions were produced by the same team that created and delivered the intervention. Unlike the standard's exception examples (where the provider did not participate in data collection, analysis, or the formulation of conclusions, or where the trial was led by an independent body), here the intervention designers remained responsible for analysis and conclusions. Criterion I is not met because the intervention designers also conducted the analysis and drew the study's conclusions, with only the blinded test administration being independent.
  • Y

    Year Duration

    • The intervention lasted more than 12 months, with outcomes measured immediately after and 8 months later, exceeding 75% of an academic year.
    • "our intervention lasted longer (>12months) than most vocabulary interventions"
    • Relevant Quotes: 1) "The students were assessed prior to the intervention (Time 1, January 2022) ... immediately after the intervention (Time 2, March 2023) ... and 8months later (Time 3, November 2023)." (p. 7) 2) "We should also note that our intervention lasted longer (>12months) than most vocabulary interventions (Rogde et al. 2019)." (p. 14) 3) "Thus, year-long systematic and intensive Tier-2 vocabulary instruction has the potential to lead to transfer effects." (p. 14) Detailed Analysis: Criterion Y requires that outcomes be measured at least 75% of one full academic year (roughly 9-10 months) after the intervention begins. Because criterion T is met, its prerequisite is satisfied. The intervention ran from January 2022 to March 2023, more than a full academic year, and outcomes were then tracked to November 2023. The interval from intervention start to measurement covers well over a full academic year. Criterion Y is met because the intervention and outcome tracking spanned more than 12 months, exceeding the year-duration requirement.
  • B

    Balanced Control Group

    • Both groups received small-group language instruction of comparable format and time, with the study deliberately matching group size so the manipulated variable was the type of instruction rather than added time or budget.
    • "we targeted small-sized classrooms which allowed us to provide small-group instruction to both experimental and control groups."
    • Relevant Quotes: 1) "A particularly strong methodological feature of our study is that in the selection process of our participants, we targeted small-sized classrooms which allowed us to provide small-group instruction to both experimental and control groups." (pp. 4-5) 2) "The students from the experimental group were removed from their classes and received the intervention in another classroom within the same school building. Each experimental group included about 6 students." (pp. 7-8) 3) "Since the class sizes were small to begin with (about 12 students in each class), the control group also benefited from small group instruction (i.e., 6 students in each group). ... The teachers were expected to deliver business-as-usual language lessons, and to document the activities in a mini-lesson plan." (p. 10) 4) "Another important feature of our study is that we compare the effects of a structured, researcher- developed language intervention delivered by trained local teachers to the effects of business-as-usual language lessons designed and taught by regular classroom teachers and also delivered in small groups." (p. 4) Detailed Analysis: Criterion B asks whether the control condition provides a comparable substitute for the intervention's inputs (time, materials, adult support). Applying the decision tree: the intervention group received a structured language comprehension program in small groups, but the control group did NOT remain on a bare business-as-usual large-class baseline. Instead, the authors deliberately designed the study so both experimental and control students received language instruction in small groups of about six students, explicitly to remove group size as a confound. The control group received business-as- usual language lessons of comparable format and time from experienced teachers. Thus the manipulated variable is the type/quality of language instruction (researcher- designed Tier 2 vocabulary program vs. teacher-designed lessons), which is integral to what the study tests, rather than an unmatched addition of time or budget. The educational time and small-group support were balanced across conditions. Criterion B is met because both the intervention and control groups received comparable small-group language instruction, with the study deliberately matching group size and time so that the difference tested is the instructional content itself.

• Level 3 Criteria

  • R

    Reproduced

    • This is described as the first such study in a Romanian-speaking, predominantly Roma sample, and no independent replication of this specific trial exists.
    • "This is the first experimental language intervention to include a sample of Romanian-speaking participants."
    • Relevant Quotes: 1) "This is the first experimental language intervention to include a sample of Romanian-speaking participants, and one of the very few such interventions conducted in a non-English-speaking country to test for far-transfer effects measured by standardized tests (Rogde et al. 2019)." (p. 4) 2) "A unique feature of our study is that it targets a sample consisting predominantly of Roma ethnic children." (p. 4) 3) "More studies are necessary to replicate our research and investigate the extent to which language interventions can lead to statistically significant positive effects on standardized vocabulary." (p. 14) Detailed Analysis: Criterion R requires that the specific study be independently replicated by a different research team in a different context and published in a peer-reviewed journal. The authors describe their trial as the first experimental language intervention in a Romanian- speaking, predominantly Roma sample, and explicitly call for future studies to replicate the work. While the broader field of language comprehension interventions has many related trials (e.g., Brinchmann et al. 2015; Esposito et al. 2024), none is an independent replication of this particular trial and design. An internet search across Wiley, ResearchGate, and Google Scholar for an independent reproduction of this specific 2026 trial returned no replication studies by other author teams as of the review date (10 July 2026); no such replication paper could be found, and none is fabricated here. Criterion R is not met because no independent replication of this specific study has been conducted.
  • A

    All-subject Exams

    • Only language and literacy outcomes were assessed; performance in other core subjects such as mathematics or science was not measured.
    • "we report the impact of a language comprehension intervention on the language, reading, and writing skills of upper elementary students"
    • Relevant Quotes: 1) "To address these issues, we report the impact of a language comprehension intervention on the language, reading, and writing skills of upper elementary students attending schools in rural, high-poverty communities from Romania." (p. 2) 2) "To what extent will a language comprehension intervention designed by researchers and delivered by trained teachers to children from high-poverty communities improve vocabulary, listening comprehension, reading comprehension, word reading, and written composition measured by standardized tests compared to small-group language activities designed and implemented by classroom teachers?" (p. 5) Detailed Analysis: Criterion A requires that the study measure impact on all main subjects taught (using standardized exams), not just the intervention subject, to detect unintended effects on non-target subjects. Although criterion E is met (standardized tests were used), all outcomes here fall within the language and literacy domain: vocabulary, listening comprehension, reading comprehension, word reading, and written composition. No standardized assessment of other core subjects such as mathematics or science was administered, and no exception justification for restricting to related subjects at this upper- elementary level is provided. Criterion A is not met because outcomes were measured only in language and literacy, not across all main school subjects.
  • G

    Graduation Tracking

    • Tracking stopped 8 months after the intervention (5th grade), with no follow-up through primary or later graduation and no follow-up publication found.
    • "the students were assessed immediately after the intervention (Time 2, March 2023) ... and 8months later (Time 3, November 2023) when the students were in 5th grade."
    • Relevant Quotes: 1) "The students were assessed prior to the intervention (Time 1, January 2022) ... immediately after the intervention (Time 2, March 2023) ... and 8months later (Time 3, November 2023) when the students were in 5th grade." (p. 7) 2) "These findings suggest that systematic language ... and vocabulary ... instruction can significantly improve the language and reading comprehension ... Importantly, these effects were sustained 8months later." (p. 13) Detailed Analysis: Criterion G requires that the study follow participants through to graduation from their educational stage to assess long-term impact. Even though criterion Y is met, the final measurement here occurred in November 2023, when students were in 5th grade, only 8 months after the intervention ended. There is no tracking through primary school graduation or beyond. An internet search for subsequent follow-up publications by the same authors (Dolean, Lervåg, Melby-Lervåg) that track this cohort to graduation returned none as of 10 July 2026; the only related forthcoming work referenced by the authors concerns homework and parental support ("Dolean et al. under review"), not graduation tracking of this cohort. Criterion G is not met because tracking ended 8 months post-intervention (5th grade) rather than continuing through graduation, and no follow-up graduation-tracking publication could be found.
  • P

    Pre-Registered

    • The study was pre-registered in the Registry of Efficacy and Effectiveness Studies before data collection, and the analyses were conducted in line with the pre-registration.
    • "To ensure transparency, we have pre-registered this study in the Registry of Efficacy and Effectiveness Studies (Registry ID: 10380.1v1) and we made data openly available."
    • Relevant Quotes: 1) "This pre-registered randomized controlled trial tests the impact of a language comprehension intervention ..." (Abstract, p. 1) 2) "To ensure transparency, we have pre-registered this study in the Registry of Efficacy and Effectiveness Studies (Registry ID: 10380.1v1) and we made data openly available (Dolean et al. 2025, September 18)." (p. 5) 3) "In line with the pre-registration, we used the post-test data (from both the immediate post-test and the delayed post-test) as outcome variables, a treatment dummy variable (control vs. intervention) to estimate the effects, and pre-test scores (baseline) and ethnicity as covariates." (p. 10) 4) "The study was preregistered in the Registry of Efficacy and Effectiveness Studies (Registry ID: 10380.1v1)." (Data Availability Statement, p. 16) Detailed Analysis: Criterion P requires that the study protocol be pre-registered before data collection begins, with hypotheses and planned analyses, and that the study adhere to the plan. The paper repeatedly and explicitly states that the study was "pre-registered" in the Registry of Efficacy and Effectiveness Studies (Registry ID: 10380.1v1), a recognized registry for education trials, and describes conducting the analyses "in line with the pre-registration" (including the pre-specified measurement-error correction). An attempt to open the REES registry entry (ID 10380.1v1) directly for an independent date check was blocked by the public registry's restricted access, so the exact registration date could not be confirmed externally. However, the version tag "v1", the explicit "pre-registered" designation, the documented adherence to the plan, and the fact that ethics approval (reference 13573/30.07.2018, dated 2018) and funding (2019) both predate the start of data collection (baseline in January 2022) are all consistent with registration before data collection, and no evidence suggests registration occurred afterward. Criterion P is met because the study was pre-registered in a recognized registry (Registry ID: 10380.1v1) and the reported analyses followed the pre-registered plan.