The Effects of an Instructional Intervention on Dual Language Development Among First-Grade Hispanic English-Learning Boys and Girls: A Two-Year Longitudinal Study

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Entire classrooms, not individual students within one classroom, were designated as treatment or control, satisfying class-level assignment, though no quote spells out the randomisation mechanics, making this the closest call among the 12 criteria.
    • "...a random sample of 70 students (35 girls and 35 boys) were drawn from treatment classrooms and another 70 (35 girls and 35 boys) from control bilingual classrooms..." (p. 89)
    • Relevant Quotes: 1) "This study is part of a larger experimental longitudinal randomized project targeting Hispanic second-language learners' English language and literacy acquisition in an urban school district in Southeast Texas." (p. 89) 2) "For a balanced design and to allow for missing data over time in the present analyses, a random sample of 70 students (35 girls and 35 boys) were drawn from treatment classrooms and another 70 (35 girls and 35 boys) from control bilingual classrooms, for a total of 140 students who began in kindergarten (2004 school year) and remained through Grade 1 (2005 school year) in their assigned group." (p. 89) 3) "All these students were placed in transitional bilingual programs from 10 schools and 12 classrooms." (p. 89) Detailed Analysis: This criterion was flagged for extra scrutiny because it was a point of disagreement among the initial blind rankings. The paper consistently refers to whole "treatment classrooms" and "control bilingual classrooms" as the pools from which the 70-plus-70 analytic sample was drawn, confirming that condition status was held at the classroom level: no quote anywhere in the paper suggests a treatment student and a control student ever shared one classroom, which is the specific contamination risk criterion C targets. However, on close re-reading, the word "random" in quote (2) modifies the SAMPLING of 140 analytic students out of the larger project population ("for a balanced design and to allow for missing data"), not the ASSIGNMENT of classrooms to treatment or control. The paper never quotes an explicit randomisation procedure (e.g., a lottery or random draw applied to classrooms or schools) explaining how a classroom came to be labeled "treatment" versus "control." The only support for randomised assignment (as opposed to some other allocation method) is the paper's own generic label of the larger project as "randomized," which this specific article does not unpack further; the randomisation mechanics may be documented in a separate, paywalled companion methods paper (e.g., Tong, Irby, Lara-Alecio, & Mathes, 2008) that could not be verified from this article's text alone, and per the verification procedure this criterion is judged from this paper's own text rather than by searching companion papers. Weighing both points: the paper gives strong, consistent evidence that condition was assigned at whole-classroom granularity (satisfying the anti-contamination purpose of the criterion), and it explicitly self-describes the parent project as "randomized," but stops short of quoting the exact randomisation procedure. On balance, the totality of indirect evidence supports "met," though this remains a closer call than most criteria in this paper and the underlying ambiguity legitimately explains why raters split on it. Criterion C is met because classrooms, not individual students within one classroom, held treatment/control status, and the paper self-identifies as part of a randomized project, even though no single quote spells out the randomisation mechanics.
  • E

    Exam-based Assessment

    • The study used the standardized, nationally normed WLPB-R and IPT-I instruments rather than custom-made tests.
    • "The Woodcock Language Proficiency Battery–Revised (WLPB-R...) is a standardized instrument assessing a broad language proficiency in oral, language, reading, and written language in English and Spanish." (p. 90)
    • Relevant Quotes: 1) "The Woodcock Language Proficiency Battery–Revised (WLPB-R; Woodcock, 1991; Woodcock & Munoz-Sandoval, 1995) is a standardized instrument assessing a broad language proficiency in oral, language, reading, and written language in English and Spanish." (p. 90) 2) "The English norm consists of a sample of 6,359 native English-speaking participants from age 2 to 99 years (3,245 in K–12)... The Spanish norm consists of a sample of 3,911 monolingual Spanish speakers from 22 countries..." (pp. 90-91) 3) "The IDEA Oral Language Proficiency Test (IPT-I; Ballard, Tighe, & Dalton, 1991) was designed to determine the level of oral language proficiency of students in Grades K–6 who are native speakers of other languages... The norming sample consisted of 1,054 students from eight states." (p. 91) Detailed Analysis: The study relies on two commercially published, nationally normed instruments: the Woodcock Language Proficiency Battery-Revised (WLPB-R) and the IDEA Oral Language Proficiency Test (IPT-I). Both are described with norming samples, reliability coefficients, and validity information drawn from published test manuals, confirming they are widely used, standardized assessments rather than instruments created by the authors specifically for this study. This matches the ERCT definition of a standardised, objective exam-based assessment. Criterion E is met because the outcome measures are established, standardized, nationally-normed instruments (WLPB-R and IPT-I), not custom researcher-made tests.
  • T

    Term Duration

    • The two-year span from kindergarten through the end of Grade 1 far exceeds the minimum one-term requirement.
    • "...a total of 140 students who began in kindergarten (2004 school year) and remained through Grade 1 (2005 school year) in their assigned group." (p. 89)
    • Relevant Quotes: 1) "...a total of 140 students who began in kindergarten (2004 school year) and remained through Grade 1 (2005 school year) in their assigned group." (p. 89) 2) "Data analyzed in the present study were collected between 2004 and 2006." (p. 91) 3) "Students were assessed at the outset of kindergarten and first grade, and the exit of first grade." (Abstract, p. 87) Detailed Analysis: The intervention and follow-up measurement span two full school years (kindergarten 2004 and Grade 1 2005), with outcome measurement occurring at the end of Grade 1, roughly two years after the intervention began. This vastly exceeds the minimum one-term (3-4 month) interval required by criterion T; in fact it satisfies the stronger Year Duration criterion (Y), which per the standard automatically satisfies T as well. Criterion T is met because the interval from intervention start (kindergarten) to final measurement (end of Grade 1) spans roughly two academic years, far exceeding one term.
  • D

    Documented Control Group

    • The control group's instructional composition, timing, curricular alignment, and baseline data are documented in detail.
    • "The control group (n = 70) were placed in typical practice bilingual classrooms with 80% Spanish and 20% English language distribution in kindergarten, with 70% and 30% in Grade 1." (p. 90)
    • Relevant Quotes: 1) "The control group (n = 70) were placed in typical practice bilingual classrooms with 80% Spanish and 20% English language distribution in kindergarten, with 70% and 30% in Grade 1. During both years, there was a 45-min separate ESL block daily (delivered in the morning or afternoon)." (p. 90) 2) "The Spanish instruction in these classrooms was identical to that in the treatment classrooms, and the control group differed in the following aspects: (a) more time allocated to instruction in Spanish, (b) lack of monitored separation of the two languages during instructional time, and (c) no specified clarifications of English by Spanish during ESL time." (p. 90) 3) "For the ESL component, the control group received the minimum state- and district-required amount of English instruction, and teachers had no specific structured lessons. However, all curricula were aligned with Texas state standards..." (p. 90) 4) "In a typical ESL lesson, the teacher first showed students the table of contents of a story, discussed the title and author, read and predicted the story, and conducted choral and partner reading... No training was provided by the research team to the control teachers." (p. 90) Detailed Analysis: The paper provides an extensive, specific description of the control condition: its language-distribution percentages by year and grade, ESL block duration, curricular alignment, typical lesson structure, and explicit statements about what the control group did NOT receive (structured lessons, research-team training, monitored language separation). Table 2 and Table 3 further supply baseline (Time 1) means and standard deviations for the control group broken out by gender, allowing readers to compare baseline comparability with the treatment group. Criterion D is met because the control condition is documented in detail, including its instructional composition, timing, and baseline performance data disaggregated by gender.

• Level 2 Criteria

  • S

    School-level RCT

    • No quote confirms randomisation at the whole- school level; the paper only supports classroom-level assignment.
    • "All these students were placed in transitional bilingual programs from 10 schools and 12 classrooms." (p. 89)
    • Relevant Quotes: 1) "All these students were placed in transitional bilingual programs from 10 schools and 12 classrooms." (p. 89) 2) "This study is part of a larger experimental longitudinal randomized project targeting Hispanic second-language learners' English language and literacy acquisition in an urban school district in Southeast Texas." (p. 89) Detailed Analysis: The paper never states that whole schools were randomly assigned wholesale to either the treatment or the control condition. Instead, it reports that the 140 sampled students came from "10 schools and 12 classrooms," a ratio that implies most schools contributed classrooms of only one type, but with only 12 classrooms spread across 10 schools it is likely (and not contradicted anywhere in the text) that at least some schools hosted both a treatment classroom and a control classroom, i.e., randomisation occurred at the classroom level within schools rather than at the school level itself. No quote anywhere in the paper states that "schools" (as opposed to classrooms) were the unit of random assignment. Criterion S is not met because the paper provides no quote confirming that entire schools (as opposed to classrooms within schools) were the unit of randomisation.
  • I

    Independent Conduct

    • The same research team designed the intervention, trained the treatment teachers, and monitored fidelity, with no mention of an independent evaluator conducting data collection or analysis.
    • "In addition, teachers in the experimental classrooms received ongoing professional training from the research team..." (p. 89)
    • Relevant Quotes: 1) "In addition, teachers in the experimental classrooms received ongoing professional training from the research team to review and practice upcoming lessons, reflect on student learning, and learn effective ESL strategies." (p. 89) 2) "Tier III was developed by the research team and delivered by highly trained paraprofessionals..." (p. 90) 3) "...Academic Oral Language in kindergarten and Academic Oral Language in Science in Grade 1, as developed by the research team." (p. 90) 4) "In general, the instruction was observed by the research team to be consistent across treatment classrooms." (p. 90) 5) "No training was provided by the research team to the control teachers." (p. 90) Detailed Analysis: The same research team that designed the instructional intervention (developing Tier II and Tier III components, writing the Academic Oral Language curriculum, and providing ongoing professional training and fidelity observation to treatment teachers) also appears to be the team running the evaluation, analyzing and reporting the results. There is no statement anywhere in the Method section identifying an external or third-party evaluator, blinded data collectors, or an independent agency responsible for data collection or analysis, distinct from the intervention designers. The description of "well- trained professionals" administering the measures does not specify independence from the intervention development team. Criterion I is not met because the paper gives no indication that data collection, analysis, or conclusions were carried out by a party independent of the team that designed and delivered the intervention.
  • Y

    Year Duration

    • The study tracked students across two full academic years, exceeding the year-duration threshold.
    • "...a total of 140 students who began in kindergarten (2004 school year) and remained through Grade 1 (2005 school year) in their assigned group." (p. 89)
    • Relevant Quotes: 1) "...a total of 140 students who began in kindergarten (2004 school year) and remained through Grade 1 (2005 school year) in their assigned group." (p. 89) 2) "Students were assessed at the outset of kindergarten and first grade, and the exit of first grade." (Abstract, p. 87) 3) "Data analyzed in the present study were collected between 2004 and 2006." (p. 91) Detailed Analysis: The intervention and outcome tracking cover two complete academic years (kindergarten 2004 school year through Grade 1, 2005 school year), with final outcome measurement at the spring/exit of Grade 1. This comfortably exceeds the 75%-of-one- academic-year threshold required by criterion Y; indeed it covers two full years rather than one. Criterion Y is met because the tracking period spans two full academic years (kindergarten through the end of Grade 1), well beyond the 75%-of-a-year minimum.
  • B

    Balanced Control Group

    • The additional instructional time and teacher training are integral, explicitly defined parts of the structured intervention being tested against a business-as-usual control.
    • "...instruction in kindergarten and Grade 1 was delivered as an English oracy intervention at a separate ESL block—focused and direct daily English teaching with amount of time increased for instruction: 75 min in kindergarten and 90 min in Grade 1." (p. 89)
    • Relevant Quotes: 1) "The treatment group (n = 70) received structured English instruction in a 70% Spanish and 30% English bilingual model in kindergarten and Grade 1... instruction in kindergarten and Grade 1 was delivered as an English oracy intervention at a separate ESL block—focused and direct daily English teaching with amount of time increased for instruction: 75 min in kindergarten and 90 min in Grade 1." (p. 89) 2) "In addition, teachers in the experimental classrooms received ongoing professional training from the research team..." (p. 89) 3) "During both years, there was a 45-min separate ESL block daily (delivered in the morning or afternoon)." (p. 90) 4) "For the ESL component, the control group received the minimum state- and district-required amount of English instruction, and teachers had no specific structured lessons... No training was provided by the research team to the control teachers." (p. 90) Detailed Analysis: Applying the criterion B decision procedure: extra resources are clearly present in the treatment group relative to control — additional daily ESL instructional minutes (75-90 min vs. 45 min), a specific structured tiered curriculum (Santillana Intensive English, storytelling and retelling, Academic Oral Language), and ongoing teacher professional development that the control teachers explicitly did not receive. The next question is whether this additional time and training is integral to the treatment variable being tested, rather than a separable confound. The paper's central research questions concern the effect of a specific "instructional intervention" that is explicitly defined, by design, as a structured, direct, increased-time ESL approach delivered by trained teachers ("focused and direct daily English teaching with amount of time increased for instruction"). The additional time and teacher training are not incidental extras layered onto an otherwise-identical lesson; they are the operational definition of the structured intervention itself, which the study explicitly contrasts against a "typical practice" business-as-usual control that received the state/district-mandated minimum. This matches the RESOURCES_ARE_TREATMENT branch of the current ERCT decision tree for criterion B: the study explicitly tests a structured, resourced instructional package against a business-as-usual control, so the control group's lower resourcing is by design rather than an unaddressed confound. Criterion B is met because the additional instructional time and teacher training in the treatment condition are integral, explicitly defined components of the structured instructional intervention being tested, and the control group is clearly documented as receiving the standard, business-as-usual level of ESL instruction.

• Level 3 Criteria

  • R

    Reproduced

    • Neither the paper's own citations nor an internet search located an independent replication by a different research team in a peer-reviewed journal; the closest large-scale validation study (ELLA-V) was still led by the same original Principal Investigators.
    • "These findings corroborate a previous study that showed that this same 2-year intervention accelerated English academic oral proficiency among a larger sample (Tong, Lara-Alecio, Irby, Mathes, & Kwok, 2008)." (p. 96)
    • Relevant Quotes: 1) "These findings corroborate a previous study that showed that this same 2-year intervention accelerated English academic oral proficiency among a larger sample (Tong, Lara-Alecio, Irby, Mathes, & Kwok, 2008)." (p. 96) 2) "Related research also suggested that students achieved greater gains in oral language proficiency when engaged in story-retelling activities (Quiros, 2008)." (p. 96) 3) "Tong, F., Irby, B. J., Lara-Alecio, R., & Mathes, P. (2008). The effectiveness of a three-year longitudinal randomized trial study on Hispanic bilingual students' English and Spanish acquisition." (Reference list, p. 98) Internet Search Findings: A search for independent replications of this specific kindergarten/Grade 1 randomized trial surfaced one closely related large-scale study: "English Language and Literacy Acquisition- Validation (ELLA-V) i3 Evaluation (Valid 22). Final Report" (Wolf, Latham, Armstrong, Ross, Laurenzano, Daniels, Eisinger, & Reilly; ERIC ED598313), produced by the Center for Research and Reform in Education (CRRE) at Johns Hopkins University. It describes "a multi-site cluster randomized trial designed to meet the What Works Clearinghouse (WWC) standards," run across ten Texas school districts from 2013-14 through 2016-17 under a federal Investing in Innovation (i3) validation grant. Critically, the report states the "Principal Investigators were Rafael Lara-Alecio, Beverly Irby, and Fuhui Tong" — the same authors who designed the original ELLA intervention. No independent replication by an unrelated research team, published in a peer-reviewed journal, was identified. Detailed Analysis: Criterion R requires independent replication by a different research team, in a different context, published in a peer-reviewed scientific journal. The only corroborating study cited within the paper (Tong, Lara-Alecio, Irby, Mathes, & Kwok, 2008) is by the same author team drawing on the same larger project. The ELLA-V validation study found via internet search, though larger in scale and evaluated on the ground by external CRRE staff, was still led by the same Principal Investigators who designed the original intervention, so it does not constitute a replication carried out by an independent research team. It is also a federal evaluation report rather than a peer-reviewed journal article, a second reason it does not meet this criterion. Criterion R is not met because the only supporting studies referenced or located are by the same research team/Principal Investigators using the same larger project, not an independent replication by a separate team published in a peer-reviewed journal.
  • A

    All-subject Exams

    • Only language and reading/literacy outcomes were measured; no assessment of mathematics, science, or social studies achievement is reported.
    • "We assessed male and female ELL students' expressive and receptive oral language skills... as well as reading skills..." (p. 95)
    • Relevant Quotes: 1) "We assessed male and female ELL students' expressive and receptive oral language skills: vocabulary (as measured by picture vocabulary and listening comprehension), and knowledge in phonology and syntax (as measured by IPT), as well as reading skills—reading identification and knowledge in phonic and structural analysis (as measured by letter-word identification and word attack), and reading comprehension (as measured by passage comprehension)." (p. 95) 2) "Tier I was the regular language arts, mathematics, science, and social studies instruction in Spanish in kindergarten and Grade 1." (p. 89-90) Detailed Analysis: All reported outcome measures (IPT, Picture Vocabulary, Listening Comprehension, Letter-Word Identification, Word Attack, Passage Comprehension) assess oral language and literacy/reading development only, in English and Spanish. Although the students also received regular instruction in mathematics, science, and social studies (Tier I), the study reports no standardized assessment results for these other core subjects; only language/literacy outcomes are measured and analyzed. The exception for all-subject exams (specialized upper-secondary/vocational programs) does not apply here, since this is a general K-1 elementary bilingual/ESL program, not a specialized vocational track, and no rationale is offered in the paper for omitting other core subjects from assessment. Criterion A is not met because the study measured only oral language and reading/literacy outcomes, not achievement across all main subjects (e.g., mathematics, science, social studies), and no qualifying exception is provided.
  • G

    Graduation Tracking

    • The authors explicitly state that outcome tracking and generalization stop at Grade 1, and neither the paper nor an internet search located any graduation-level follow-up of this specific cohort.
    • "...generalization cannot go beyond Grade 1." (p. 96)
    • Relevant Quotes: 1) "Results should be generalized with caution, and only to those districts that share similar characteristics with the district in the study, including students' demographics, instructional resources, and family-level SES. Further, generalization cannot go beyond Grade 1." (p. 96) 2) "At this stage, we are conducting a comprehensive analysis of this larger research project over 4 years so as to provide a better understanding regarding the causality of observed effective strategies on ELL students' language development and gender effect." (p. 96) 3) "Further research is needed to determine long-term language and literacy outcomes of bilingual students at higher grade levels in various program types." (p. 95-96) Internet Search Findings: A search of papers citing this article and of other publications by the same author team (Tong, Lara-Alecio, Irby, Mathes, and collaborators) did not surface any follow-up study that tracks this specific 2004-2006 kindergarten-through-Grade-1 cohort of 140 students through to graduation from elementary, middle, or high school. Related same-team publications identified (e.g., on professional development and teachers' instructional time, and on literacy/science integration for ELLs) concern the broader, multi-year Project ELLA research program and later program iterations (including the 2013-2017 ELLA-V validation study in different districts) rather than documented longitudinal graduation-tracking of the original 140-student sample described in this paper. No graduation-tracking follow-up of this specific cohort was found. Detailed Analysis: The authors explicitly state that "generalization cannot go beyond Grade 1," confirming that outcome tracking in this paper stops at the end of first grade, far short of graduation from the relevant educational stage. The mention of an ongoing "4-year" analysis is described as a future/ in-progress undertaking at the time of writing (2011) covering early elementary years, not graduation, and no follow-up publication tracking this specific cohort through graduation was identified either in the paper's own references or via internet search. Criterion G is not met because the study explicitly limits its tracking and generalization to the end of Grade 1, with no reported or located follow-up through graduation.
  • P

    Pre-Registered

    • No pre-registration statement, registry ID, or registration date is mentioned in the paper, and none was located in external trial registries.
    • Relevant Quotes: 1) No statement referencing a trial registry, registration ID, or a pre-registered study protocol appears anywhere in the Method, Results, or Discussion sections of the paper. Internet Search Findings: A search of common trial registries and evaluation clearinghouses did not locate a pre-registration entry for this specific study (data collected 2004-2006, published 2011). Registries commonly used for education RCTs today (e.g., the AEA RCT Registry, founded 2013) postdate this study's data collection by several years, consistent with the absence of any registration statement in the paper. Detailed Analysis: A full-text review of the Method section (Context, Participants, Instructional Intervention, Control Group, Measures, Data Collection and Plan of Analyses) and the Discussion/Limitations sections reveals no mention of pre-registration on any public registry (e.g., ClinicalTrials.gov, ISRCTN, AEA RCT Registry, or an equivalent), nor any registration date or protocol link. The paper predates common norms of trial pre-registration in education research (published 2011, data collection began 2004), and gives no indication that hypotheses, methods, or analysis plans were published prior to data collection. Criterion P is not met because no quote or reference to a pre-registered protocol, registry ID, or registration date is present anywhere in the paper, and no external registry record was located via internet search.