Reading and Language Outcomes of a Multiyear Randomized Evaluation of Transitional Bilingual Education

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomization occurred at the individual student level within schools rather than at the class or school level, so this criterion is not met.
    • "The study used a randomized within-school design in which children entering kindergarten...were randomly assigned to TBE or SEI conditions." (p. 51)
    • Relevant Quotes: 1) "The study used a randomized within-school design in which children entering kindergarten, whose parents had agreed to study participation, were randomly assigned to TBE or SEI conditions." (p. 51) 2) "Parents had to sign a form allowing project staff to randomly assign their children to English or Spanish instruction." (p. 51) 3) "During their kindergarten year, children in the TBE classes were taught reading entirely in Spanish, using the Success for All Spanish kindergarten program Descubre Conmigo (Discover With Me)." (p. 51) Detailed Analysis: The paper explicitly states that randomization was performed at the individual child level within each participating school ("children entering kindergarten...were randomly assigned to TBE or SEI conditions"; "randomly assign their children to English or Spanish instruction"). This is a student-level randomization design, not randomization of intact classes or whole schools. Although students were subsequently grouped into separate "TBE classes" versus SEI classes for instruction (limiting contamination during actual delivery), the unit that was randomized was the individual student, not a pre-existing class or school. The exception for personal tutoring or one-to-one interventions does not apply here, since TBE and SEI are classroom-based group instructional programs, not one-to-one tutoring. No quote in the paper indicates that entire classes or schools were the units of random assignment. Criterion C is not met because the paper explicitly describes random assignment of individual students within schools to conditions, not randomization of entire classes or schools.
  • E

    Exam-based Assessment

    • The study used well-established standardized tests (PPVT/TVIP and Woodcock Reading/ Language scales), satisfying the exam-based assessment requirement.
    • "Children were pretested in the fall of kindergarten on the English Peabody Picture Vocabulary Test (PPVT) and its Spanish equivalent, the Test de Vocabulario en Imagenes Peabody (TVIP)." (p. 51)
    • Relevant Quotes: 1) "Children were pretested in the fall of kindergarten on the English Peabody Picture Vocabulary Test (PPVT) and its Spanish equivalent, the Test de Vocabulario en Imagenes Peabody (TVIP)." (p. 51) 2) "...in the second and third cohorts, the Spanish and English versions of the Woodcock Language Proficiency Battery Letter–Word Identification and Word Attack scales." (p. 52) 3) "Students in Grades 1 through 4 were given the same measures in addition to the Woodcock Passage Comprehension Scale." (p. 52) 4) "On both scales, the norm is 100, and the standard deviation is scaled to 15." (p. 52) Detailed Analysis: The outcome measures used - the PPVT/TVIP and the Woodcock Language Proficiency Battery subtests (Letter-Word Identification, Word Attack, Passage Comprehension) - are all well-established, nationally normed, standardized psychometric instruments, not assessments custom-built for this study. Their explicit norming (mean 100, SD 15) further confirms their standardized nature. This satisfies the ERCT requirement for standardized, exam-based assessment. Criterion E is met because the study used standardized, nationally normed instruments rather than custom-made tests.
  • T

    Term Duration

    • The study tracked outcomes annually over multiple years (up to 5 years), far exceeding the one-term minimum.
    • "The present article is the first to report findings of a randomized evaluation of TBE as long as 5 years." (p. 50)
    • Relevant Quotes: 1) "Three successive cohorts of students entering kindergarten in 2004, 2005, and 2006 were combined to increase sample size, which meant that by 2009, the end of the study, one cohort had reached the fourth grade..." (p. 50) 2) "...then followed for a period of up to 5 years. It compares early-exit TBE and structured English immersion (SEI)..." (p. 50) 3) "In the spring of their kindergarten year, children were individually assessed..." (p. 52) Detailed Analysis: Outcomes were tracked from the intervention's start in kindergarten through as many as fourth grade (up to 5 years for the earliest cohort), with annual spring assessments. This vastly exceeds the minimum one-term interval required by the ERCT standard. Since the stronger Year Duration criterion (Y) is separately met, the weaker Term Duration criterion is automatically satisfied as well. Criterion T is met because outcomes were tracked annually over a multi-year period, far exceeding the minimum one-term requirement.
  • D

    Documented Control Group

    • The comparison (SEI) group's demographics, baseline scores, and instructional content are thoroughly documented.
    • "Once randomly assigned to conditions, basic comparisons did not show any significant differences in home language...gender...or age..." (p. 51)
    • Relevant Quotes: 1) "Table 1 summarizes data on the demographic characteristics of the schools... All schools served very impoverished neighborhoods (76% to 100% free lunch)." (p. 50) 2) "Once randomly assigned to conditions, basic comparisons did not show any significant differences in home language, χ2(2, n = 357) = 0.91, p = .64; gender, χ2(1, n = 376) = 0.07, p = .79; or age, t(360) = 0.78, p = .435." (p. 51) 3) Table 3 "Outmover Pretests" reports PPVT and TVIP means, SDs, and sample sizes for both the transitional bilingual and English immersion groups at every grade level. (p. 53) 4) "SEI. The children assigned to the SEI treatment used the English Success for All programs, which have the instructional elements described above but use English only." (p. 52) Detailed Analysis: The paper provides extensive documentation of the comparison (SEI) group's composition, including school-level demographics (Table 1), formal pretest-equivalence checks on home language, gender, and age, and detailed baseline PPVT/TVIP scores by condition and grade (Table 3). The instructional content received by the comparison group is also explicitly described. This level of detail on baseline comparability and treatment received satisfies the requirement for a well- documented control/comparison group. Criterion D is met because the paper provides detailed baseline demographic and pretest data along with a clear description of what the comparison group received.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomization was conducted within each school at the student level rather than across whole schools, so this criterion is not met.
    • "The study used a randomized within-school design in which children entering kindergarten...were randomly assigned to TBE or SEI conditions." (p. 51)
    • Relevant Quotes: 1) "The study used a randomized within-school design in which children entering kindergarten...were randomly assigned to TBE or SEI conditions." (p. 51) 2) "Students in the TBE study attended 6 schools located in Los Angeles, California; Denver, Colorado; Albuquerque, New Mexico; St. Paul, Minnesota; Rockford, Illinois; and Alamo, Texas." (p. 50) Detailed Analysis: The paper explicitly describes a "within-school" randomization design, meaning students within each of the six participating schools were individually randomized to TBE or SEI conditions, rather than randomizing entire schools to one condition or the other. Every school in the sample contributed students to both conditions. This is the opposite of the school-level RCT design required by the stronger Level 2 "S" criterion, which requires random assignment of whole schools (or comparable institutional units) to conditions. Criterion S is not met because randomization occurred within each school at the individual student level, not between schools.
  • I

    Independent Conduct

    • The study was conducted by the same team that created and leads the Success for All program used in both conditions, with a declared conflict of interest and no independent evaluator.
    • "The first two authors listed are the creators of the Success for All program discussed in the article, and the last two are employees of the Success for All Foundation." (p. 57)
    • Relevant Quotes: 1) "The authors declared a potential conflict of interest (e.g. a financial relationship with the commercial organizations or products discussed in this article) as follows: The first two authors listed are the creators of the Success for All program discussed in the article, and the last two are employees of the Success for All Foundation. Any conflict of interest was resolved in the review process." (p. 57) 2) "ROBERT E. SLAVIN is currently Director of the Center for Research and Reform in Education at Johns Hopkins University...and Chairman of the Success for All Foundation." (p. 58) 3) "NANCY MADDEN is President and co-founder of the Success for All Foundation..." (p. 58) 4) "ANNE CHAMBERLAIN...worked at the Success for All Foundation in Baltimore, where she managed several large, randomized studies..." (p. 58) Detailed Analysis: The research team that designed, implemented, and evaluated this study is directly affiliated with the program (Success for All, in its Spanish and English variants) used to deliver both the TBE and SEI conditions. Two authors are the creators and current leaders of the Success for All Foundation (Chairman and President/co-founder respectively), and two others are or were employees of the Foundation who managed the study's large randomized trials. The paper's own Declaration of Conflicting Interests explicitly acknowledges this financial and professional relationship. There is no mention of an external, third- party organization independently conducting data collection or analysis, nor of blinded or independent assessors. Although the specific experimental contrast being tested is language of instruction (TBE vs. SEI) rather than the Success for All curriculum itself, the same team that created and promotes the underlying instructional program also designed, staffed, and analyzed this evaluation, with no independent oversight described. Criterion I is not met because the study was designed, conducted, and analyzed by the same team that created and administers the program used in both conditions, with a declared conflict of interest and no described independent evaluator.
  • Y

    Year Duration

    • Students were followed for up to five academic years, well beyond the one-year minimum duration required.
    • "The present article is the first to report findings of a randomized evaluation of TBE as long as 5 years." (p. 50)
    • Relevant Quotes: 1) "Three successive cohorts of students entering kindergarten in 2004, 2005, and 2006 were combined to increase sample size, which meant that by 2009, the end of the study, one cohort had reached the fourth grade, two had reached the third grade, and all three had reached the second grade." (p. 50) 2) "...then followed for a period of up to 5 years. It compares early-exit TBE and structured English immersion (SEI)..." (p. 50) 3) "The present article is the first to report findings of a randomized evaluation of TBE as long as 5 years." (p. 50) Detailed Analysis: Students were followed from kindergarten entry through as many as four subsequent grades (kindergarten through fourth grade for the earliest cohort), representing a tracking period of up to five academic years. This vastly exceeds the ERCT requirement of tracking outcomes for at least 75% of one academic year. Criterion Y is met because outcomes were tracked for up to five academic years after the intervention began, far exceeding the one-year minimum.
  • B

    Balanced Control Group

    • Both conditions received matched instructional time and an equivalent program (Success for All), differing only in language of instruction, so resources were balanced.
    • "In the present study, this design feature was accomplished by providing the Success for All program to children in both conditions." (p. 50)
    • Relevant Quotes: 1) "In the present study, this design feature was accomplished by providing the Success for All program to children in both conditions." (p. 50) 2) "This approach is identical to that of the English KinderCorner program used in the SEI schools, except that it uses materials written in Spanish that follow sequences of letter sounds, phonics, vocabulary, and concept development appropriate to the Spanish language." (p. 51) 3) "SEI. The children assigned to the SEI treatment used the English Success for All programs, which have the instructional elements described above but use English only." (p. 52) 4) "Little or no English was used during reading periods in TBE...Children participated in daily ESL sessions, as in the TBE condition, to help them build English oracy." (p. 52) Detailed Analysis: Applying the criterion B decision procedure, the first step is to determine whether the intervention condition received additional time or budget relative to the comparison condition. Here, both the TBE and SEI groups received the same underlying reading program (Success for All), with parallel Spanish and English versions that are structurally identical ("identical to...the English KinderCorner program...except that it uses materials written in Spanish"), delivered for the same amount of instructional time, and both groups additionally received daily ESL sessions. The manipulated variable is exclusively the language of instruction, not the amount of instructional time, materials, or budget provided, which were explicitly designed to be equivalent across conditions ("a critical design feature...is that of providing consistent high-quality curriculum and instruction to all children to ensure that language of instruction is the only factor that differentiates experimental groups"). Since no extra time or budget was given to either condition (EXTRA_RESOURCES_PRESENT = false per the decision tree), the resource-balance requirement is trivially satisfied at the first branch of the procedure. Criterion B is met because both conditions received an equivalent instructional program with matched instructional time and materials quality, differing only in language of instruction.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this specific randomized within-school TBE-vs-SEI study has been found in the paper or in subsequent literature; the paper describes itself as the first study of its kind.
    • "The present article is the first to report findings of a randomized evaluation of TBE as long as 5 years." (p. 50)
    • Relevant Quotes: 1) "The present article is the first to report findings of a randomized evaluation of TBE as long as 5 years." (p. 50) 2) "The findings of this randomized evaluation of bilingual education agree with those of the only previous long-term evaluations of bilingual education—namely, the matched evaluations by Maldonado (1977) and Ramirez et al. (1991)." (p. 56) Internet Search (Criterion R): The article's citation record (Semantic Scholar, DOI 10.3102/0162373711398127, 175 citing works as of the check date) was reviewed for any independent attempt to reproduce this exact randomized within-school TBE-vs-SEI comparison. Citing studies address related but distinct questions and designs (e.g., Valentino & Reardon, 2015, "Effectiveness of Four Instructional Programs Designed to Serve English Learners"; Polanco & Baker, 2018, "Transitional Bilingual Education and Two-Way Immersion Programs"; Umansky & Reardon, 2014, on reclassification patterns), none of which replicate this study's specific randomized within-school design comparing TBE to SEI using the Success for All curriculum. No independent replication by a different research team using a comparable randomized design was located. Detailed Analysis: The paper itself states that it is the first randomized, multiyear (up to 5-year) evaluation of TBE, and that only two prior long-term evaluations exist, both non-randomized (matched designs) rather than replications of this randomized study. The subsequent citation record likewise does not contain an independent replication of this precise randomized study by a different team. Criterion R is not met because no independent replication of this specific randomized, multiyear TBE evaluation is reported in the paper or identified via internet search of the subsequent literature.
  • A

    All-subject Exams

    • Only reading and language outcomes were assessed; no other core subjects (e.g., math, science) were measured, and no valid specialized-intervention exception applies.
    • "In the spring of their kindergarten year, children were individually assessed on the PPVT and the TVIP and...the Spanish and English versions of the Woodcock Language Proficiency Battery..." (p. 52)
    • Relevant Quotes: 1) "In the spring of their kindergarten year, children were individually assessed on the PPVT and the TVIP and, in the second and third cohorts, the Spanish and English versions of the Woodcock Language Proficiency Battery Letter–Word Identification and Word Attack scales." (p. 52) 2) "Students in Grades 1 through 4 were given the same measures in addition to the Woodcock Passage Comprehension Scale." (p. 52) 3) Title: "Reading and Language Outcomes of a Multiyear Randomized Evaluation of Transitional Bilingual Education" (p. 47) Detailed Analysis: All outcome measures in this study are confined to English and Spanish language and reading constructs (vocabulary, letter-word identification, passage comprehension, word attack/phonics). No measures of mathematics, science, social studies, or other core academic subjects are reported anywhere in the paper. While the study's exclusive focus on reading/language outcomes follows directly from its research question (the effect of language of instruction on reading), the ERCT standard's exception for narrow subject coverage applies specifically to "highly specialized interventions in upper secondary or vocational education," which does not describe this elementary-level, general-population reading/ language intervention. The exception therefore does not apply. Criterion A is not met because only English and Spanish reading/language outcomes were measured, with no assessment of other core subjects and no applicable specialized-intervention exception.
  • G

    Graduation Tracking

    • Tracking stopped at fourth grade for the most advanced cohort, well short of graduation, and no follow-up publications extending tracking were identified via internet search.
    • "Table 7 shows that by fourth grade (the highest grade reached only by the first cohort of students)..." (p. 54)
    • Relevant Quotes: 1) "Three successive cohorts of students entering kindergarten in 2004, 2005, and 2006 were combined to increase sample size, which meant that by 2009, the end of the study, one cohort had reached the fourth grade, two had reached the third grade, and all three had reached the second grade." (p. 50) 2) "Table 7 shows that by fourth grade (the highest grade reached only by the first cohort of students), there was no longer any statistically significant difference on any of the English language or reading outcomes..." (p. 54) Internet Search (Criterion G): The citation record for this article (Semantic Scholar, DOI 10.3102/0162373711398127, 175 citing works as of the check date) and searches for subsequent publications by Slavin, Madden, Calderón, Chamberlain, or Hennessy were reviewed for any follow-up study extending tracking of the same 2004/2005/2006 kindergarten cohorts into grade 5, grade 6, middle school, or through graduation. No such follow-up publication was found; related later work by these authors (e.g., Cheung & Slavin on reading programs for Spanish-dominant students; Calderón et al. on effective instruction for English learners) addresses other samples or is a review/synthesis rather than a longitudinal extension of this specific cohort. Detailed Analysis: The longest follow-up reported in this study reaches only the fourth grade for the earliest cohort, far short of graduation from elementary school, let alone secondary school. There is no discussion in the paper of any planned or completed follow-up beyond fourth grade, and the internet search for subsequent publications tracking these same cohorts through to graduation did not locate any such publication. Criterion G is not met because the longest tracking period reported reaches only fourth grade, with no evidence of follow-up to graduation found in the paper or in subsequent literature.
  • P

    Pre-Registered

    • No mention of a pre-registered protocol, trial registry, or registration date is found anywhere in the paper, and no registry record was located via internet search.
    • Relevant Quotes: No quotes were found. A thorough review of the paper's Method, Design, Measures, Analyses, and Discussion sections, as well as the Declaration of Conflicting Interests and Note, found no statement indicating that the study protocol, hypotheses, or planned analyses were registered on any public registry (e.g., ClinicalTrials.gov, ISRCTN, or an education-specific registry) prior to the start of data collection in 2004. Internet Search (Criterion P): The publisher landing page (SAGE Journals) and the CrossRef and OpenAlex metadata records for DOI 10.3102/0162373711398127 were checked for any linked trial-registration identifier; none was found. No registry entry for this study was located. Detailed Analysis: The paper was published in 2011, with data collection for the three cohorts beginning in 2004, 2005, and 2006 respectively - a period before pre-registration became a widely adopted or expected practice in educational RCTs (such norms and registries became common only from around the early-to-mid 2010s onward). The complete absence of any pre-registration statement in the paper, combined with the lack of any located registry record, is consistent with this historical context. Because no evidence of pre-registration is present, this criterion cannot be considered met. Criterion P is not met because the paper contains no reference to a pre-registered study protocol, registry, or registration date, and none was found via internet search.