Immigrant Integration in the United States: The Role of Adult English Language Training

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation was at the individual applicant level via an enrollment lottery for group ESOL classes, not at the class or school level, and the tutoring exception does not apply.
    • "While continuing students are guaranteed a spot the following semester, admission for all other students is determined by a random lottery conducted in January and August every year." (p. 6)
    • Relevant Quotes: 1) "We leverage the enrollment lottery of a publicly-funded adult English for Speakers of Other Languages (ESOL) program in Massachusetts to estimate the effects of English language training on voting behavior and employer-reported earnings." (p. 1, Abstract) 2) "While continuing students are guaranteed a spot the following semester, admission for all other students is determined by a random lottery conducted in January and August every year." (p. 6) 3) "After applications are submitted, FAESL+ staff members publicly draw lottery numbers and invite selected applicants to take a formal placement exam. Seats are allocated to students based on their level and time preference in the order in which their lottery number was drawn." (p. 6) 4) "Students are placed in classes based on their English proficiency level, with a mix of primary languages represented in each classroom." (p. 5) 5) "Our analytic sample includes 4,761 individuals (1,248 winners and 3,513 non-winners) who applied to this program between fall 2008 and spring 2016..." (p. 7) Detailed Analysis: The unit of randomisation is the individual applicant: lottery numbers are drawn per person and individual winners are offered seats. This is student-level randomisation, not class-level or school-level assignment. The ERCT exception for personal tutoring or one-to-one teaching does not apply because the intervention is classroom-based group ESOL instruction ("Students are placed in classes... with a mix of primary languages represented in each classroom"). Because all applicants compete for seats in the same program site, individual-level randomisation is used throughout. Contamination risk between winners and non-winners is admittedly low because non-winners are simply not admitted, but the criterion as written requires class-level or stronger randomisation, which is absent here. Criterion C is not met because randomisation occurred at the individual applicant level for group classes, with no class-level or school-level assignment and no applicable tutoring exception.
  • E

    Exam-based Assessment

    • Outcomes are voter registration, voter participation, and employer-reported earnings from administrative records; no standardised exam-based educational assessment was used as an outcome.
    • "We leverage the enrollment lottery... to estimate the effects of English language training on voting behavior and employer-reported earnings." (p. 1, Abstract)
    • Relevant Quotes: 1) "We leverage the enrollment lottery of a publicly-funded adult English for Speakers of Other Languages (ESOL) program in Massachusetts to estimate the effects of English language training on voting behavior and employer-reported earnings." (p. 1, Abstract) 2) "Massachusetts voter registration and participation data were purchased from NationBuilder." (p. 8) 3) "Employer-reported earnings data in Massachusetts were provided by the Massachusetts Department of Unemployment Assistance (MA DUA)." (p. 8) 4) "While we cannot observe program impacts on language skills directly, we expect these differences in program participation to meaningfully improve adult students' English language skills." (p. 10) Detailed Analysis: Criterion E requires that educational outcomes be measured with standard, widely recognised standardised exams. This study measures no educational achievement outcome at all: the outcomes are civic (voter registration and voting) and economic (employer-reported earnings), drawn from administrative records. The authors explicitly state they "cannot observe program impacts on language skills directly." A placement exam exists for admitted students but it is used for class placement, not as a study outcome. Since no standardised exam-based assessment of educational outcomes was used, the criterion fails. Criterion E is not met because the study measures voting and earnings outcomes rather than any standardised exam-based educational assessment.
  • T

    Term Duration

    • Outcomes were measured from two up to ten years after the first lottery (intervention start), far exceeding the one-term minimum tracking interval.
    • "Our sample includes over 4,700 individuals who applied to this program for the first time between fall 2008 and spring 2016, and we observe applicants for up to ten years after their first lottery attempt." (p. 3)
    • Relevant Quotes: 1) "Our sample includes over 4,700 individuals who applied to this program for the first time between fall 2008 and spring 2016, and we observe applicants for up to ten years after their first lottery attempt." (p. 3) 2) "A typical FAESL+ student attends classes for six hours per week over a 15-week fall or spring semester." (p. 5) 3) "Beginning two years after their first lottery application, individuals who are induced to enroll report $2,400 more in annual earnings, about 56% more than the control group." (p. 3) 4) "Over the three to ten years of post-lottery earning data we observe—the average applicant is observed for 6.9 years—FAESL+ enrollees report an additional 1.64 (0.67) quarters of earnings." (p. 12) Detailed Analysis: Criterion T requires that outcomes be measured at least one full academic term after the intervention begins. Here the intervention itself spans at least one 15-week semester (roughly one term), and outcome measurement extends from the lottery date through three to ten years afterwards, with the average applicant observed for 6.9 years. Voting outcomes cover elections from 2010 to 2016 and registration through December 2018; earnings cover January 2010 to September 2019. The interval from intervention start to outcome measurement therefore vastly exceeds one term. Criterion T is met because outcomes were tracked for years after the intervention began, well beyond one academic term.
  • D

    Documented Control Group

    • The control group (lottery non-winners) is documented in detail, including size, baseline demographics, baseline earnings, and the conditions they experienced.
    • "Our analytic sample includes 4,761 individuals (1,248 winners and 3,513 non-winners) who applied to this program between fall 2008 and spring 2016 and have non-missing date-of-birth and initial level information..." (p. 7)
    • Relevant Quotes: 1) "Our analytic sample includes 4,761 individuals (1,248 winners and 3,513 non-winners) who applied to this program between fall 2008 and spring 2016 and have non-missing date-of-birth and initial level information..." (p. 7) 2) "Students who do not win a spot in the FAESL+ program are encouraged to re-apply and are given information about other adult ESOL programs and volunteer-led classes in the area." (p. 6) 3) "About a quarter of applicants in our sample who do not win a spot in the program on their first lottery attempt ultimately enroll in the FAESL+ program in the future, 2.5 semesters later, on average." (p. 6) 4) "Table 3 reports results from a balance test for baseline covariates by lottery outcome. Column (1) presents the mean of each covariate for applicants who did not win their first lottery attempt." (p. 9) 5) "There are no significant differences in characteristics between the treatment and control groups." (p. 9) 6) "We observe less than 1% (42/4,761) of individuals in our lottery sample ever participating in another publicly-funded ESOL program in the state." (p. 6, footnote 8) Detailed Analysis: The control group of 3,513 lottery non-winners is thoroughly documented. Table 3 reports control-group means for age, gender, surname-based race/ethnicity indicators, and baseline quarterly earnings ($804), plus a joint F-test showing balance with the treatment group. The paper also documents what the control group experienced: referral to other programs, the option to re-apply (24.4% eventually enrolled), and near-zero participation in other publicly-funded ESOL programs. Crossover and non-compliance rates are quantified. This satisfies the requirement for detailed control-group documentation. Criterion D is met because the control group's size, baseline characteristics, and conditions are clearly documented with balance tests.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation was among individual applicants within a single adult education program, not among schools or institutional units.
    • "After applications are submitted, FAESL+ staff members publicly draw lottery numbers and invite selected applicants to take a formal placement exam." (p. 6)
    • Relevant Quotes: 1) "Framingham Adult ESL Plus (FAESL+) is one of the largest adult education programs in the state, enrolling over 750 students each year in Framingham, Massachusetts..." (p. 5) 2) "After applications are submitted, FAESL+ staff members publicly draw lottery numbers and invite selected applicants to take a formal placement exam." (p. 6) 3) "Specifically, we reconstruct eight years of twice-annual randomized enrollment lotteries for one of the largest adult ESOL programs in Massachusetts (Framingham Adult ESL Plus) to identify the impact of ESOL services on voter registration, voter participation, and employer-reported earnings." (p. 3) Detailed Analysis: Criterion S requires random assignment of whole schools or institutional units. This study concerns a single program site (FAESL+); randomisation occurred among individual applicants to that one program via lottery. No schools, centers, or sites were randomised to conditions. Criterion S is not met because randomisation was at the individual applicant level within one program, not at the school or institutional-unit level.
  • I

    Independent Conduct

    • The evaluation was conducted by university researchers who did not design or operate the FAESL+ program, using administrative lotteries run by program staff and outcome data from independent state agencies and vendors.
    • "Ours is the first study we are aware of that uses random assignment to study the impact of ESOL services delivered in a contemporary, business-as-usual setting—that is, operating under typical conditions with a general population of students and no additional interventions." (p. 5)
    • Relevant Quotes: 1) "Ours is the first study we are aware of that uses random assignment to study the impact of ESOL services delivered in a contemporary, business-as-usual setting—that is, operating under typical conditions with a general population of students and no additional interventions." (p. 5) 2) "After applications are submitted, FAESL+ staff members publicly draw lottery numbers and invite selected applicants to take a formal placement exam." (p. 6) 3) "We reconstruct lottery outcomes for individuals who applied to the FAESL+ program using three data sources: (1) statewide enrollment data... from the Massachusetts Department of Elementary and Secondary Education (MA DESE), (2) statewide waitlist records... and (3) administrative records and course lottery notes from the FAESL+ program." (p. 7) 4) "Massachusetts voter registration and participation data were purchased from NationBuilder." (p. 8) 5) "Employer-reported earnings data in Massachusetts were provided by the Massachusetts Department of Unemployment Assistance (MA DUA)." (p. 8) 6) "Correspondence can be addressed to the authors at: bheller@g.harvard.edu (Heller) and kslung@g.harvard.edu (Slungaard Mumma)." (p. 1, author note) Detailed Analysis: Criterion I requires that the study be conducted independently from the designers of the intervention. Here the intervention (the FAESL+ ESOL program and its curriculum) was designed and delivered by the Framingham Adult ESL Plus program, a pre-existing public adult education provider. The evaluators are Harvard University researchers (confirmed by their harvard.edu correspondence addresses) with no role in designing or delivering the program; they retrospectively reconstructed the program's own routine admission lotteries from administrative records. Outcome data come entirely from third parties: state education and unemployment agencies, a commercial voter-file vendor, and public voter files from other states, so the program provider had no role in outcome measurement or analysis. This mirrors the ERCT exception examples where an evaluation team distinct from the program developer satisfies independence. Criterion I is met because the Harvard research team independently evaluated a program it did not design, using lotteries run by program staff and outcomes from independent administrative sources.
  • Y

    Year Duration

    • Participants were tracked for up to ten years after the lottery (6.9 years on average), far exceeding 75% of an academic year from intervention start.
    • "Over the three to ten years of post-lottery earning data we observe—the average applicant is observed for 6.9 years—FAESL+ enrollees report an additional 1.64 (0.67) quarters of earnings." (p. 12)
    • Relevant Quotes: 1) "Our sample includes over 4,700 individuals who applied to this program for the first time between fall 2008 and spring 2016, and we observe applicants for up to ten years after their first lottery attempt." (p. 3) 2) "Over the three to ten years of post-lottery earning data we observe—the average applicant is observed for 6.9 years—FAESL+ enrollees report an additional 1.64 (0.67) quarters of earnings." (p. 12) 3) "Treatment-on-the-treated (TOT) estimates imply that individuals who are induced to enroll by winning their first lottery enroll for just over three semesters on average." (p. 10) 4) "Figure 2 plots coefficients estimating the effect of attending the FAESL+ program on annual earnings reported from five years before an individual's first lottery attempt through ten years after..." (p. 13) Detailed Analysis: Criterion Y requires outcome tracking covering at least 75% of an academic year from intervention start. Compliers enrolled for just over three semesters (about 1.5 academic years of instruction), and outcomes were measured annually for three to ten years after the first lottery, with an average observation window of 6.9 years. This comfortably exceeds the year-duration threshold, so both Y and, by implication, the weaker T are satisfied on duration grounds. Criterion Y is met because outcome tracking spanned multiple years from intervention start, far beyond 75% of one academic year.
  • B

    Balanced Control Group

    • The additional educational input (access to ESOL classes) is itself the treatment variable tested against a documented business-as-usual control, so the resource imbalance is by design.
    • "Ours is the first study we are aware of that uses random assignment to study the impact of ESOL services delivered in a contemporary, business-as-usual setting..." (p. 5)
    • Relevant Quotes: 1) "We leverage the enrollment lottery of a publicly-funded adult English for Speakers of Other Languages (ESOL) program in Massachusetts to estimate the effects of English language training on voting behavior and employer-reported earnings." (p. 1, Abstract) 2) "Ours is the first study we are aware of that uses random assignment to study the impact of ESOL services delivered in a contemporary, business-as-usual setting—that is, operating under typical conditions with a general population of students and no additional interventions." (p. 5) 3) "A typical FAESL+ student attends classes for six hours per week over a 15-week fall or spring semester." (p. 5) 4) "Students who do not win a spot in the FAESL+ program are encouraged to re-apply and are given information about other adult ESOL programs and volunteer-led classes in the area." (p. 6) 5) "If there are no more seats available, students whose lottery numbers were drawn are offered a seat in a weekly volunteer-led prep class and may join a teacher-led course if a seat becomes available in the first three weeks of class." (p. 6) Detailed Analysis: The intervention group received substantial additional educational time and resources: about six hours per week of ESOL instruction over one or more 15-week semesters (125 additional hours on average from winning the lottery). Control-group members received no matched educational input beyond referrals to other programs. Applying the ERCT decision tree: extra resources are present, but they are the treatment itself — the study's explicit purpose is to estimate the effect of receiving ESOL services (access to the program) versus not receiving them under business-as-usual conditions. The additional instruction is integral to the intervention being tested, exactly the RESOURCES_ARE_TREATMENT branch, so a business-as-usual control is appropriate by design. This additional resource (about 216 hours of instruction for compliers) is clearly the treatment variable, not a separable confounding add-on. Criterion B is met because the extra educational time is the explicit treatment variable tested against a business-as-usual control group.

• Level 3 Criteria

  • R

    Reproduced

    • The authors state this is the first random-assignment study of business-as-usual US adult ESOL services; internet searches of the paper's citation record (40 citing works, as of July 2026) found related international RCTs on refugee language training but no independent replication of this specific study.
    • "Ours is the first study we are aware of that uses random assignment to study the impact of ESOL services delivered in a contemporary, business-as-usual setting..." (p. 5)
    • Relevant Quotes: 1) "Ours is the first study we are aware of that uses random assignment to study the impact of ESOL services delivered in a contemporary, business-as-usual setting—that is, operating under typical conditions with a general population of students and no additional interventions." (p. 5) 2) "...few well-identified studies have considered the effects of adult language instruction, and all are based on programs outside the United States (e.g., Sarvimaki & Hamalainen, 2016; Lochman, Rapoport, & Speciale, 2019; Arendt et al., 2020)." (p. 4) 3) "Two studies of programs in the 1980s and 1990s that used random assignment to assign individuals to job training paired with adult education classes (including adult ESOL for some participants) found positive effects on earnings and employment, though results were not broken down separately for ESOL students..." (p. 5) Detailed Analysis: Criterion R requires independent replication of this specific study by a different team, in a different context, published in a peer-reviewed journal. The authors explicitly position their study as the first of its kind in the US. Related studies of adult language training exist (Finland, France, Denmark) but they evaluate different programs, populations, and designs, and predate this paper — they are not replications of this Massachusetts ESOL lottery study. Internet Research (2026): The paper was formally published as Heller & Slungaard Mumma (2023), American Economic Journal: Economic Policy 15(3), 407-437 (DOI 10.1257/pol.20210336). Its citation record (40 citing works per OpenAlex as of July 2026) was reviewed in full. Citing works evaluate language-training or integration programs for refugees/migrants in Denmark, Sweden, Norway, Finland, Amsterdam, and other contexts (e.g., Foged et al., "Language Training and Refugees' Integration," 2022; Dahlberg et al., "Labor market integration of refugees: RCT evidence from an early intervention program in Sweden," 2023), but each evaluates a distinct program, population, and design rather than reproducing the FAESL+ Massachusetts lottery evaluation. One citing work, "The Decline of Adult Education" (Heller, 2026), shares an author with the original study and is therefore not an independent replication. No independent, peer-reviewed replication of this specific study was identified. Criterion R is not met because no independent, peer-reviewed replication of this specific study exists, based on both the paper's own claims and a review of its citation record.
  • A

    All-subject Exams

    • Criterion E fails (no standardised exam outcomes at all), so the all-subject exam requirement automatically fails as well.
    • "While we cannot observe program impacts on language skills directly, we expect these differences in program participation to meaningfully improve adult students' English language skills." (p. 10)
    • Relevant Quotes: 1) "We leverage the enrollment lottery... to estimate the effects of English language training on voting behavior and employer-reported earnings." (p. 1, Abstract) 2) "While we cannot observe program impacts on language skills directly, we expect these differences in program participation to meaningfully improve adult students' English language skills." (p. 10) Detailed Analysis: Criterion A requires standardised exam-based assessment across all main subjects, and per the instructions it cannot be met when criterion E is not met. This study measures no academic subject with any exam: not even English proficiency (the intervention subject) is measured as an outcome, let alone other subjects. The adult ESOL context also has no multi-subject curriculum being assessed. Criterion A is not met because criterion E is not met and no subject was assessed with a standardised exam.
  • G

    Graduation Tracking

    • Long-run follow-up tracked earnings and voting, not educational progression; no publication tracking this cohort to program completion, credential attainment, or graduation was found in the paper or in internet research.
    • "While we do observe a handful of ESOL students enrolling in high school equivalency preparation classes at FAESL+, lottery winners are no more likely to earn a credential than non-winners..." (p. 5, footnote 5)
    • Relevant Quotes: 1) "Our sample includes over 4,700 individuals who applied to this program for the first time between fall 2008 and spring 2016, and we observe applicants for up to ten years after their first lottery attempt." (p. 3) 2) "While we do observe a handful of ESOL students enrolling in high school equivalency preparation classes at FAESL+, lottery winners are no more likely to earn a credential than non-winners, so we do not think differential access to high school equivalency preparation courses could explain the observed impacts on reported earnings or civic engagement." (p. 5, footnote 5) 3) "While we cannot observe program impacts on language skills directly, we expect these differences in program participation to meaningfully improve adult students' English language skills." (p. 10) Detailed Analysis: Criterion G requires following participants until graduation from their educational stage to evaluate long-term educational impacts. Although the study tracks individuals for up to ten years, the tracked outcomes are earnings and voting, not educational progression. Adult ESOL has no defined graduation milestone, and the paper does not track completion of proficiency levels, program graduation, or credential attainment as a study outcome (credentials are mentioned only in passing as a robustness note). Internet Research (2026): A search of works by Heller and Slungaard Mumma (via OpenAlex author records) found one directly related follow-up project: "The Effect of English Language Training on Health and Health Care Utilization" (Agarwal, Heller & Slungaard Mumma), registered in the AEA RCT Registry (ID AEARCTR-0014774, registered November 19, 2024; DOI 10.1257/rct.14774). This follow-up uses the same ESOL-lottery design to study health care utilization and health outcomes, not educational graduation, program completion, or credential attainment, so it does not satisfy criterion G. No publication tracking this cohort's graduation or program completion was identified. Criterion G is not met because participants were not tracked to graduation or completion of any educational stage, and no qualifying follow-up publication was found.
  • P

    Pre-Registered

    • No pre-registration of hypotheses, methods, or analysis plans is mentioned in the paper or its published journal version; the study retrospectively reconstructs lotteries held in 2008-2016.
    • Relevant Quotes: 1) "Specifically, we reconstruct eight years of twice-annual randomized enrollment lotteries for one of the largest adult ESOL programs in Massachusetts (Framingham Adult ESL Plus) to identify the impact of ESOL services on voter registration, voter participation, and employer-reported earnings." (p. 3) 2) "This study was approved by the Harvard University Institutional Review Board (IRB18-0521)." (p. 1, author note) Detailed Analysis: Criterion P requires that a full study protocol with hypotheses, methods, and planned analyses be registered before data collection began. The paper contains no mention of any registry (AEA RCT Registry, OSF, ClinicalTrials.gov, etc.) or of a pre-analysis plan. The design is a retrospective reconstruction of naturally occurring admission lotteries from fall 2008 to spring 2016, analysed years later (IRB approval in 2018), so registration before data collection did not occur and is not claimed. This is also structurally impossible for this study design, since the lotteries being analysed were run by the program long before the researchers began the evaluation. Internet Research (2026): The study's official publisher page (American Economic Journal: Economic Policy 15(3), 2023, DOI 10.1257/pol.20210336, aeaweb.org) was checked directly and lists no associated AEA RCT Registry ID for this article. A separate, later AEA RCT Registry entry (ID AEARCTR-0014774, registered November 19, 2024) exists for a different study by an overlapping author team ("The Effect of English Language Training on Health and Health Care Utilization"), which is not a pre-registration of this paper and postdates this paper's publication by over a year. Criterion P is not met because no pre-registered protocol is referenced for this study, the retrospective design precludes prospective registration, and no qualifying registry entry was found for this specific paper.