Effect of Full- Versus Half-Day Pre-K on Grade K–3 English Language Learner Designations

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Random assignment was at the individual applicant (family/ student) level via lotteries, not by whole class or whole school.
    • "the district used a lottery to randomly allocate offers among applicants."
    • Relevant Quotes: 1) "Because WPS initially had funding for only a limited num- ber of full-day classes—and wanted to learn whether full- day pre-K indeed yielded greater benefits for children and families before expanding it districtwide—the district used a lottery to randomly allocate offers among applicants." (p. 2) 2) "Across the three cohorts of students beginning pre-K over the course of this RCT (2016–17 through 2018–19), 399 applicants were randomly offered a full-day seat, whereas 396 applicants were randomly offered a half-day seat (business as usual)." (p. 2) 3) "This was done using a block randomized design, with randomized admissions lotteries conducted within families’ first-choice school site." (p. 2) Detailed Analysis: Criterion C requires that the RCT be conducted at the class level (or stronger), meaning whole classes (or whole schools) are randomized, to reduce contamination risks. The paper describes randomization as a lottery allocating offers "among applicants" (i.e., individual children/families), with lotteries conducted within families’ first-choice site. This is individual-level randomization, not class- or school-level assignment. The tutoring/personal-teaching exception does not apply because the intervention is a change in pre-K schedule dosage (full-day vs. half-day), not 1:1 tutoring. Criterion C is not met because the unit of randomization is individual applicants rather than classes or schools.
  • E

    Exam-based Assessment

    • ELL designation is based on the standardized WIDA-ACCESS Placement Test (W-APT) with nationally normed thresholds.
    • "If parents report a home language other than English, students undergo an English language proficiency assessment, the WIDA-ACCESS Placement Test (W-APT), covering listening, speaking, reading, and writing."
    • Relevant Quotes: 1) "If parents report a home language other than English, students undergo an English language proficiency assessment, the WIDA-ACCESS Placement Test (W-APT), covering listening, speaking, reading, and writing." (p. 3) 2) "The W-APT is used to establish students’ initial ELL designation based on students’ assessment scores in reference to nationally normed performance thresholds." (p. 3) 3) "A series of additional WIDA ACCESS assessments are used to reassess ELL designation status throughout grades K–12." (p. 3) Detailed Analysis: Criterion E requires outcomes be measured using standardized, widely recognized exam-based assessments rather than researcher-created tests. The paper states that ELL identification relies on the WIDA- ACCESS Placement Test (W-APT) and explicitly notes "nationally normed performance thresholds," with continued reassessment via WIDA ACCESS. The primary outcome analyzed (ELL designation in grades K–3) is determined by these standardized assessment procedures. Criterion E is met because the outcome is grounded in standardized WIDA assessments rather than a custom test.
  • T

    Term Duration

    • Outcomes were tracked from pre-K into kindergarten and through grades 1–3, far exceeding a one-term minimum.
    • "Finally, WPS and the Colorado Department of Education provided administrative data on students’ formal ELL designation status throughout grades K–3 for all study students who remained enrolled in the state of Colorado..."
    • Relevant Quotes: 1) "Across the three cohorts of students beginning pre-K over the course of this RCT (2016–17 through 2018–19)..." (p. 2) 2) "Finally, WPS and the Colorado Department of Education provided administrative data on students’ formal ELL desig- nation status throughout grades K–3 for all study students who remained enrolled in the state of Colorado..." (p. 4) 3) "In the context of this study, which follows students over 4 subsequent academic years following pre-K, we therefore would expect to observe some effects, should they exist." (p. 4) Detailed Analysis: Criterion T requires outcomes be measured at least one full academic term after the intervention begins. The study begins at pre-K entry and measures outcomes starting in kindergarten and continuing through grade 3. The paper also explicitly describes following students over "4 subsequent academic years following pre-K," which is well beyond one term. Criterion T is met because follow-up from pre-K start to outcome measurement extends well beyond one academic term.
  • D

    Documented Control Group

    • The control condition (half-day, business as usual) is clearly defined and baseline/outcome descriptives are reported by treatment assignment in multiple tables.
    • "Across the three cohorts of students beginning pre-K over the course of this RCT (2016–17 through 2018–19), 399 applicants were randomly offered a full-day seat, whereas 396 applicants were randomly offered a half-day seat (business as usual)."
    • Relevant Quotes: 1) "Across the three cohorts of students beginning pre-K over the course of this RCT (2016–17 through 2018–19), 399 applicants were randomly offered a full-day seat, whereas 396 applicants were randomly offered a half-day seat (business as usual)." (p. 2) 2) "In Table 1, we present information on the full RCT study sample, comprising 795 total study children (399 offered full- day pre-K, 396 offered half-day pre-K)." (p. 4) 3) "Table 2 further disaggregates the available data on students’ ELL designation status by treatment status..." (p. 5) 4) "Table 4 Baseline Covariate Balance Among Full RCT Sample" (p. 7) Detailed Analysis: Criterion D requires that the control group be well documented, including what it received and baseline characteristics. The paper explicitly defines the control as a half-day seat and labels it "business as usual." It provides sample sizes by assignment and extensive descriptives and balance checks (e.g., Table 1 demographics and baseline assessments, Table 2 outcome descriptives by grade and assignment, and Table 4 baseline balance). Criterion D is met because the control condition and its characteristics are clearly documented with detailed tables.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomization was conducted among applicants within school-site lotteries, not by randomly assigning whole schools to treatment or control.
    • "This was done using a block randomized design, with randomized admissions lotteries conducted within families’ first-choice school site."
    • Relevant Quotes: 1) "This was done using a block randomized design, with randomized admissions lotteries conducted within families’ first-choice school site." (p. 2) 2) "Across the three cohorts of students beginning pre-K over the course of this RCT (2016–17 through 2018–19), 399 applicants were randomly offered a full-day seat, whereas 396 applicants were randomly offered a half-day seat..." (p. 2) Detailed Analysis: Criterion S requires school-level randomization, meaning schools (or analogous sites) are randomly assigned to treatment vs. control. The paper describes lotteries run within a family’s first- choice school site to allocate individual offers of full-day vs. half-day seats. This is not random assignment of entire schools. Criterion S is not met because the unit of randomization is not the school.
  • I

    Independent Conduct

    • The full-day expansion was designed and implemented by the district, while the evaluation activities (e.g., baseline assessments and analysis) were carried out by a research team rather than the intervention designers.
    • "District leaders speculated that this shortfall might reflect a mismatch between the half-day schedule and families’ child- care needs and in 2016–17 began piloting some full-day classes to explore whether this approach would better meet commu- nity demand."
    • Relevant Quotes: 1) "District leaders speculated that this shortfall might reflect a mis-match between the half-day schedule and families’ childcare needs and in 2016–17 began piloting some full-day classes to explore whether this approach would better meet commu-nity demand." (p. 2) 2) "Because WPS initially had funding for only a limited num- ber of full-day classes—and wanted to learn whether full- day pre-K indeed yielded greater benefits for children and families before expanding it districtwide—the district used a lottery to randomly allocate offers among applicants." (p. 2) 3) "Second, the research team administered baseline assess- ments at the start of the pre-K year to all study children." (p. 4) Detailed Analysis: Criterion I requires that the evaluation be conducted independently from the intervention designers. Here, the paper attributes the creation and piloting of full- day pre-K to "District leaders" and describes the district’s use of a lottery to allocate offers. Separately, it describes evaluation work done by "the research team," including administering baseline assessments and conducting the analyses. While the paper does not use the explicit phrase "independent evaluation," the described division of roles indicates that the intervention was a district policy/program feature and the evaluation activities were conducted by a research team, i.e., not by the actors described as designing/deciding the policy shift. Criterion I is met because the paper documents that district leaders drove the intervention design/expansion while the research team conducted evaluation activities.
  • Y

    Year Duration

    • Outcomes were followed from pre-K entry through grade 3, which spans multiple academic years and exceeds the 75% of a year requirement.
    • "Finally, WPS and the Colorado Department of Education provided administrative data on students’ formal ELL designation status throughout grades K–3 for all study students who remained enrolled in the state of Colorado..."
    • Relevant Quotes: 1) "Across the three cohorts of students beginning pre-K over the course of this RCT (2016–17 through 2018–19)..." (p. 2) 2) "Finally, WPS and the Colorado Department of Education provided administrative data on students’ formal ELL desig- nation status throughout grades K–3 for all study students who remained enrolled in the state of Colorado..." (p. 4) 3) "In the context of this study, which follows students over 4 subsequent academic years following pre-K..." (p. 4) Detailed Analysis: Criterion Y requires that outcomes be measured at least 75% of an academic year after the intervention begins. The study begins at pre-K entry and measures outcomes in kindergarten through grade 3, which necessarily spans multiple academic years. The paper explicitly describes following students for "4 subsequent academic years following pre-K." Criterion Y is met because the follow-up period far exceeds one academic year.
  • B

    Balanced Control Group

    • Although full-day pre-K provides substantially more time than half-day, the paper explicitly defines this time/dosage increase as the treatment contrast and states content was the same aside from time.
    • "Half- and full-day class environments were not systematically different in terms of programmatic content (i.e., students in both treatment and control environments experienced the same instructional experiences, where the only difference was the amount of time spent in those environments)."
    • Relevant Quotes: 1) "In the years preceding this study, WPS offered only half- day pre-K, 3 hours per day (8:00–11:00 a.m. or 12:00–3:00 p.m.), 4 days a week (Monday–Thursday), for a total of 12 hours per week." (p. 2) 2) "Full-day classes operated from 8:30 a.m. to 3:30 p.m. (7 hours per day), 5 days per week, for a total of 35 hours per week." (p. 2) 3) "As a result, full-day classes provided more than dou- ble the amount of school hours as the half-day alternative." (p. 2) 4) "Half- and full-day class environments were not systematically different in terms of programmatic content (i.e., students in both treatment and control environments experienced the same instructional experiences, where the only difference was the amount of time spent in those environments)." (p. 2) 5) "Across the three cohorts of students beginning pre-K over the course of this RCT (2016–17 through 2018–19), 399 applicants were randomly offered a full-day seat, whereas 396 applicants were randomly offered a half-day seat (business as usual)." (p. 2) Detailed Analysis: Criterion B compares the nature, quantity, and quality of resources provided to treatment vs. control, and asks whether any extra time/budget is either balanced or is explicitly the treatment variable being tested. The intervention clearly adds substantial time (12 hours/week vs. 35 hours/week). This is not a negligible difference and is not matched in the control group. However, the paper explicitly frames the causal contrast as full-day vs. half-day "dosage" and states that the program content was not systematically different and that "the only difference" was time spent. Under ERCT’s Criterion B exception, when the additional resource (here, time/dosage) is the core treatment variable explicitly being tested against business-as-usual, imbalance in time is not a confound but the intended intervention contrast. Criterion B is met because the extra instructional time is the intentional treatment contrast and is explicitly described as the only systematic difference between conditions.

• Level 3 Criteria

  • R

    Reproduced

    • No peer-reviewed, independent replication of this specific study’s key outcome finding (K–3 ELL designation reductions) was identified.
    • "Future research should continue, where possible, to replicate this type of causal analysis in states or districts with varying pre-K program structures to probe the consistency of these effects."
    • Relevant Quotes: 1) "Future research should continue, where possible, to replicate this type of causal analysis in states or districts with varying pre-K program structures to probe the consistency of these effects." (p. 16) 2) "First, generalizability requires special attention. This study examined a specific population of students in a particular full- versus half-day pre-K treatment contrast, and results may not fully extend to other geographic regions, demographic compositions, or policy environments." (p. 16) Detailed Analysis: Criterion R requires that the study’s core finding be independently replicated by a different research team in a different context and published in a peer-reviewed outlet. The paper itself does not document an existing independent replication; instead, it highlights generalizability limits and calls for future replication work. An internet search conducted for this ERCT check did not identify a peer-reviewed, independent replication that explicitly attempts to reproduce this paper’s key outcome finding (effects on K–3 ELL designations from randomized offers of full- vs. half-day pre-K in this type of setting). Some related work exists on full- vs. half-day pre-K and on this Westminster RCT, but that is not the same as an independent replication of this paper’s focal outcome and analysis. Criterion R is not met because independent peer-reviewed replication of this study’s key finding was not identified.
  • A

    All-subject Exams

    • Although standardized language assessments underlie ELL designation, the study does not assess impacts across all core academic subjects using standardized exams.
    • "we estimate the intent-to-treat (ITT) effects of randomized offers of full-day versus half-day pre-K on students’ likelihood of subsequently receiving an ELL designation in grades K–3."
    • Relevant Quotes: 1) "This paper estimates the intent-to-treat (ITT) effects of randomized offers of full-day versus half-day pre-K on students’ likelihood of subsequently receiving an ELL desig- nation in grades K–3." (p. 2) 2) "If parents report a home language other than English, students undergo an English language proficiency assessment, the WIDA-ACCESS Placement Test (W-APT), covering listening, speaking, reading, and writing." (p. 3) Detailed Analysis: Criterion A requires standardized exam-based assessment across all main subjects taught at that level (and Criterion E must be met). Criterion E is met because WIDA assessments are standardized, but the outcome studied here is ELL designation, not broad achievement across core subjects. The paper does not indicate that standardized achievement outcomes in math, reading/language arts, science, and social studies were measured as part of this evaluation. It focuses on ELL designation outcomes. Criterion A is not met because the study does not assess impacts across all core subjects with standardized exams.
  • G

    Graduation Tracking

    • The paper reports follow-up only through grade 3, and no follow-up paper tracking this cohort through graduation was identified.
    • "Finally, WPS and the Colorado Department of Education provided administrative data on students’ formal ELL designation status throughout grades K–3 for all study students who remained enrolled in the state of Colorado..."
    • Relevant Quotes: 1) "Finally, WPS and the Colorado Department of Education provided administrative data on students’ formal ELL desig- nation status throughout grades K–3 for all study students who remained enrolled in the state of Colorado..." (p. 4) 2) "This study estimates the causal effect of randomized offers of full-day versus half-day pre-K on students’ likelihood of having English language learner (ELL) designations in early elementary grades." (p. 1) Additional Sources Checked (internet search): 3) "This follow-up study is for Phase 2... estimate longer-term effects on academic and behavioral outcomes through at least grade 3–5." (IES project description; page text) 4) "The current program will follow those study families to evaluate whether students who attend full-day pre-kindergarten exhibit stronger outcomes initially, whether those benefits persist as children move through K-12..." (CU Boulder outreach program description; page text) Detailed Analysis: Criterion G requires tracking participants until graduation (i.e., through a graduation milestone), and per the ERCT rule, this criterion can only be considered if Criterion Y is met (it is met here). The paper’s own outcome window is explicitly limited to early elementary grades (kindergarten through grade 3). The paper does not report tracking this cohort to any graduation point (end of elementary school, middle school, or high school). Following the instructions, an internet search was conducted for subsequent papers by the same authors that track this cohort to graduation. No peer-reviewed follow-up publication reporting graduation tracking for this cohort was identified. The only clearly relevant public materials found described planned or ongoing follow-up (e.g., through grades 3–5 and potentially through K–12), which is not the same as published graduation-tracking results. Criterion G is not met because graduation tracking is not reported, and no follow-up paper establishing graduation tracking was found.
  • P

    Pre-Registered

    • The paper provides a data/code repository link but does not provide a protocol preregistration entry (with a verifiable pre-data-collection date) for this study.
    • "The data and analysis files for this article can be found at https://doi.org/10.3886/E243548V1."
    • Relevant Quotes: 1) "Open Practices Statement The data and analysis files for this article can be found at https:// doi.org/10.3886/ E243548V1." (p. 17) Additional Sources Checked (internet search): 2) "Each summary includes... a link to the study’s pre- specified analysis plan." (Arnold Ventures grant summary page; page text) 3) "The study’s pre-specified analysis plan is linked here." (Arnold Ventures grant summary page; page text) Detailed Analysis: Criterion P requires that the study protocol be preregistered before data collection begins, with a registry link/ID and a registration date that can be verified as preceding data collection. The paper itself provides an open-data / analysis-files link, but it does not state that the study was preregistered, does not provide a registry entry (e.g., OSF Registration / AEA RCT Registry), and does not provide a registration date. Internet searching found references (outside the paper) to a "pre-specified analysis plan" for a related full- vs. half-day preschool RCT follow-up project. However, for ERCT Criterion P, this is insufficient without (a) a directly accessible preregistration record for this specific study and (b) a verifiable registration date that predates the start of data collection for the cohort(s) analyzed in this paper. Criterion P is not met because the paper does not document preregistration and a verifiable pre-data-collection preregistration record was not confirmed.