A national randomized controlled trial of the impact of public Montessori preschool at the end of kindergarten

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomization was at the individual child level (lottery seats), not at the class or school level, and no tutoring-style exception applies.
    • "The study had individual-level random assignment within lottery blocks in which children were clustered for analyses."
    • Relevant Quotes: 1) "We took advantage of random lotteries for admission to 24 oversubscribed public Montessori schools across the United States to examine the impact of being offered a seat (intention-to-treat or ITT design) at PK3 (3 y old) on end of kindergarten (5 to 6 y old) outcomes." (p. 2) 2) "The study had individual-level random assignment within lottery blocks in which children were clustered for analyses." (p. 2) Detailed Analysis: Criterion C requires the unit of randomization to be at least the class level (or school level, which also satisfies C), unless the intervention is inherently one-to-one tutoring or personal teaching. This study uses individual-level random assignment via school admission lotteries (children are offered or not offered a Montessori seat), explicitly stating "individual-level random assignment." The intervention is a preschool program model (Montessori) rather than a personal tutoring intervention, so the tutoring exception does not apply. Therefore, despite being a rigorous randomized design, the randomization unit is below class level, so Criterion C is not met. Final Summary: Criterion C is not met because assignment is individual, not class- or school-level.
  • E

    Exam-based Assessment

    • Outcomes were measured using widely used standardized assessments (e.g., Woodcock-Johnson, HTKS, digit span), not custom tests created for the study.
    • "Children were individually tested by professional data collectors in the fall of 2021 and for the three subsequent springs (2022, 2023, 2024) on reading, vocabulary, math, executive function (HTKS), memory (forward and backward digit span)..."
    • Relevant Quotes: 1) "Children were individually tested by professional data collectors in the fall of 2021 and for the three subsequent springs (2022, 2023, 2024) on reading, vocabulary, math, executive function (HTKS), memory (forward and backward digit span), persistence, social problem-solving, and social understanding (Theory of Mind Scale)..." (p. 9) 2) "Age is factored into the Woodcock-Johnson Z score, acting as a built-in control." (p. 2) Detailed Analysis: Criterion E requires exam-based assessment using standardized, validated instruments rather than study-authored tests. The paper describes outcome measurement via established standardized assessments and tasks commonly used in education and developmental research (Woodcock-Johnson, HTKS, digit span, Theory of Mind Scale), and it explicitly references the Woodcock-Johnson Z score norming ("age is factored into" the score). The outcome battery is not presented as a bespoke test constructed to match the Montessori curriculum, and the measures listed are recognizable, standardized instruments. Therefore, Criterion E is met based on the use of standardized assessments. Final Summary: Criterion E is met because the study uses established, standardized assessment instruments rather than custom tests.
  • T

    Term Duration

    • The study tracked outcomes from baseline in fall 2021 through spring 2024, far exceeding one academic term.
    • "Children were tested at baseline (starting PK3) in the fall of 2021 and each subsequent spring through kindergarten (2024)."
    • Relevant Quotes: 1) "Children were tested at baseline (starting PK3) in the fall of 2021 and each subsequent spring through kindergarten (2024)." (p. 2) 2) "Children were individually tested by professional data collectors in the fall of 2021 and for the three subsequent springs (2022, 2023, 2024) ..." (p. 9) Detailed Analysis: Criterion T requires outcomes to be measured at least one full academic term after the intervention begins. The paper states baseline testing in fall 2021 (start of PK3) and repeated spring assessments through 2024 (end of kindergarten). This is multiple years of follow-up, so the minimum term-duration requirement is comfortably satisfied. Final Summary: Criterion T is met because tracking spans multiple years from fall 2021 through spring 2024.
  • D

    Documented Control Group

    • The control group is clearly defined as lottery non-winners, with sample sizes and alternative preschool enrollment described.
    • "Among the included 24 schools, 588 children were consented, 242 who won a seat and 346 who did not."
    • Relevant Quotes: 1) "Among the included 24 schools, 588 children were consented, 242 who won a seat and 346 who did not." (p. 9) 2) "Nearly half, 44.8%, of the control group attended other public programs (e.g., district public schools) in PK3. The remainder attended private preschool programs (13.6%) or stayed home (6.1%) at PK3; for 56 children (16.2%) PK3 enrollment was unknown." (p. 9) Detailed Analysis: Criterion D requires a well-documented control group. The study defines controls as children not offered a Montessori seat via lottery, reports the control sample size (n = 346), and describes what control-group children did instead (other public programs, private preschool, stayed home, or unknown). This provides clear documentation of the control condition and the counterfactual experiences. Final Summary: Criterion D is met because the control group is explicitly defined and described, including alternative enrollments.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomization occurred within lotteries at the child level rather than by random assignment of schools to conditions.
    • "The study had individual-level random assignment within lottery blocks in which children were clustered for analyses."
    • Relevant Quotes: 1) "We took advantage of random lotteries for admission to 24 oversubscribed public Montessori schools..." (p. 2) 2) "The study had individual-level random assignment within lottery blocks in which children were clustered for analyses." (p. 2) Detailed Analysis: Criterion S requires school-level randomization (schools assigned to treatment or control). Here, schools are sites from which lottery offers are made, but assignment is explicitly "individual-level" within lottery blocks. Therefore, this is not a school-level cluster randomized trial. Final Summary: Criterion S is not met because assignment is not at the school level.
  • I

    Independent Conduct

    • The intervention studied was a business-as-usual Montessori program model not designed by the research team.
    • "The findings are especially notable because this is a field study of a business-as-usual intervention that was not designed by the researchers: Montessori preschool in public U.S. schools."
    • Relevant Quotes: 1) "The findings are especially notable because this is a field study of a business-as-usual intervention that was not designed by the researchers: Montessori preschool in public U.S. schools." (p. 9) 2) "The study was approved by the Institutional Review Board of the American Institutes for Research, Protocol 87236." (p. 2) Detailed Analysis: Criterion I focuses on independence from intervention designers to reduce bias. The paper explicitly states the program model evaluated was "not designed by the researchers" and describes it as a business-as-usual field setting. This directly supports the claim that the research team is not evaluating a researcher-created intervention, reducing the designer-led bias risk central to Criterion I. Final Summary: Criterion I is met because the paper explicitly states the intervention was not designed by the researchers.
  • Y

    Year Duration

    • Outcomes were tracked from fall 2021 through spring 2024, spanning multiple academic years.
    • "Children were tested at baseline (starting PK3) in the fall of 2021 and each subsequent spring through kindergarten (2024)."
    • Relevant Quotes: 1) "Children were tested at baseline (starting PK3) in the fall of 2021 and each subsequent spring through kindergarten (2024)." (p. 2) 2) "Children were individually tested ... in the fall of 2021 and for the three subsequent springs (2022, 2023, 2024) ..." (p. 9) Detailed Analysis: Criterion Y requires at least one full academic year between intervention start and outcome measurement. The study tracks children across PK3, PK4, and kindergarten, which is substantially longer than one academic year. Final Summary: Criterion Y is met because the follow-up window spans multiple academic years.
  • B

    Balanced Control Group

    • The study explicitly evaluates the real-world Montessori program package (including its resource structure) against typical alternatives, making resource differences part of the treatment definition.
    • "Further, a cost analysis suggested three years of public Montessori preschool costs less per child than traditional programs, largely due to Montessori having higher child:teacher ratios in PK3 and PK4."
    • Relevant Quotes: 1) "Further, a cost analysis suggested three years of public Montessori preschool costs less per child than traditional programs, largely due to Montessori having higher child:teacher ratios in PK3 and PK4." (p. 1) 2) "The cost difference is primarily attributable to the intentionally higher child-to-adult ratios in Montessori classrooms at PK3 and PK4..." (p. 6) 3) "We took advantage of random lotteries for admission to 24 oversubscribed public Montessori schools..." (p. 2) Detailed Analysis: Criterion B asks whether time and resources are balanced across conditions unless the resource difference is itself integral to what is being tested. This study evaluates the impact of being offered a Montessori seat in public schools as they operate in practice. The paper emphasizes that Montessori has a distinct resource structure (notably child-to-adult ratios and overall costs), and it treats that structure as part of the program model being evaluated, including a dedicated cost analysis. Because the estimand is the real-world "Montessori seat offer" package relative to typical alternatives (the counterfactual experiences children would otherwise have), the relevant resource differences are not an incidental confound added on top of the pedagogy, but part of the intervention definition. Under the ERCT Criterion B exception logic, this supports marking B as met. Final Summary: Criterion B is met because the study explicitly evaluates a full program package where resource differences are integral to the treatment being tested.

• Level 3 Criteria

  • R

    Reproduced

    • The paper reports that key findings replicate across multiple Montessori preschool RCTs, including at least one independent RCT in another context.
    • "This finding has replicated across all four recent RCTs examining the impact of public Montessori preschool..."
    • Relevant Quotes: 1) "This finding has replicated across all four recent RCTs examining the impact of public Montessori preschool..." (p. 6) External Replication Evidence (other authors): 2) Courtier, Gardes, & Noveck (2021) report that disadvantaged preschoolers "were randomly assigned" to Montessori vs. conventional classrooms. (Courtier et al., 2021, Child Development) Detailed Analysis: Criterion R requires independent replication by other researchers in other contexts. The paper itself states that the end-of-kindergarten early reading pattern "has replicated" across four recent Montessori preschool RCTs, indicating a replication literature for the intervention outcome pattern. Among the cited RCTs, at least one is led by a different author team in a different national context (e.g., Courtier et al., 2021, France), satisfying the "other authors, other context" requirement for replication. Therefore, Criterion R is met based on peer-reviewed randomized trials by other researchers that reproduce similar outcome patterns for Montessori preschool. Final Summary: Criterion R is met because independent Montessori preschool RCTs exist and the paper notes replication across multiple RCTs.
  • A

    All-subject Exams

    • The study does not assess effects across all core school subjects via standardized exam batteries; it reports a selected set of academic and nonacademic outcomes.
    • "In addition to examining a range of academic (reading, vocabulary, and math) and nonacademic ... outcomes..."
    • Relevant Quotes: 1) "In addition to examining a range of academic (reading, vocabulary, and math) and nonacademic (executive function, memory, theory of mind, social problem-solving, and persistence) outcomes..." (p. 2) Detailed Analysis: Criterion A requires standardized exam-based assessment across all main subjects, to detect broad spillovers and displacement. The paper measures several academic domains (reading, vocabulary, math) plus multiple nonacademic outcomes, but it does not present a comprehensive all-subject standardized exam battery covering the full set of core school subjects. Given the criterion's intent, the outcome coverage is partial rather than all-subject. Final Summary: Criterion A is not met because the study does not measure a full all-subject standardized exam set.
  • G

    Graduation Tracking

    • Outcomes are tracked only through the end of kindergarten, and the authors explicitly note that longer-run impacts are unknown.
    • "We followed from age 3 through kindergarten..."
    • Relevant Quotes: 1) "We followed from age 3 through kindergarten..." (p. 1) 2) "Results Are End-of-Kindergarten. We do not know whether the study children will continue to do well in their later school years." (p. 9) Detailed Analysis: Criterion G requires tracking participants through graduation. Although the study is longitudinal across preschool years, it ends at kindergarten and explicitly states that later outcomes are unknown. There is no evidence in the paper of tracking through later grades, let alone to graduation. Final Summary: Criterion G is not met because follow-up ends at kindergarten, not at graduation.
  • P

    Pre-Registered

    • The paper reports registration on REES but does not provide a registration date relative to the study start, so pre-registration before data collection cannot be verified here.
    • "The analysis plan for this study was registered prior to examining PK3 and later spring outcome data at the Registry of Efficacy and Effectiveness Studies in Education (REES) (#15183.1v4)..."
    • Relevant Quotes: 1) "The analysis plan for this study was registered prior to examining PK3 and later spring outcome data at the Registry of Efficacy and Effectiveness Studies in Education (REES) (#15183.1v4)..." (p. 2) 2) "Children were tested at baseline (starting PK3) in the fall of 2021..." (p. 2) Detailed Analysis: Criterion P requires that the full study protocol be pre-registered before the study begins (before data collection starts). The paper states that an analysis plan was registered on REES and provides the registry identifier. However, the wording specifies registration prior to examining PK3 and later spring outcome data, not explicitly prior to study start or prior to baseline data collection (fall 2021). The paper does not include the REES registration date or a clear statement that registration occurred before baseline data collection began. Without a verifiable registry timestamp in the paper itself, the strict ERCT timing requirement cannot be confirmed from the provided materials. Therefore, this criterion is marked not met. Final Summary: Criterion P is not met because pre-registration before study start cannot be verified from the paper text provided.