Improving College Readiness in Mathematics in the Context of a Comprehensive High School Reform

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • The study randomized individual students via admissions lotteries, not classes or schools.
    • "Participating schools agreed to use a lottery to select students, and the study is tracking outcomes for students randomly accepted into the program (treatment) and those not accepted who enrolled in some other school in the state, typically in the same district (control)." (p. 18)
    • Relevant Quotes: 1) "Participating schools agreed to use a lottery to select students, and the study is tracking outcomes for students randomly accepted into the program (treatment) and those not accepted who enrolled in some other school in the state, typically in the same district (control)." (p. 18) 2) "In this paper, we include results from analyses completed on a longitudinal sample of 1,434 treatment and 995 control students who applied to 19 early colleges for the fall of 2005 through 2009." (p. 19) Detailed Analysis: Criterion C requires class-level (or stronger school-level) randomization, unless the intervention is personal tutoring. Here, students apply to Early College High Schools and are assigned by lottery at the individual student level (treatment = accepted; control = not accepted). The paper does not describe random assignment of whole classes or whole schools. Even though assignment to different schools can reduce within-classroom contamination, the ERCT criterion is about the unit of randomization, and the paper explicitly states student-level lottery assignment. Final sentence explaining if criterion C is not met because randomization occurred at the student level, not at the class or school level.
  • E

    Exam-based Assessment

    • Outcomes are measured via course take-up and course passing, not via a standardized exam score.
    • "For each course, we report two outcomes: the percentage of all students in the grade in our sample who have taken the course (take-up) and the percentage of all students in the grade (not only those who have taken the course) who have passed the course (progress)." (p. 25)
    • Relevant Quotes: 1) "For each course, we report two outcomes: the percentage of all students in the grade in our sample who have taken the course (take-up) and the percentage of all students in the grade (not only those who have taken the course) who have passed the course (progress)." (p. 25) 2) "Students’ academic performance in mathematics was tracked through student-level data collected by the North Carolina Department of Public Instruction and housed at the North Carolina Education Research Data Center." (p. 20) Detailed Analysis: Criterion E requires standardized exam-based assessments that are widely recognized (not course grades or other non-standardized measures). The paper’s primary reported outcomes are course take-up and course progress (passing), which are not presented as standardized exam scores. While the paper uses state administrative data, it does not report a standardized test instrument as the primary outcome for the intervention impacts in grades 9-11, and it operationalizes success as passing courses. Final sentence explaining if criterion E is not met because the main outcomes are course-taking and course passing rather than standardized exam scores.
  • T

    Term Duration

    • Outcomes are measured across multiple grade levels (9th-11th), exceeding the one-term follow-up requirement.
    • "This mixed methods experimental study examined the impacts of the Early College High School model on students’ college readiness in mathematics as measured by their success in college preparatory mathematics courses from 9th through 11th grade..." (p. 2)
    • Relevant Quotes: 1) "This mixed methods experimental study examined the impacts of the Early College High School model on students’ college readiness in mathematics as measured by their success in college preparatory mathematics courses from 9th through 11th grade..." (p. 2) 2) "Table 3 shows the results for 9th grade, 10th grade, and 11th grade mathematics outcomes for the longitudinal sample of students." (p. 25) Detailed Analysis: Criterion T requires that outcomes be measured at least one academic term after the intervention begins. The paper reports outcomes for grades 9, 10, and 11 (a multi-year window), which necessarily includes at least one term of follow-up after program start. Final sentence explaining if criterion T is met because outcomes are measured over grades 9-11, far exceeding one term.
  • D

    Documented Control Group

    • The paper documents the control group’s composition and baseline characteristics using a detailed characteristics table and narrative.
    • "There were no statistically significant differences in any other background characteristics, as can be seen in Table 1." (p. 19)
    • Relevant Quotes: 1) "In this paper, we include results from analyses completed on a longitudinal sample of 1,434 treatment and 995 control students..." (p. 19) 2) "There were no statistically significant differences in any other background characteristics, as can be seen in Table 1." (p. 19) 3) "Table 1. Longitudinal Sample Characteristics, by Treatment Status" (pp. 19-20) Detailed Analysis: Criterion D requires that the control group be well-documented, including size and baseline characteristics. The paper specifies the control group size (995) and presents a dedicated table of characteristics by treatment status (Table 1), alongside narrative discussion of baseline differences. Final sentence explaining if criterion D is met because the paper provides both narrative and tabular documentation of the control group and baseline characteristics.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomization is done via student admission lotteries, not by randomizing schools.
    • "Participating schools agreed to use a lottery to select students..." (p. 18)
    • Relevant Quotes: 1) "Participating schools agreed to use a lottery to select students, and the study is tracking outcomes for students randomly accepted into the program (treatment) and those not accepted who enrolled in some other school in the state, typically in the same district (control)." (p. 18) Detailed Analysis: Criterion S requires school-level randomization (schools assigned to treatment vs control). The paper describes lotteries used to select students into Early College schools, which is student-level randomization. Final sentence explaining if criterion S is not met because the unit of randomization is students rather than schools.
  • I

    Independent Conduct

    • The intervention model is supported by a separate organization, while the study is conducted by researchers affiliated with research institutions.
    • "As implemented in North Carolina, U.S. (where this study was conducted), ECHS were guided by the six Design Principles developed by North Carolina New Schools, the non-profit organization supporting the implementation of this model..." (p. 7)
    • Relevant Quotes: 1) "As implemented in North Carolina, U.S. (where this study was conducted), ECHS were guided by the six Design Principles developed by North Carolina New Schools, the non-profit organization supporting the implementation of this model..." (p. 7) 2) "This material is based upon work supported by the Institute of Education Sciences under grant number #R305R060022." (p. 42) Detailed Analysis: Criterion I requires that the evaluation be conducted independently from the intervention designers. The paper identifies North Carolina New Schools as the non-profit supporting the model’s implementation, and the study is presented as an IES-funded research effort rather than an internal program self-evaluation by North Carolina New Schools. The paper does not explicitly state an arms-length governance structure, but the described roles separate the implementation-support organization (North Carolina New Schools) from the research study reporting results. Final sentence explaining if criterion I is met because the model’s implementation-support organization is distinct from the IES-funded study team reporting the evaluation.
  • Y

    Year Duration

    • The paper reports outcomes spanning grades 9-11, satisfying a one-year duration requirement.
    • "Table 3 shows the results for 9th grade, 10th grade, and 11th grade mathematics outcomes for the longitudinal sample of students." (p. 25)
    • Relevant Quotes: 1) "This mixed methods experimental study examined the impacts of the Early College High School model on students’ college readiness in mathematics as measured by their success in college preparatory mathematics courses from 9th through 11th grade..." (p. 2) 2) "Table 3 shows the results for 9th grade, 10th grade, and 11th grade mathematics outcomes for the longitudinal sample of students." (p. 25) Detailed Analysis: Criterion Y requires outcomes to be measured at least one academic year after intervention start. Reporting outcomes across grades 9-11 implies multi-year measurement well beyond one year. Final sentence explaining if criterion Y is met because outcomes are measured across multiple academic years (grades 9-11).
  • B

    Balanced Control Group

    • The intervention is explicitly described as a comprehensive reform model whose supports and resource-intensive elements are integral to what is being tested.
    • "ECHS is a comprehensive reform model. It is purposefully designed to include multiple elements that are believed to work best together to ensure students’ success in college preparatory and college level courses." (p. 39)
    • Relevant Quotes: 1) "In addition to extensive tutoring, often provided by ECHS teachers to struggling students, academic supports in ECHS were integrated into the school day, both during and outside of regular instructional time..." (p. 10) 2) "These Design Principles include: (1) a focus on college readiness... (3) a personalized learning environment with strong staff-student relationships and academic and social supports for students; ... and (6) use of time, resources and structures to support the other principles, including a small school size of less than 400 students and flexible scheduling." (p. 7) 3) "ECHS is a comprehensive reform model. It is purposefully designed to include multiple elements that are believed to work best together to ensure students’ success in college preparatory and college level courses." (p. 39) Detailed Analysis: Criterion B asks whether the intervention provides additional time/budget/ resources to treatment students without a comparable substitute in the control condition, unless the additional resources are integral to the treatment being tested. The paper describes resource-intensive elements (tutoring integrated into the school day, small school size, and broad supports) as part of the ECHS design principles and explicitly frames ECHS as a comprehensive model intended to work "together" as the tested package. In this framing, the additional resources are not incidental add-ons; they are core components of the intervention definition. Under the ERCT decision rule for criterion B, when additional resources are integral to the intervention being tested (a comprehensive school model), the control group can remain business-as-usual by design, and B can be marked met, provided the resource intensity is explicitly surfaced in the analysis (as done here). Final sentence explaining if criterion B is met because the paper defines the intervention as a comprehensive model whose additional supports and resource structures are integral to what is being tested.

• Level 3 Criteria

  • R

    Reproduced

    • Independent researchers (AIR) report a follow-up study of Early Colleges based on admission lotteries, providing replication evidence for the model.
    • "Following up on a previous impact study of Early Colleges (EC) based on retrospective admission lotteries, this study assessed longer-term impacts on students’ postsecondary outcomes with 4 more years of data." (Song et al., 2021, p. 1)
    • Relevant Quotes: 1) "Following up on a previous impact study of Early Colleges (EC) based on retrospective admission lotteries, this study assessed longer-term impacts on students’ postsecondary outcomes with 4 more years of data." (Song et al., 2021, p. 1) 2) "The study found that students who won EC admission lotteries were significantly more likely to enroll in college, enroll in 2-year colleges, complete a college degree, complete associate’s degrees or certificates, and complete bachelor’s degrees within 6 years after expected high school graduation than control students." (Song et al., 2021, p. 1) Detailed Analysis: Criterion R requires an independent reproduction by a different research team. This working paper’s authors are not part of AIR, while the cited follow-up study is authored by researchers at the American Institutes for Research (AIR) and reports lottery-based impact findings for Early Colleges. The AIR study uses admission lotteries and reports longer-term impacts, providing replication evidence (in a different sample and research team) that the Early College model yields positive impacts on key educational outcomes. Final sentence explaining if criterion R is met because an independent research team (AIR) reports lottery-based impact evidence consistent with the Early College model’s effects.
  • A

    All-subject Exams

    • The study reports mathematics course outcomes only and does not use standardized exam outcomes across all core subjects.
    • "This mixed methods experimental study examined the impacts of the study examined the impacts of the Early College High School model on students’ college readiness in mathematics..." (p. 2)
    • Relevant Quotes: 1) "This mixed methods experimental study examined the impacts of the study examined the impacts of the Early College High School model on students’ college readiness in mathematics..." (p. 2) 2) "Table 3 shows the results for 9th grade, 10th grade, and 11th grade mathematics outcomes for the longitudinal sample of students." (p. 25) Detailed Analysis: Criterion A requires exam-based assessment across all main subjects and depends on criterion E being met. Criterion E is not met because the main outcomes are course take-up and passing rather than standardized exam scores. Independently, the outcomes reported here are limited to mathematics course outcomes and do not cover all main subjects. Final sentence explaining if criterion A is not met because the study does not satisfy exam-based assessment and does not measure outcomes across all core subjects.
  • G

    Graduation Tracking

    • Follow-up publications by the same research program report high school graduation outcomes and longer-term outcomes after high school.
    • "The early college model in North Carolina (one of the states to most fully embrace the model) has been the subject of a 14-year longitudinal experimental study that has found a variety of positive impacts at both the high school and postsecondary levels." (Edmunds et al., 2020, p. 2)
    • Relevant Quotes: 1) "An 18-year longitudinal experimental study of the early college model established that six years after graduating from high school, more early college students earned postsecondary credentials than control students..." (p. 9) 2) "The early college model in North Carolina (one of the states to most fully embrace the model) has been the subject of a 14-year longitudinal experimental study that has found a variety of positive impacts at both the high school and postsecondary levels." (Edmunds et al., 2020, p. 2) 3) "They also had higher attendance, fewer suspensions, and were more likely to graduate from high school than students in the control group..." (Edmunds et al., 2020, p. 2) Detailed Analysis: Criterion G requires tracking participants through graduation, and it can be satisfied via subsequent publications by the same authors. The working paper itself points to an "18-year longitudinal experimental study" and explicitly references outcomes "six years after graduating from high school," indicating graduation-tracking exists in the broader study program. In Edmunds et al. (2020), which includes Arshavsky as a co-author, the authors describe a multi-year longitudinal experimental study and report that early college students were "more likely to graduate from high school" than control students. This directly supports that the research program tracked students to (and through) the graduation milestone. Because criterion Y is met in this paper and the broader study program includes graduation tracking evidence, criterion G is met. Final sentence explaining if criterion G is met because follow-up work by the same research program reports high school graduation outcomes and post-high-school follow-up.
  • P

    Pre-Registered

    • No public pre-registration record (with a registration date prior to study start) is identified in the paper or via registry searches.
    • Relevant Quotes: 1) "This material is based upon work supported by the Institute of Education Sciences under grant number #R305R060022." (p. 42) Detailed Analysis: Criterion P requires a publicly accessible pre-registered protocol with a registration date before data collection starts. The paper provides an IES grant number but does not provide any registry link or registration identifier (for example, OSF, AEA RCT Registry, ClinicalTrials.gov, or an equivalent). After searching for registrations linked to the study and grant number, no public pre-registration record that clearly matches this specific lottery-based study and predates study start was found. Final sentence explaining if criterion P is not met because no evidence of a pre-registered protocol is provided or discoverable in public registries.