SEEDS of Early Learning: A Randomized Controlled Trial of an Early Literacy Teacher Professional Development Program

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Centers (preschool sites) were randomly assigned, which is class-level or stronger randomization.
    • "Centers were randomly assigned to receive the program or serve as a comparison control group." (p. 5)
    • Relevant Quotes: 1) "The remaining two-thirds included 26 preschool centers with 459 students - aged four or five years." (p. 5) 2) "Centers were randomly assigned to receive the program or serve as a comparison control group." (p. 5) Detailed Analysis: Criterion C requires randomization at the class level or stronger to avoid within-class contamination. Here, assignment occurs at the "preschool center" level, which is stronger than class-level assignment and therefore satisfies the requirement. Criterion C is met because entire centers (an institution-level unit) were randomly assigned to treatment versus control.
  • E

    Exam-based Assessment

    • The outcomes use established, widely used assessments (IGDI, FastBridge, HTKS) rather than a custom test created for this study.
    • "To assess student language, emergent literacy, and executive function outcomes, we used three measures:" (p. 5)
    • Relevant Quotes: 1) "To assess student language, emergent literacy, and executive function outcomes, we used three measures:" (p. 5) 2) "Individual Growth and Developmental Indicators (IGDI Version 1.0) which assesses children in fall and spring (with a winter option) in skills that predict future reading proficiency." (p. 5) 3) "FastBridge Learning: a valid, reliable measure of young children’s letter name and letter sound fluency (http://www.fastbridge.org/)" (p. 6) 4) "Head, Toes, Knees, and Shoulders assessment (HTKS): a task-based assessment that assesses inhibitory control, working memory, and attention focusing (Cameron Ponitz, McClelland, Matthews, & Morrison, 2008)." (p. 6) Detailed Analysis: Criterion E requires standardized, widely recognized assessments rather than a bespoke measure aligned to the intervention. The paper names three established measurement systems: IGDI, FastBridge Learning, and HTKS. The text also explicitly characterizes FastBridge as "valid, reliable" and positions IGDI as used "at scale," which is consistent with standardized use rather than a study-specific test. Criterion E is met because the study relies on established assessment instruments (IGDI, FastBridge, HTKS) rather than a custom-made exam.
  • T

    Term Duration

    • Outcomes are measured from fall to spring within a school year, which is at least one academic term (and in practice close to a full year).
    • "This assessment is typically conducted 1:1 between teacher and student. In the context of the RCT, external assessors conducted the assessments for the fall and spring assessment periods." (p. 5)
    • Relevant Quotes: 1) "Half the eligible centers were randomly assigned to receive SEEDS during the 2017-2018 school year (SEEDS Implementation Year 1, N=13), and the other half served as comparison controls (N=13)." (p. 5) 2) "Individual Growth and Developmental Indicators (IGDI Version 1.0) which assesses children in fall and spring (with a winter option) in skills that predict future reading proficiency." (p. 5) 3) "In the context of the RCT, external assessors conducted the assessments for the fall and spring assessment periods." (p. 5) Detailed Analysis: Criterion T requires at least one term between intervention start and the primary outcome measurement. The intervention is described as occurring during the "2017-2018 school year," and student measurement occurs in "fall and spring," which necessarily spans more than a single term. Criterion T is met because the study measures outcomes across fall-to-spring within a school year, exceeding the one-term minimum.
  • D

    Documented Control Group

    • The paper clearly defines a comparison control group (including sample sizes) and reports balance checks and baseline equivalence statements.
    • "Half the eligible centers were randomly assigned to receive SEEDS during the 2017-2018 school year (SEEDS Implementation Year 1, N=13), and the other half served as comparison controls (N=13)." (p. 5)
    • Relevant Quotes: 1) "Centers were randomly assigned to receive the program or serve as a comparison control group." (p. 5) 2) "Half the eligible centers were randomly assigned to receive SEEDS during the 2017-2018 school year (SEEDS Implementation Year 1, N=13), and the other half served as comparison controls (N=13)." (p. 5) 3) "Balance analyses found no statistically significant differences between the two experimental conditions in SEEDS Implementation Year 2 of the study there were sample differences in student race and ethnicity when compared to SEEDS Implementation Year 1." (p. 5) 4) "Centers were randomized to treatment and control conditions, with all teachers within a center assigned to the center condition. Groups were equivalent at baseline and statistical adjustments were made to account for differential attrition from the study in year 2 based on student race and ethnicity (See Appendix A)." (p. 7) Detailed Analysis: Criterion D requires that the control condition be clearly described and sufficiently documented to support interpretation. The paper explicitly defines a "comparison control group," provides the number of centers in each arm (N=13 / N=13), and reports that balance analyses were conducted. It also states that groups were "equivalent at baseline" and describes adjustment for differential attrition, indicating that baseline and group comparability were assessed and addressed. A limitation is that the paper’s sample characteristics table is organized by implementation year rather than by treatment vs. control. However, the control group identity, size, assignment mechanism, and baseline equivalence statements are explicit and usable for evaluation. Criterion D is met because the control group is clearly defined (who, how assigned, and N) and baseline equivalence/balance work is documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomization occurred at the preschool center level, which meets the school-level RCT requirement in this context.
    • "Centers were randomly assigned to receive the program or serve as a comparison control group." (p. 5)
    • Relevant Quotes: 1) "The remaining two-thirds included 26 preschool centers with 459 students - aged four or five years." (p. 5) 2) "Centers were randomly assigned to receive the program or serve as a comparison control group." (p. 5) Detailed Analysis: Criterion S requires randomization at the level of the educational institution implementing the intervention. In early childhood settings, "preschool centers" function as the institutional unit analogous to a school site. Randomizing 26 centers into treatment and control satisfies school-level assignment. Criterion S is met because whole preschool centers were randomized.
  • I

    Independent Conduct

    • The evaluation is conducted by NORC at the University of Chicago, while the program is implemented in collaboration with Kidango as the implementation partner.
    • "NORC at the University of Chicago designed and implemented an impact evaluation of the SEEDS of Learning (SEEDS) professional development (PD) program on behalf of the Kenneth Rainin Foundation, in collaboration with Kidango." (p. 1)
    • Relevant Quotes: 1) "NORC at the University of Chicago designed and implemented an impact evaluation of the SEEDS of Learning (SEEDS) professional development (PD) program on behalf of the Kenneth Rainin Foundation, in collaboration with Kidango." (p. 1) 2) "Kidango, a leading early learning organization, was the primary research partner." (p. 4) 3) "The study tested the impact of a new SEEDS Program implemented by Kidango (the implementation partner) ..." (p. 7) Detailed Analysis: Criterion I is about separation between the evaluator/research team and the intervention implementer/designer to reduce bias. The paper describes NORC (University of Chicago) as the group that "designed and implemented" the impact evaluation. It also explicitly labels Kidango as the "implementation partner" for the SEEDS program. This indicates a structural separation between evaluation and implementation roles. Criterion I is met because the evaluation is led by NORC, while Kidango is identified as the implementation partner for the intervention.
  • Y

    Year Duration

    • The study measures outcomes from fall to spring within a school year, and the intervention is described as occurring during the 2017-2018 school year.
    • "Half the eligible centers were randomly assigned to receive SEEDS during the 2017-2018 school year (SEEDS Implementation Year 1, N=13) ..." (p. 5)
    • Relevant Quotes: 1) "Half the eligible centers were randomly assigned to receive SEEDS during the 2017-2018 school year (SEEDS Implementation Year 1, N=13), and the other half served as comparison controls (N=13)." (p. 5) 2) "Individual Growth and Developmental Indicators (IGDI Version 1.0) which assesses children in fall and spring (with a winter option) in skills that predict future reading proficiency." (p. 5) 3) "In the context of the RCT, external assessors conducted the assessments for the fall and spring assessment periods." (p. 5) Detailed Analysis: Criterion Y requires outcome measurement at least one academic year after the intervention begins (or at minimum spanning the full academic cycle). The intervention is situated in the "2017-2018 school year," and outcomes are collected in fall and spring within that year. This aligns with a full-year academic cycle in typical US preschool calendars. Criterion Y is met because measurement spans fall-to-spring within the 2017-2018 school year, representing a full academic cycle.
  • B

    Balanced Control Group

    • Any additional resources (PD and coaching) are the intervention being tested, and the comparison group is business-as-usual with delayed treatment.
    • "It provides preschool teachers with content knowledge and pedagogical training via PD and coaching." (p. 2)
    • Relevant Quotes: 1) "It provides preschool teachers with content knowledge and pedagogical training via PD and coaching." (p. 2) 2) "Centers were randomly assigned to receive the program or serve as a comparison control group." (p. 5) 3) "The intervention had a staggered adoption such that the teachers in the control centers of the 2017-2018 school year (SEEDS Implementation Year 1) would receive SEEDS in the 2018-2019 school year (SEEDS Implementation Year 2)." (p. 5) 4) "In the present study, we evaluated the impacts of SEEDS on students through a cluster randomized controlled trial design with a delayed treatment for teachers in the control group." (p. 6) Detailed Analysis: Criterion B asks whether differences in time/budget/resources are balanced across conditions, unless the additional resource is explicitly the treatment variable being tested. Here, the intervention is professional development and coaching for teachers. That is inherently an "extra resource" relative to business-as-usual, but it is also the core treatment whose impact the study is designed to estimate. The paper also describes a delayed-treatment design, where control teachers receive SEEDS later, which is consistent with business-as-usual during the control period. Under the ERCT decision tree, this falls under the case where additional resources are integral to the intervention being tested; therefore a business-as-usual control without those resources is acceptable by design. Criterion B is met because the additional resources (PD/coaching) are the treatment variable, with a delayed-treatment control design.

• Level 3 Criteria

  • R

    Reproduced

    • No peer-reviewed, independent replication of this specific SEEDS PD RCT was found in the available literature.
    • Relevant Quotes: 1) "While earlier studies reflected the promise of the SEEDS program, the present study rigorously tested the causal impact of SEEDS on student learning." (p. 10) 2) "However, because SEEDS was confounded with other Reading Corps program features, it was impossible to isolate the effects of SEEDS." (p. 2) Detailed Analysis: Criterion R requires an independent replication by a different research team in a different context, ideally published in a peer-reviewed journal. The paper frames itself as an effort to isolate SEEDS effects that were previously confounded in other program contexts. After searching broadly for replication studies of this specific SEEDS PD RCT, no qualifying independent replication publication was identified. Criterion R is not met because no independent, peer-reviewed replication of this specific RCT was found.
  • A

    All-subject Exams

    • The study focuses on language/literacy and executive function outcomes and does not assess all core subjects (for example, mathematics).
    • "To assess student language, emergent literacy, and executive function outcomes, we used three measures:" (p. 5)
    • Relevant Quotes: 1) "To assess student language, emergent literacy, and executive function outcomes, we used three measures:" (p. 5) 2) "Student outcome measures included scales from two assessments: the Individual Growth and Development Indicators (IGDI) ... and the FastBridge assessment ..." (p. 6) Detailed Analysis: Criterion A requires exam-based assessment across all main subjects taught at that educational level to detect spillovers or tradeoffs. The listed outcomes are literacy/language measures (IGDI and FastBridge components) plus executive function (HTKS). There is no math achievement assessment or other core-subject coverage described in the outcome measures. Criterion A is not met because outcomes do not cover all core subjects and omit mathematics assessment.
  • G

    Graduation Tracking

    • The study reports outcomes within preschool years (fall-to-spring) and does not track students through graduation from the educational stage.
    • "Individual Growth and Developmental Indicators (IGDI Version 1.0) which assesses children in fall and spring (with a winter option) ..." (p. 5)
    • Relevant Quotes: 1) "The study occurred in San Francisco’s East Bay Area from 2017 to 2019." (p. 4) 2) "Individual Growth and Developmental Indicators (IGDI Version 1.0) which assesses children in fall and spring (with a winter option) in skills that predict future reading proficiency." (p. 5) 3) "In the context of the RCT, external assessors conducted the assessments for the fall and spring assessment periods." (p. 5) 4) "Interviews were conducted in Study Year 3 among both original treatment and control group teachers ..." (p. 7) Detailed Analysis: Criterion G requires tracking participants until graduation from the relevant educational stage. The paper documents outcome measurement within fall-to-spring assessment windows during implementation years and adds qualitative interviews in a later year. It does not describe following the student cohort to a graduation milestone (for example, kindergarten entry) and does not report later-stage outcomes. Additional searching for follow-up publications by the same authors that track the same cohort to graduation did not yield a qualifying paper with graduation-tracking results. Criterion G is not met because there is no evidence of student tracking to graduation from the educational stage.
  • P

    Pre-Registered

    • The paper does not cite a pre-registration record or registry identifier, and no verified pre-registration entry was found.
    • Relevant Quotes: 1) (No statement or registry identifier for pre-registration was found in the paper.) Detailed Analysis: Criterion P requires a publicly accessible pre-registered protocol with a registration date before data collection begins. The paper does not provide a registry name, link, identifier, or registration timing statement. A targeted search for a corresponding registration entry did not identify a verified pre-registration record for this study. Criterion P is not met because the study protocol is not shown to be pre-registered.