Replicating a Middle-School Belonging Intervention: Evidence from a Randomized Trial Within a New School District

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • The study randomized individual students within schools rather than assigning entire classes or schools to conditions.
    • "We conducted a preregistered student-level randomized controlled trial to assess the replicability of these findings..."
    • Relevant Quotes: 1) "We conducted a preregistered student-level randomized controlled trial to assess the replicability of these findings..." (p. 1) 2) "A total of 808 sixth-grade students were randomly assigned to either the treatment or control conditions within each school." (p. 5) 3) "We randomized only those students who were on the original rosters at the beginning of the school year..." (p. 6) Detailed Analysis: The ERCT standard requires that the study be conducted at the class level or school level to prevent contamination, unless it involves personal tutoring. The paper explicitly states that this was a "student-level" randomized controlled trial where students were assigned to conditions "within each school." While the intervention was administered within classrooms using packets, randomization was not done by class. Since the intervention involves writing exercises done in a shared classroom environment, student-level randomization does not meet the strict class-level requirement designed to minimize contamination. Final sentence explaining if criterion C is not met because the study used student-level randomization rather than class-level or school-level randomization.
  • E

    Exam-based Assessment

    • The study relied on school-assigned GPAs and grades rather than widely recognized standardized exam-based assessments.
    • "We calculated a student's GPA using administrative data provided by the school district."
    • Relevant Quotes: 1) "We calculated a student's GPA using administrative data provided by the school district. The letter grades were converted into a numerical score (i.e., A=4, B=3, C=2, D=1, F=0), and the average score was computed." (p. 6) 2) "The analysis focused on core academic courses only, including math, science, English language arts, and history/social studies." (p. 6) 3) "Outcomes... D and F grade counts, school attendance rates, and disciplinary referral and suspension counts." (p. 2) Detailed Analysis: The ERCT standard requires the use of standardized exam-based assessments that are widely recognised. This study uses Grade Point Averages (GPA) derived from teacher-assigned letter grades in core courses, as well as counts of D and F grades. These are school-based measures, not standardized external exams. There is no mention of state standardized tests (e.g., STAAR in Texas) being used as the outcome measure. Final sentence explaining if criterion E is not met because the outcomes measured were teacher-assigned grades (GPA) rather than standardized exams.
  • T

    Term Duration

    • Outcomes were measured over the final three quarters of the school year, which is significantly longer than one academic term.
    • "we created the GPA outcome measure using the post-treatment GPA by averaging students' GPAs from the final three quarters of the school year."
    • Relevant Quotes: 1) "The first exercise... was administered early in the school year (September)... The process was then repeated approximately 6 weeks later (November)..." (p. 5) 2) "we created the GPA outcome measure using the post-treatment GPA by averaging students' GPAs from the final three quarters of the school year. The first term GPA served as the baseline measure." (pp. 6-7) 3) "We measured student suspension as the total number of suspensions accumulated after the date of the second intervention administration..." (p. 7) Detailed Analysis: The intervention began in September and the second part occurred in November. The outcomes were measured based on the "final three quarters of the school year" (Terms 2, 3, and 4). A typical academic year runs until May or June. The interval from the start of the intervention (September) to the end of data collection (end of school year) exceeds one full academic term (3-4 months). Final sentence explaining if criterion T is met because the outcomes were tracked over the course of the remainder of the school year, exceeding the one-term duration requirement.
  • D

    Documented Control Group

    • The control group's demographics, baseline data, and specific activities (neutral writing exercises) are well-documented.
    • "In the control condition, students also completed two writing exercises of equal length, but the prompts focused on neutral middle school experiences..."
    • Relevant Quotes: 1) "In the control condition, students also completed two writing exercises of equal length, but the prompts focused on neutral middle school experiences unrelated to belonging uncertainty, including dealing with a noisy lunchroom and learning about politics." (p. 5) 2) "Table 1... Student Sample Characteristics and Baseline Equivalence... Control (N=310)... White 0.17... Black 0.37..." (p. 10) 3) "The packets given to treatment and control students had identical cover sheets..." (p. 5) Detailed Analysis: The paper provides a clear description of the control group's activities (neutral writing exercises) and documents their demographic characteristics and baseline performance in Table 1. The sample size and composition are explicitly detailed. Final sentence explaining if criterion D is met because the study provides comprehensive documentation of the control group's characteristics, size, and the specific "business as usual" or placebo activity they undertook.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomization occurred at the student level within schools, not at the school level.
    • "A total of 808 sixth-grade students were randomly assigned to either the treatment or control conditions within each school."
    • Relevant Quotes: 1) "We conducted a preregistered student-level randomized controlled trial..." (p. 1) 2) "A total of 808 sixth-grade students were randomly assigned to either the treatment or control conditions within each school." (p. 5) Detailed Analysis: The ERCT standard for Level 2 requires randomization at the school level. This study explicitly used student-level randomization within schools. School-level randomization was not employed. Final sentence explaining if criterion S is not met because the study randomized individual students rather than schools.
  • I

    Independent Conduct

    • The study was conducted by the authors themselves, who are also associated with the design of the intervention in previous studies.
    • "We conducted a preregistered student-level randomized controlled trial to assess the replicability of these findings..."
    • Relevant Quotes: 1) "Geoffrey D. Borman... Trisha H. Borman... So Jung Park Bo Zhu" (p. 1) 2) "We conducted a preregistered student-level randomized controlled trial..." (p. 1) 3) "Here, we utilize the same intervention materials and procedures as used in the two prior experiments..." (p. 1) Detailed Analysis: The authors of the paper are the ones who conducted the trial ("We conducted..."). They are also the designers/adapters of the intervention based on their prior work (Borman et al., 2019). There is no statement indicating that an independent third party conducted the data collection or analysis. Final sentence explaining if criterion I is not met because the authors themselves designed and conducted the study.
  • Y

    Year Duration

    • Outcomes were measured across the full academic year (Terms 2, 3, and 4) following the start of the intervention in September.
    • "we created the GPA outcome measure using the post-treatment GPA by averaging students' GPAs from the final three quarters of the school year."
    • Relevant Quotes: 1) "The first exercise... was administered early in the school year (September)..." (p. 5) 2) "post-treatment GPA by averaging students' GPAs from the final three quarters of the school year." (pp. 6-7) 3) "outcomes measured at least one full academic year after the intervention begins" (ERCT Standard) Detailed Analysis: The intervention started in September, at the beginning of the academic year. The outcomes were aggregated over Terms 2, 3, and 4, effectively covering the rest of the academic year (typically ending in May/June). The tracking covers the full academic cycle (~9 months) from the start of the intervention. Final sentence explaining if criterion Y is met because the study tracked outcomes through the end of the academic year, covering the required duration.
  • B

    Balanced Control Group

    • The control group received a placebo activity that matched the treatment group in terms of time and resources.
    • "In the control condition, students also completed two writing exercises of equal length..."
    • Relevant Quotes: 1) "In the control condition, students also completed two writing exercises of equal length..." (p. 5) 2) "The packets given to treatment and control students had identical cover sheets..." (p. 5) 3) "The intervention involved two 15-minute in-class exercises..." (p. 5) Detailed Analysis: The study does not utilize additional resources (budget, extra tutoring time) as the treatment variable; it tests the content of a writing exercise. The control group activity was matched in terms of time (two 15-minute exercises) and resources (identical packets, just different prompts). Both groups performed the task as "routine assignments within the sixth-grade English language arts classes." There was no imbalance in educational time or budget. Final sentence explaining if criterion B is met because the control group received an equivalent placebo intervention matching the time and materials of the treatment group.

• Level 3 Criteria

  • R

    Reproduced

    • The replication was conducted by the same authors as the original study, not by an independent research team.
    • "Previous research reported by Borman et al. (2019)... Here, we utilize the same intervention materials... We conducted a direct... replication"
    • Relevant Quotes: 1) "We conducted a direct... replication of a brief middle-school belonging intervention... Previous research reported by Borman et al. (2019)..." (p. 1) 2) "Geoffrey D. Borman... Trisha H. Borman..." (p. 1 - Authors) 3) "Borman et al. (2019) first studied the middle-school belonging intervention..." (p. 3) Detailed Analysis: The ERCT standard requires *independent* replication by a *different* research team. While this study is a replication of previous work, the lead author (Geoffrey D. Borman) is the same lead author of the original study (Borman et al., 2019). An independent search for replications of this specific middle-school belonging intervention by strictly external teams yielded no published results in peer-reviewed journals. Therefore, it does not constitute an independent replication. Final sentence explaining if criterion R is not met because the replication was conducted by the same primary researcher as the original study and no independent replications were found.
  • A

    All-subject Exams

    • Although the study covered core subjects, it relied on GPA rather than standardized exams, failing the prerequisite Criterion E.
    • "The analysis focused on core academic courses only, including math, science, English language arts, and history/social studies."
    • Relevant Quotes: 1) "The analysis focused on core academic courses only, including math, science, English language arts, and history/social studies." (p. 6) Detailed Analysis: The study assessed GPA across all main core subjects (math, science, ELA, history). However, the ERCT standard states that if Criterion E (Exam-based Assessment) is not met, then Criterion A is automatically not met. Since this study used teacher-assigned grades (GPA) and not standardized exams, it fails this criterion despite covering multiple subjects. Final sentence explaining if criterion A is not met because the outcome measures were not standardized exams (Criterion E not met).
  • G

    Graduation Tracking

    • The study tracked students only through the end of the sixth-grade year, not until graduation.
    • "we created the GPA outcome measure using the post-treatment GPA by averaging students' GPAs from the final three quarters of the school year."
    • Relevant Quotes: 1) "The study was conducted during the 2018-19 academic year..." (p. 5) 2) "post-treatment GPA by averaging students' GPAs from the final three quarters of the school year." (pp. 6-7) Detailed Analysis: The study tracked students only until the end of the sixth-grade academic year. The ERCT standard requires tracking until graduation (e.g., end of 8th grade for middle school or 12th grade for high school). A search for subsequent publications using the Killeen ISD sample (2018-19 cohort) authored by Borman et al. did not yield any follow-up studies tracking graduation rates, likely due to the null main effects reported in this study. Final sentence explaining if criterion G is not met because tracking ended at the conclusion of the single academic year, not upon graduation.
  • P

    Pre-Registered

    • The study protocol was preregistered on OSF before the start of the study.
    • "We preregistered the study prior to recruitment of KISD"
    • Relevant Quotes: 1) "We conducted a preregistered student-level randomized controlled trial..." (p. 1) 2) "the purpose of the current preregistered study (see https://osf.io/45nek/)..." (p. 4) 3) "We preregistered the study prior to recruitment of KISD..." (p. 6) Detailed Analysis: The paper explicitly states that the study was preregistered before the recruitment of the school district (i.e., before data collection). A link to the Open Science Framework (OSF) registration (https://osf.io/45nek/) is provided and confirms the registration details. Final sentence explaining if criterion P is met because the study provides evidence of preregistration prior to the start of data collection.