Online Tutoring, School Performance, and School-to-Work Transitions: Evidence from a Randomized Controlled Trial

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Student-level randomization is acceptable because the intervention is one-to-one tutoring.
    • "This paper investigates the effects of a one-on-one online tutoring program in Germany on both academic performance and school-to-work transitions." (p. 2)
    • Relevant Quotes: 1) "This paper investigates the effects of a one-on-one online tutoring program in Germany on both academic performance and school-to-work transitions." (p. 2) 2) "The students were then pair-wise randomly assigned to either a treatment or a control group." (p. 11) Detailed Analysis: Criterion C normally requires class-level (or school-level) randomization, but the ERCT standard provides an explicit exception for personal tutoring or one-to-one teaching. The paper states the intervention is "one-on-one online tutoring" and then describes student-level randomization ("pair-wise randomly assigned"). Under the ERCT tutoring exception, student-level randomization is acceptable because contamination within a classroom is not the core risk for a one-to-one tutoring offer. Criterion C is met because this is a one-to-one tutoring intervention and randomization is clearly described.
  • E

    Exam-based Assessment

    • Outcomes rely on self-reported grades rather than standardized exam-based assessments.
    • "All grades are self-reported by the young participants and relate to the final grades from school years 2021/2022 and 2022/2023, respectively." (p. 22)
    • Relevant Quotes: 1) "All grades are self-reported by the young participants and relate to the final grades from school years 2021/2022 and 2022/2023, respectively." (p. 22) 2) "As German data restrictions did not allow us to obtain grades from administrative sources, we cannot test the reliability of the students' self-reports using an external source." (p. 41) Detailed Analysis: Criterion E requires standardized exam-based assessments that are widely recognized and not bespoke to the study. The paper measures academic outcomes using school grades, but explicitly notes these grades are self-reported and that administrative grade records could not be obtained. Self-reported grades are not the same as standardized, externally administered exam measures, and they also introduce additional reporting noise and potential bias. Criterion E is not met because outcomes are not based on standardized exam-based assessments.
  • T

    Term Duration

    • The first follow-up occurs about six months after randomization, exceeding one academic term.
    • "After randomization, we conducted two follow-up surveys. The first, six months after randomization, coincided with the start of the new school year; the second took place around 18 months later." (p. 3)
    • Relevant Quotes: 1) "After randomization, we conducted two follow-up surveys. The first, six months after randomization, coincided with the start of the new school year; the second took place around 18 months later." (p. 3) 2) "The first survey (follow-up I) took place in October and November 2022, during the new school year, after the treatment group had been invited to participate in Lern-Fair." (p. 14) Detailed Analysis: Criterion T requires that outcomes are measured at least one academic term (about 3-4 months) after the intervention begins. The paper explicitly states the first follow-up is "six months after randomization" and places it in Oct-Nov 2022 after the invitation. Six months is comfortably longer than one term, so the timing requirement is satisfied. Criterion T is met because the first outcome measurement is about six months after randomization.
  • D

    Documented Control Group

    • The paper clearly describes the control condition and reports baseline balance between groups.
    • "Students randomized into the control group received neither targeted information about the free online tutoring offered by Lern-Fair, nor easy and preferential access to the online tutoring program." (p. 12)
    • Relevant Quotes: 1) "Students randomized into the control group received neither targeted information about the free online tutoring offered by Lern-Fair, nor easy and preferential access to the online tutoring program." (p. 12) 2) "Table 1: Balance test" (p. 13) Detailed Analysis: Criterion D requires a well-documented control group, including what the control group received and evidence that groups are comparable at baseline. The paper explicitly defines the control group as receiving no targeted information and no preferential access to the program. It also provides a baseline balance table (Table 1) comparing treatment and control on sociodemographic and school-related variables. This combination provides both a clear control condition and baseline comparability documentation. Criterion D is met because the control condition and baseline balance are documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomization occurred at the student level, not at the school level.
    • "The students were then pair-wise randomly assigned to either a treatment or a control group." (p. 11)
    • Relevant Quotes: 1) "The students were then pair-wise randomly assigned to either a treatment or a control group." (p. 11) Detailed Analysis: Criterion S requires school-level randomization (assignment among schools or equivalent institutions). The paper describes pair-wise randomization of students within the study sample, not assignment of schools. Therefore, the unit of randomization does not satisfy the school-level RCT requirement. Criterion S is not met because randomization is not conducted at the school level.
  • I

    Independent Conduct

    • The tutoring program is run by an external nonprofit, while the evaluation is conducted by academic researchers.
    • "The nonprofit organization Lern-Fair provides the online tutoring evaluated in this study." (p. 7)
    • Relevant Quotes: 1) "The nonprofit organization Lern-Fair provides the online tutoring evaluated in this study." (p. 7) 2) "The program, operated by the nonprofit organization Lern-Fair, uses a web-based platform to connect university student volunteers with disadvantaged secondary school students in need of academic support." (p. 3) Detailed Analysis: Criterion I requires that the evaluation is conducted independently of the intervention designer/operator. The paper states Lern-Fair (a nonprofit) provides and operates the tutoring program. The study is written by university and research-institute affiliated authors, indicating the evaluators are not the program operator. The paper also acknowledges help from Lern-Fair members, consistent with an external provider supporting implementation while researchers evaluate. Criterion I is met because the program is operated by Lern-Fair and the evaluation is conducted by external academic researchers.
  • Y

    Year Duration

    • Outcomes are tracked from early 2022 to late 2023, exceeding one academic year.
    • "The second survey (follow-up II) took place in November and December 2023." (p. 14)
    • Relevant Quotes: 1) "Our baseline survey, which was part of the CoDu panel survey, took place between February and March 2022..." (p. 11) 2) "The second survey (follow-up II) took place in November and December 2023." (p. 14) Detailed Analysis: Criterion Y requires outcome measurement at least one full academic year after the intervention begins. The study baseline and randomization occur in Feb-Mar 2022, and follow-up II occurs in Nov-Dec 2023, a span of roughly 20-22 months. This exceeds a full academic year in any standard school calendar. Criterion Y is met because follow-up II is conducted well beyond one year after baseline/randomization.
  • B

    Balanced Control Group

    • Any additional resources (free tutoring access) are the treatment variable, so a business-as-usual control is acceptable under ERCT.
    • "Students randomized into the control group received neither targeted information about the free online tutoring offered by Lern-Fair, nor easy and preferential access to the online tutoring program." (p. 12)
    • Relevant Quotes: 1) "Students assigned to the treatment group received an invitation to participate in the Lern-Fair online tutoring program." (p. 11) 2) "Students randomized into the control group received neither targeted information about the free online tutoring offered by Lern-Fair, nor easy and preferential access to the online tutoring program." (p. 12) 3) "This paper investigates the effects of a one-on-one online tutoring program in Germany on both academic performance and school-to-work transitions." (p. 2) Detailed Analysis: Criterion B checks whether intervention and control conditions are balanced in time and resources, unless the additional resources are explicitly the treatment variable being tested. Here, the core treatment is access to free, one-to-one online tutoring (an additional educational resource). The study's stated purpose is to estimate the effects of the tutoring offer, and the control group is intentionally not given targeted information or preferential access. This fits the ERCT exception case: the additional resource (tutoring) is the intervention being evaluated, and the control group is business as usual by design. Criterion B is met because the resource difference is the explicit treatment variable being tested.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this specific Lern-Fair RCT by another team was found.
    • "Our study is the first to extend this research by exploring the impact of tutoring on labor market transitions..." (p. 6)
    • Relevant Quotes: 1) "Our study is the first to extend this research by exploring the impact of tutoring on labor market transitions..." (p. 6) Detailed Analysis: Criterion R requires independent replication by other authors in another context, published in a peer-reviewed journal. The paper positions itself as a first contribution on labor-market transitions in this tutoring context, and it does not report a replication of this specific RCT. A targeted literature search did not identify a separate study by an independent team that explicitly reproduces this specific Lern-Fair RCT design and cohort. Criterion R is not met because no independent replication of this study was found.
  • A

    All-subject Exams

    • Criterion E is not met, so Criterion A is automatically not met.
    • "All grades are self-reported by the young participants..." (p. 22)
    • Relevant Quotes: 1) "All grades are self-reported by the young participants and relate to the final grades from school years 2021/2022 and 2022/2023, respectively." (p. 22) Detailed Analysis: Criterion A requires standardized exam-based assessment across all main subjects, and the ERCT rules state that if Criterion E is not met then Criterion A is not met as well. Since outcomes are based on self-reported grades rather than standardized exams, the prerequisite fails. Criterion A is not met because Criterion E is not met.
  • G

    Graduation Tracking

    • The study follows participants for about 18 months, not until graduation, and no follow-up graduation-tracking paper was found.
    • "After randomization, we conducted two follow-up surveys... the second took place around 18 months later." (p. 3)
    • Relevant Quotes: 1) "After randomization, we conducted two follow-up surveys. The first, six months after randomization, coincided with the start of the new school year; the second took place around 18 months later." (p. 3) 2) "While life cycle outcomes-such as long-term wages and employment in adulthood-remain beyond the scope of this study..." (p. 42) Detailed Analysis: Criterion G requires tracking participants until graduation from the relevant educational stage. The paper explicitly describes follow-up only through about 18 months post-randomization, and it also notes that longer life-cycle outcomes are beyond scope. Although follow-up II captures an important transition point for some students, it does not constitute tracking the cohort through graduation. A targeted search for subsequent papers by the same author team that track this cohort through graduation did not identify any publicly available graduation-tracking follow-up. Criterion G is not met because the study does not track participants until graduation and no graduation follow-up paper was found.
  • P

    Pre-Registered

    • The study cites an AEA RCT Registry ID but no publicly accessible record with the registration date could be retrieved to verify pre-registration timing.
    • "The experiment was preregistered in the AEA RCT registry, AEARCTR-0008937." (p. 1)
    • Relevant Quotes: 1) "The experiment was preregistered in the AEA RCT registry, AEARCTR-0008937." (p. 1) Detailed Analysis: Criterion P requires verifying that the protocol was registered before the study began (registration date must precede trial start / data collection). The paper provides the registry identifier (AEARCTR-0008937), which is evidence of a registry entry, but the paper itself does not state the registry registration date. A targeted attempt to retrieve the public registry record and its registration date did not yield an accessible public page with the needed date information. Without a verifiable registration date, the ERCT rule for confirming pre-registration timing cannot be satisfied. Criterion P is not met because the registration date could not be verified as preceding the study start.