Positive evaluation of an online intervention that promotes digital health literacy in adolescents as recommended by the German Digital Healthcare Act: Randomized controlled trial

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomization was performed at the individual participant level, not at the class (or stronger) level, and no tutoring exception applies.
    • "Randomization was based on a centralized computerized procedure with random number allocation corresponding to the study groups for each participant"
    • Relevant Quotes: 1) "Participants were recruited from a German online panel." (Section 2.4, p. 3) 2) "participants were assigned to the experimental or the control group through randomization" (Section 2.4, p. 3) 3) "Randomization was based on a centralized computerized procedure with random number allocation corresponding to the study groups for each participant" (Section 2.4, p. 3) Detailed Analysis: Criterion C requires random assignment at the class level (or stronger, e.g., school-level), to reduce contamination, unless the intervention is clearly one-to-one tutoring/personal teaching. The paper describes recruitment from an "online panel" and states that randomization corresponded to study groups "for each participant." This indicates an individually randomized trial, rather than a clustered (class- or school-level) educational RCT. The intervention is an online e-learning course, not one-to-one tutoring, so the tutoring exception does not apply. Therefore, the unit of randomization does not meet the ERCT class-level (or stronger) requirement. Criterion C is not met because participants (not classes/schools) were randomized.
  • E

    Exam-based Assessment

    • Outcomes were measured using questionnaires and a self-developed knowledge quiz rather than standardized exam-based assessments.
    • "A multiple-select quiz was self-developed to assess the participants’ knowledge"
    • Relevant Quotes: 1) "The evaluation questionnaires included objective and subjective knowledge" (Section 2.5, p. 3) 2) "A multiple-select quiz was self-developed to assess the participants’ knowledge" (Section 2.5.2, p. 3) 3) "In addition, subject knowledge was assessed with ten newly developed items." (Section 2.5.2, p. 3) Detailed Analysis: Criterion E requires standardized exam-based assessments (e.g., state/national standardized tests or widely recognized achievement exams). It is not satisfied by self-report questionnaires or a researcher-made test tailored to the intervention. The paper explicitly reports questionnaires and a "self-developed" objective knowledge quiz. While some instruments are described as validated questionnaires, they are not exam-based standardized educational assessments in the ERCT sense. Criterion E is not met because the study does not use standardized exam-based outcome assessments.
  • T

    Term Duration

    • The intervention and post-test occurred within days (data collection within about one month), far shorter than one academic term from intervention start to outcome measurement.
    • "Data collection took place from 14th of October to 12th of November 2024."
    • Relevant Quotes: 1) "Completing the entire course takes approximately 60–90 min." (Section 2.3, p. 3) 2) "Data collection took place from 14th of October to 12th of November 2024." (Section 2.4, p. 3) 3) "Control group participants were invited to the T2 measurements a few days after the completion of the T1 measurements" (Section 2.4, p. 3) 4) "the post assessments took place in the days immediately after the participants completed the online intervention" (Section 4.2, p. 10) Detailed Analysis: Criterion T requires outcomes to be measured at least one full academic term (roughly 3–4 months) after the intervention begins. The intervention is very short ("60–90 min"), and the paper states that post-assessments took place "in the days immediately after" completion. The full data collection window is about one month, and the control group timing was only "a few days" from T1 to T2. Thus, the intervention-start to outcome-measurement interval is far shorter than one academic term. Criterion T is not met because outcomes were assessed within days, not at least one term after intervention start.
  • D

    Documented Control Group

    • The paper documents control and experimental group sizes and provides baseline demographic characteristics and pre/post outcome descriptives by group.
    • "Detailed sample characteristics for the two groups and the entire sample can be found in Table 1."
    • Relevant Quotes: 1) "the control group participants (N = 238 randomized; N = 212 analyzed) were in a waiting period." (Abstract, p. 1) 2) "Detailed sample characteristics for the two groups and the entire sample can be found in Table 1." (Section 3.1, p. 5) 3) "There were no statistically significant differences regarding age" (Section 3.1, p. 5) 4) "Table 2 Means (M), standard deviation (SD), and standard errors (SE) for the questionnaire scales." (Table 2, p. 5) Detailed Analysis: Criterion D requires the control group to be well documented, including sample size, the control condition, and baseline characteristics and/or baseline outcome descriptives. The paper identifies the control group as a waiting control, reports randomized and analyzed group sizes, and describes baseline sociodemographics and pre/post descriptives by group (Tables 1 and 2). This provides the minimum documentation needed to interpret comparisons. Criterion D is met because the control condition and baseline characteristics/outcomes are documented.

• Level 2 Criteria

  • S

    School-level RCT

    • The unit of randomization was individual participants from an online panel rather than schools (or other educational institutions) randomized as clusters.
    • "Participants were recruited from a German online panel"
    • Relevant Quotes: 1) "Participants were recruited from a German online panel" (Section 2.4, p. 3) 2) "participants were assigned to the experimental or the control group through randomization" (Section 2.4, p. 3) 3) "Randomization was based on a centralized computerized procedure with random number allocation corresponding to the study groups for each participant" (Section 2.4, p. 3) Detailed Analysis: Criterion S requires a school-level (or site-level) cluster RCT, where schools (or analogous institutions implementing the intervention) are randomized. The paper describes individual recruitment from an online panel and randomization for each participant, with no indication that schools/classes were recruited and randomized as clusters. Criterion S is not met because randomization did not occur at the school (or equivalent institutional) level.
  • I

    Independent Conduct

    • The intervention was developed by the authors’ organization, and the paper does not document that the overall evaluation and analysis were led by an independent external team.
    • "developed by the independent non-profit foundation Stiftung Gesundheitswissen"
    • Relevant Quotes: 1) "developed by the independent non-profit foundation Stiftung Gesundheitswissen" (Section 2.3, p. 3) 2) "The recruitment, randomization, survey administration and data collection were carried out by a market research institute." (Section 2.1, p. 2) 3) "This research was funded by the independent, nonprofit foundation Stiftung Gesundheitswissen." (Funding statement, p. 9) Detailed Analysis: Criterion I requires that the study be conducted independently of the intervention developer to reduce risks of biased conduct, measurement, analysis, and reporting. The paper states that the course was developed by Stiftung Gesundheitswissen, which is also the authors’ affiliated organization. The paper also states that recruitment and data collection tasks were carried out by a market research institute, which supports partial operational separation. However, the paper does not clearly document that the overall evaluation leadership and analysis were performed by an independent external evaluation team separate from the developer organization. Criterion I is not met because independence of the full evaluation and analysis from the intervention developer is not clearly documented.
  • Y

    Year Duration

    • Outcomes were assessed within days after intervention, far below 75% of an academic year; additionally, since criterion T is not met, criterion Y is not met.
    • "Data collection took place from 14th of October to 12th of November 2024."
    • Relevant Quotes: 1) "Data collection took place from 14th of October to 12th of November 2024." (Section 2.4, p. 3) 2) "Control group participants were invited to the T2 measurements a few days after the completion of the T1 measurements" (Section 2.4, p. 3) 3) "the post assessments took place in the days immediately after the participants completed the online intervention" (Section 4.2, p. 10) Detailed Analysis: Criterion Y requires that outcomes be measured at least 75% of an academic year after intervention start. The paper reports a short time window (about one month total), and explicitly states that post assessment occurred "in the days immediately after" completion, with control post measurement only "a few days" after baseline. Additionally, under the ERCT dependency rule, if criterion T is not met, criterion Y is not met. Since T is not met here, Y cannot be met. Criterion Y is not met because the follow-up duration is far shorter than an academic year (and T is not met).
  • B

    Balanced Control Group

    • The intervention required substantial participant time while the control group was a waiting control with no comparable substitute activity, creating an unbalanced input condition.
    • "the control group participants were in the waiting period."
    • Relevant Quotes: 1) "the control group participants were in the waiting period." (Section 2.4, p. 3) 2) "Completing the entire course takes approximately 60–90 min." (Section 2.3, p. 3) 3) "In future studies, an active control group completing a different intervention could help to circumvent this problem." (Section 4.2, p. 10) Detailed Analysis: Criterion B compares the nature, quantity, and quality of resources/time provided to intervention versus control, unless the extra resources are explicitly the treatment variable. Here, participants in the intervention group completed a structured course that takes approximately "60–90 min." The control group was explicitly in a waiting period and did not complete an alternative activity that would provide comparable time-on-task, attention, or engagement. The paper itself notes that an "active control group" could help address this issue, reinforcing that the control condition did not provide a comparable substitute for the intervention inputs. Criterion B is not met because the control condition does not balance the intervention’s additional time and engagement inputs.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication by a different research team in a different context was found or documented.
    • Relevant Quotes: 1) "These findings not only align with other findings from our working group evaluating similar e-learning courses" (Section 4.1, p. 8) 2) "Future studies should focus on the stability of these effect sizes over time" (Section 4.1, p. 8) Detailed Analysis: Criterion R requires independent reproduction: a replication of the same intervention and core claims by a different research team, in a different context, published in a peer-reviewed outlet. The paper references other studies from the authors’ own working group and calls for future research, but does not provide evidence of an independent replication of this specific "Stress im Netz" RCT. An internet search for replication studies referring to this paper and/or this specific course did not identify an independent peer-reviewed replication by other authors. Criterion R is not met because independent reproduction evidence was not found.
  • A

    All-subject Exams

    • Criterion E is not met (no standardized exam-based outcomes), so criterion A cannot be met; additionally, outcomes do not cover all core school subjects.
    • "The evaluation questionnaires included objective and subjective knowledge"
    • Relevant Quotes: 1) "The evaluation questionnaires included objective and subjective knowledge" (Section 2.5, p. 3) 2) "A multiple-select quiz was self-developed" (Section 2.5.2, p. 3) Detailed Analysis: Criterion A requires standardized exam-based assessment across all main subjects, and it has an explicit dependency: if criterion E is not met, criterion A is not met. The study measures intervention-related knowledge and questionnaire outcomes (digital health literacy and digital competence), not standardized exams across core subjects. Criterion A is not met because standardized exam-based assessments are not used (E is not met) and all-subject coverage is not provided.
  • G

    Graduation Tracking

    • No tracking through graduation is reported, no follow-up papers tracking this cohort to graduation were found, and criterion G cannot be met because criterion Y is not met.
    • "The stability of these findings over time ... should be investigated in the future."
    • Relevant Quotes: 1) "The stability of these findings over time ... should be investigated in the future." (Abstract, p. 1) 2) "Future studies need to show whether the observed positive changes remain stable over time." (Section 4.3, p. 9) Detailed Analysis: Criterion G requires follow-up tracking until graduation (end of an educational stage). It also has a dependency: if criterion Y is not met, criterion G is not met. The paper reports only immediate post-intervention outcomes and repeatedly frames longer-term stability as future work, without any graduation outcomes. A targeted internet search for subsequent follow-up publications by the same author team reporting graduation tracking for this cohort did not identify such papers. Criterion G is not met because the study does not track to graduation and Y is not met.
  • P

    Pre-Registered

    • The study reports OSF preregistration dated October 4, 2024, which precedes the stated data collection period beginning October 14, 2024.
    • "The following hypotheses were preregistered prior to data collection (4th of October 2024; https://osf.io/prqmk)"
    • Relevant Quotes: 1) "The following hypotheses were preregistered prior to data collection (4th of October 2024; https://osf.io/prqmk)" (Introduction, p. 2) 2) "After ethical review, the study was preregistered on the Open Science Framework (OSF; https://osf.io/prqmk)." (Section 2.1, p. 2) 3) "Data collection took place from 14th of October to 12th of November 2024." (Section 2.4, p. 3) Detailed Analysis: Criterion P requires pre-registration of the study protocol before data collection begins, and evidence that the registration timing precedes data collection. The paper provides a specific preregistration date (October 4, 2024) and an OSF link, and separately states the data collection window began October 14, 2024. This supports that preregistration preceded data collection. I attempted to verify the OSF record directly via the OSF link, but the OSF page content was not retrievable in the current tool environment. The paper’s explicit preregistration statement with a date remains the best available evidence here. Criterion P is met because the paper documents preregistration on OSF dated before data collection began.