Reflection prompts are not promising: a randomised controlled trial in a short MOOC shows no positive effects

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation was at the individual learner level, not classes.
    • Learners were randomly assigned to one of three versions upon clicking the ‘participate’ button.
    • Relevant Quotes: 1) "Learners were randomly assigned to one of three groups: (1) prompts directly reflecting specific SRL strategies (directed prompts), (2) prompts reflecting general reflection (generic prompts), and (3) a control group with no prompts." (p. 2) 2) "The study sample comprised all learners who enrolled in the course between autumn 2021 and spring 2022. Randomisation was achieved by duplicating the initial course and adding prompts. Learners were randomly assigned to one of three versions upon clicking the ‘participate’ button." (p. 7) Detailed Analysis: Criterion C requires randomisation at the class level (or stronger), to avoid within-class contamination, unless the intervention is clearly personal tutoring. The paper describes a MOOC where individual learners are assigned to versions of the course at the moment they click the participation button. No class-level unit exists here, and the paper does not frame the intervention as one-to-one tutoring. Therefore, the unit of randomisation is individual learners, which does not satisfy the ERCT class-level requirement. Final sentence explaining if criterion C is not met because randomisation occurred at the individual learner level rather than at the class (or school) level.
  • E

    Exam-based Assessment

    • Outcomes used a course-specific final test rather than a recognised standardised exam.
    • A final multiple-choice test, upon successful completion, granted learners a certificate of advanced training.
    • Relevant Quotes: 1) "A final multiple-choice test, upon successful completion, granted learners a certificate of advanced training." (p. 5) 2) "Final test results (had 14 questions with multiple choice and drag and drop);" (p. 6) 3) "The study utilised four dependent variables to operationalise learners’ results: ... (4) the highest score attained on the final test – to indicate achievements." (p. 7) Detailed Analysis: Criterion E requires a standardised, widely recognised exam-based assessment, not an assessment created for (or specific to) the studied course/platform. The paper’s achievement measure is the MOOC’s internal “final multiple-choice test” (14 items) used to award a course certificate. The paper does not describe this as a national/state exam or any externally validated standardised test used beyond this course. Therefore, the assessment does not meet ERCT’s exam-based requirement. Final sentence explaining if criterion E is not met because the outcomes rely on a course-specific final test rather than a recognised standardised exam.
  • T

    Term Duration

    • Outcomes were measured within a short-duration course without term-long follow-up.
    • First, the intervention was brief and limited to short text-based prompts within a short-duration course.
    • Relevant Quotes: 1) "This study investigates the effectiveness of brief reflection interventions designed to support self-regulated learning in a short, Massive Open Online Course for in-service teachers." (p. 2) 2) "First, the intervention was brief and limited to short text-based prompts within a short-duration course." (p. 11) Detailed Analysis: Criterion T requires that outcomes be measured at least one full academic term after the intervention begins (roughly 3-4 months), even if the intervention itself is brief. The paper characterises the course as “short” and later explicitly calls it a “short-duration course,” with the intervention embedded in-module and outcomes defined as course activity and final-test performance within that same course. The paper provides no evidence of a term-later follow-up measurement after course completion. Therefore, the study does not satisfy the ERCT term-duration follow-up requirement. Final sentence explaining if criterion T is not met because the study does not report outcome measurement at least one academic term after the intervention began.
  • D

    Documented Control Group

    • The control condition is described, but baseline and full control-group characteristics are not documented for most participants.
    • The Control Group (CG) did not receive any questions and studied the course without any additional reflection tasks; at the end of each unit they had only the page with congratulations and the link to download unit materials.
    • Relevant Quotes: 1) "The Control Group (CG) did not receive any questions and studied the course without any additional reflection tasks; at the end of each unit they had only the page with congratulations and the link to download unit materials." (p. 6) 2) "The completion of the questionnaire was optional for the students." (p. 6) 3) "It is worth noting that upon dividing the participants into groups, the sample size becomes considerably smaller (partly due to not all participants undertaking the questionnaire), potentially leading to an insufficient sample size for detecting the significance of the effect." (p. 9) Detailed Analysis: Criterion D requires a well-documented control group, including baseline information and key characteristics that allow readers to judge comparability. The paper clearly describes what the control group received (no prompts), but it states that the survey capturing learner characteristics was optional. It also notes that subgroup analyses are limited because not all participants completed the questionnaire. The paper does not provide comprehensive baseline performance measures (for example, prior achievement) and does not demonstrate that the full control group’s characteristics are documented beyond the optional survey subset. Therefore, the control group is not sufficiently documented for ERCT purposes. Final sentence explaining if criterion D is not met because baseline and participant characteristics for the control group are incomplete due to optional data collection and limited reporting.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation was not conducted at the school (institution) level.
    • Learners were randomly assigned to one of three versions upon clicking the ‘participate’ button.
    • Relevant Quotes: 1) "Learners were randomly assigned to one of three versions upon clicking the ‘participate’ button." (p. 7) Detailed Analysis: Criterion S requires randomisation at the school (or equivalent institution/site) level. The study is a MOOC experiment in which individual learners are assigned to one of three course versions at the point of participation. No schools or institutional units are randomised. Final sentence explaining if criterion S is not met because the unit of randomisation is individual learners, not schools or equivalent institutions.
  • I

    Independent Conduct

    • The paper describes the authors implementing the intervention themselves rather than an independent evaluator.
    • To do this, we conducted a randomised control trial that prompted learners’ reflection by adding different questions to the course content.
    • Relevant Quotes: 1) "To do this, we conducted a randomised control trial that prompted learners’ reflection by adding different questions to the course content." (p. 3) 2) "We integrated self-regulation tasks directly into the course..." (p. 10) Detailed Analysis: Criterion I requires the study to be conducted independently from the designers of the intervention. The paper uses first-person language to describe conducting the trial and integrating the prompts into the course, indicating the intervention designers and the evaluators are the same team. The paper does not provide evidence of an external, independent evaluation organisation responsible for implementation and analysis. Final sentence explaining if criterion I is not met because the same team that designed and embedded the prompts also conducted the evaluation, with no documented independent evaluator.
  • Y

    Year Duration

    • Outcomes were not tracked for a full academic year after the intervention began.
    • The study sample comprised all learners who enrolled in the course between autumn 2021 and spring 2022.
    • Relevant Quotes: 1) "The study sample comprised all learners who enrolled in the course between autumn 2021 and spring 2022." (p. 7) 2) "First, the intervention was brief and limited to short text-based prompts within a short-duration course." (p. 11) Detailed Analysis: Criterion Y requires outcome measurement at least one academic year after the intervention begins. The paper describes enrolment spanning autumn 2021 to spring 2022, but it does not describe year-later outcome tracking for participants after their intervention exposure. Instead, it frames the course as “short-duration” and analyses outcomes tied to course engagement and course test performance. There is no evidence of measurement one academic year after intervention start for any cohort. Final sentence explaining if criterion Y is not met because the paper does not report year-later tracking of outcomes after the intervention began.
  • B

    Balanced Control Group

    • Any added time from writing responses is the treatment itself and is described as minimal.
    • Finally, even when participants formally completed the reflection task, the minimal input required (three characters) may not have elicited substantive cognitive engagement, limiting the actual impact of the intervention.
    • Relevant Quotes: 1) "The Control Group (CG) did not receive any questions and studied the course without any additional reflection tasks..." (p. 6) 2) "To answer each question, learners were offered a text field, the answer was counted if they used at least three characters." (p. 6) 3) "Finally, even when participants formally completed the reflection task, the minimal input required (three characters) may not have elicited substantive cognitive engagement, limiting the actual impact of the intervention." (p. 11) Detailed Analysis: Criterion B requires comparable time/resources across groups unless the additional time/resource is the treatment variable or the difference is negligible. Here, the only added “resource” is requiring learners in the two prompt conditions to type short responses in a text field, while the control group receives no prompts. The study’s explicit treatment is the presence/type of reflection prompts (and the act of responding to them), so the time-on-task difference is integral to what is being tested. Additionally, the paper indicates the threshold for counting engagement is “at least three characters,” and later characterises this as “minimal input,” supporting that the added burden is small. Final sentence explaining if criterion B is met because the added time for typing responses is the treatment being tested and is described as minimal.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this specific RCT is identified.
    • As far as we know, this research is the first attempt to draw attention to the existence of different prompt types in SRL in MOOCs.
    • Relevant Quotes: 1) "As far as we know, this research is the first attempt to draw attention to the existence of different prompt types in SRL in MOOCs." (p. 11) Detailed Analysis: Criterion R requires an independent replication by other authors in a different context, published in a peer-reviewed outlet. The paper does not report any such replication, and it explicitly frames itself as a first attempt in this specific direction. A web search for replication studies of this exact RCT (by title, DOI, and authors) did not identify any published independent replication as of the ERCT check date. Final sentence explaining if criterion R is not met because no independent replication study of this specific trial could be found.
  • A

    All-subject Exams

    • Because E is not met, A is automatically not met.
    • Relevant Quotes: 1) "A final multiple-choice test, upon successful completion, granted learners a certificate of advanced training." (p. 5) Detailed Analysis: Criterion A requires standardised exam-based measurement across all core subjects, and it depends on Criterion E being met. The only achievement assessment here is a course-specific final test within a single MOOC, not standardised exams across subjects. Since Criterion E is not met, Criterion A is not met by definition. Final sentence explaining if criterion A is not met because criterion E is not met and the study does not measure standardised outcomes across all core subjects.
  • G

    Graduation Tracking

    • Because Y is not met, G is automatically not met; no evidence of graduation tracking was found.
    • Relevant Quotes: 1) "First, the intervention was brief and limited to short text-based prompts within a short-duration course." (p. 11) Detailed Analysis: Criterion G requires tracking participants until graduation, and ERCT rules state that if Criterion Y is not met, Criterion G is not met. This study is a short MOOC intervention with outcomes tied to course activity and a final course test, and the paper provides no evidence of long-term follow-up to any graduation milestone. A web search for follow-up papers by the same authors reporting graduation tracking for this cohort did not identify any such publication as of the ERCT check date. Final sentence explaining if criterion G is not met because criterion Y is not met and no graduation tracking (in this paper or identifiable follow- up papers) is documented.
  • P

    Pre-Registered

    • The paper provides an OSF link for data and code, but it does not report a pre-registered protocol with a pre-data-collection date.
    • The data, R code for the analysis and tables with the results of regression models are available via the link: https://osf.io/49vpe/?view_only=594109bdac6544d29900cf4561f6be85
    • Relevant Quotes: 1) "The data, R code for the analysis and tables with the results of regression models are available via the link: https://osf.io/49vpe/?view_only=594109bdac6544d29900cf4561f6be85" (p. 8) Detailed Analysis: Criterion P requires a pre-registered study protocol (hypotheses, methods, planned analyses) registered before data collection begins, with evidence of timing. The paper does not state that the study was pre-registered, does not provide a registry identifier (for example, OSF registration/registry entry, AEA RCT registry, ClinicalTrials.gov, ISRCTN), and provides no registration date relative to study start. The only transparency link is to “data” and “R code,” which is not sufficient evidence of pre- registration. Attempts to verify a corresponding pre-registration entry via web search did not yield a clearly identifiable pre-registered protocol for this study with a pre-study date. Final sentence explaining if criterion P is not met because there is no documented pre-registered protocol with verifiable registration timing before data collection.