Evaluating medical students’ engagement and confidence across three simulation-based education methods: standardized patient, high fidelity simulator, and virtual reality

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • The study is not randomized at the class (or school) level; the same cohort experienced all modalities in a fixed sequence.
    • "Third, all students completed the three simulation modalities in the same fixed sequence (HFS → SP → VR), raising the possibility of order effects." (p. 6)
    • Relevant Quotes: 1) "This study conducted quantitative research through a self-reported survey to examine the impact of simulation-based education experiences, using SP, HFS, and VR, on the engagement and confidence of participating medical students." (p. 3) 2) "During the one-week respiratory rotation, the education was conducted in the order of HFS, SP, and VR." (p. 3) 3) "Third, all students completed the three simulation modalities in the same fixed sequence (HFS → SP → VR), raising the possibility of order effects." (p. 6) Detailed Analysis: Criterion C requires a randomized controlled trial where the unit of randomization is at least the class level (or stronger, e.g., school/site), unless the study is clearly about one-to-one tutoring where student-level assignment is appropriate. The paper describes a within-cohort comparison in which the same medical students experienced all three modalities (HFS, SP, VR). The intervention exposure is explicitly delivered in a fixed order, and the paper itself highlights this fixed sequence as a limitation because of possible order effects. The paper does not describe any random assignment of classes, sites, or even randomization of the order of modalities. Therefore, this study is not a class-level (or stronger) RCT under the ERCT definition. Criterion C is not met because there is no documented random assignment at the class (or school) level.
  • E

    Exam-based Assessment

    • Outcomes are measured using self-report surveys rather than a standardized exam-based assessment.
    • "Engagement was measured using self-reported surveys distributed after each simulation module." (p. 3)
    • Relevant Quotes: 1) "Engagement was measured using self-reported surveys distributed after each simulation module." (p. 3) 2) "These survey items measured confidence levels in history-taking, physical examination, interacting and communicating with patients, and clinical reasoning, using a 10-point rating scale." (p. 3) 3) "Each item was rated on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree)." (p. 3) Detailed Analysis: Criterion E requires outcomes measured via standardized exam-based assessments (i.e., official or widely recognized tests), rather than researcher-administered self-report scales. The paper clearly states that both engagement and confidence were assessed with self-report survey instruments (Likert-type items and 10-point ratings). These are not standardized exams of educational achievement, and no external exam-based assessment is described. Criterion E is not met because the study relies on self-report surveys instead of standardized exams.
  • T

    Term Duration

    • Measurements are taken immediately around a one-week rotation and pre/post sessions, not at least one academic term after start.
    • "During the one-week respiratory rotation, the education was conducted in the order of HFS, SP, and VR." (p. 3)
    • Relevant Quotes: 1) "During the one-week respiratory rotation, the education was conducted in the order of HFS, SP, and VR." (p. 3) 2) "Engagement was measured using self-reported surveys distributed after each simulation module." (p. 3) 3) "The survey was administered as a self-reported questionnaire both before and after the three simulation practice sessions." (p. 3) Detailed Analysis: Criterion T requires outcome measurement at least one academic term after the intervention begins (or at least term-long follow-up from the start, even if the intervention itself is brief). Here, the intervention exposures (HFS, SP, VR) occur during a "one-week respiratory rotation," and engagement is measured after each module. Confidence is measured immediately before and after the three simulation practice sessions. This design indicates short-term, immediate measurement rather than term-long follow-up. Criterion T is not met because outcomes are measured within the one-week rotation and immediately pre/post sessions, not after a term.
  • D

    Documented Control Group

    • There is no separate documented control group; the design is a single cohort experiencing all three modalities.
    • "The students participated in SP-, HFS-, and VR-based simulation education." (p. 1)
    • Relevant Quotes: 1) "The students participated in SP-, HFS-, and VR-based simulation education." (p. 1) 2) "This study analyzed self-reported survey results from 43 fifth-year medical students (31 males and 12 females) at the Ajou University School of Medicine in 2023." (p. 3) 3) "During the one-week respiratory rotation, the education was conducted in the order of HFS, SP, and VR." (p. 3) Detailed Analysis: Criterion D requires a clearly documented control group, including who the control participants are and what they received, with enough detail to support meaningful comparison. This paper describes one cohort of medical students who all participate in three simulation-based education modalities, with outcomes measured by surveys after each module and pre/post across sessions. No parallel control group (business-as-usual or active control) is described, and therefore the control condition cannot be documented as required by ERCT. Criterion D is not met because there is no separate, documented control group condition.

• Level 2 Criteria

  • S

    School-level RCT

    • The study is conducted within one institution and does not randomize schools/sites to conditions.
    • "This study analyzed self-reported survey results from 43 fifth-year medical students (31 males and 12 females) at the Ajou University School of Medicine in 2023." (p. 3)
    • Relevant Quotes: 1) "This study analyzed self-reported survey results from 43 fifth-year medical students (31 males and 12 females) at the Ajou University School of Medicine in 2023." (p. 3) Detailed Analysis: Criterion S requires randomization at the school (site/institution) level. The paper describes a single university cohort and does not describe recruitment of multiple schools/sites nor any school-level random assignment to intervention versus control. Criterion S is not met because there is no school-level randomization.
  • I

    Independent Conduct

    • Independent third-party conduct is not documented; the authors report conducting the analysis themselves.
    • "Formal analysis : Jihye Yu, Yun Jung Jung" (p. 6)
    • Relevant Quotes: 1) "Authors’ contributions" (p. 6) 2) "Formal analysis : Jihye Yu, Yun Jung Jung" (p. 6) 3) "Statistical analysis was performed using SPSS Statistics 24." (p. 3) Detailed Analysis: Criterion I requires evidence that the evaluation (implementation, data collection, and/or analysis) was conducted independently from the intervention designers/implementers (e.g., external evaluators, independent data-collection team, or clear separation of roles). The paper lists author contributions and explicitly credits the "Formal analysis" to two of the paper’s authors. The paper does not include statements indicating an external evaluation team, an independent data-collection contractor, or other independence safeguards. Criterion I is not met because independent conduct is not documented and the authors report performing the formal analysis.
  • Y

    Year Duration

    • The study does not measure outcomes over 75% of an academic year, and term duration is also not met.
    • "Fourth, the study period was brief, limiting the evaluation of long-term educational effects." (p. 6)
    • Relevant Quotes: 1) "During the one-week respiratory rotation, the education was conducted in the order of HFS, SP, and VR." (p. 3) 2) "Fourth, the study period was brief, limiting the evaluation of long-term educational effects." (p. 6) Detailed Analysis: Criterion Y requires outcomes measured at least 75% of an academic year after intervention begins. Per ERCT dependencies, if Criterion T is not met, Criterion Y is not met. The paper describes the educational exposure as occurring during a one-week rotation and acknowledges the study period is brief and limits evaluation of long-term effects. This is incompatible with year-long tracking, and Criterion T is also not satisfied. Criterion Y is not met because the study is brief and does not track outcomes over most of an academic year.
  • B

    Balanced Control Group

    • There is no control group and the compared modalities differ in format and resources (e.g., one-on-one vs small group), so balanced inputs cannot be established.
    • "In the SP-based simulation, students conducted one-on-one patient interviews and performed clinical assessments with standardized patients." (p. 3)
    • Relevant Quotes: 1) "The HFS used Laerdal’s SimMan simulator and was conducted in groups of 2–3 students." (p. 3) 2) "In the SP-based simulation, students conducted one-on-one patient interviews and performed clinical assessments with standardized patients." (p. 3) 3) "The simulation was conducted in groups of 2–3 students and was based on a sepsis scenario." (p. 3) 4) "After each simulation session, a 30-minute group debriefing, including feedback from the instructor, was conducted." (p. 3) Detailed Analysis: Criterion B requires that time, attention, and other educational resources be balanced between intervention and control conditions, unless the study explicitly frames additional resources as the treatment variable being tested. This paper does not include a separate control condition at all; instead, it compares three different modalities experienced by the same cohort. Moreover, the modalities differ in format and resources (e.g., SP is explicitly "one-on-one" while HFS/VR are conducted in groups of "2–3 students"), and also differ by scenario and delivery. Because there is no control group to balance against, and because the interventions differ in multiple resource-relevant ways, ERCT’s balanced-control requirement (as a way to isolate effects) cannot be satisfied here. Criterion B is not met because there is no control group and the compared conditions do not establish balanced inputs.

• Level 3 Criteria

  • R

    Reproduced

    • No independent peer-reviewed replication of this specific study was found, and the paper itself reports a single-university sample.
    • "First, the sample was limited to medical students from a single university, potentially restricting the generalizability of the findings." (p. 6)
    • Relevant Quotes: 1) "First, the sample was limited to medical students from a single university, potentially restricting the generalizability of the findings." (p. 6) Detailed Analysis: Criterion R requires that the study be independently replicated by a different research team, in a different context, and published in a peer-reviewed outlet. The paper is newly published (2026) and describes a single- university cohort. A targeted internet search by DOI and full title was performed to identify independent replications that explicitly reproduce this three-modality comparison; no such independent replication papers were identified in the sources checked. Criterion R is not met because independent replication of this specific study was not found.
  • A

    All-subject Exams

    • Because exam-based assessment (E) is not met, all-subject standardized exams (A) are also not met.
    • "Engagement was measured using self-reported surveys distributed after each simulation module." (p. 3)
    • Relevant Quotes: 1) "Engagement was measured using self-reported surveys distributed after each simulation module." (p. 3) 2) "Confidence was measured using four items adapted from Ytterberg et al. [26] that were relevant to this study’s context." (p. 3) Detailed Analysis: Criterion A requires standardized exam-based assessments across all main subjects. Per ERCT dependencies, if Criterion E is not met, Criterion A cannot be met. This study uses self-report survey measures (engagement and confidence) and does not use standardized exams in any subject. Criterion A is not met because the study does not use standardized exam-based assessment (E is not met).
  • G

    Graduation Tracking

    • The study is brief and does not track participants to graduation; no follow-up graduation-tracking publications were found.
    • "Fourth, the study period was brief, limiting the evaluation of long-term educational effects." (p. 6)
    • Relevant Quotes: 1) "Fourth, the study period was brief, limiting the evaluation of long-term educational effects." (p. 6) 2) "Future research should include larger samples from multiple universities, use various evaluators and objective assessment tools, and assess long-term educational impacts." (p. 6) Detailed Analysis: Criterion G requires tracking participants until graduation from the relevant educational stage. Per ERCT dependencies, if Criterion Y is not met, Criterion G is not met. This paper explicitly characterizes the study as brief and calls for future work to assess long-term impacts, indicating that graduation tracking was not performed in this study. An internet search using the paper’s DOI, title, and author names was also conducted to identify subsequent follow-up publications by the same author team that track this cohort to graduation; no such graduation-tracking follow-up papers were found in the sources checked. Criterion G is not met because year-long tracking is absent (Y is not met) and no graduation-tracking follow-up was identified.
  • P

    Pre-Registered

    • The paper reports no clinical trial number and describes the work as retrospective, providing no evidence of prospective pre-registration.
    • "Clinical trial number" (p. 6)
    • Relevant Quotes: 1) "Clinical trial number" (p. 6) 2) "Not applicable." (p. 6) 3) "The requirement for informed consent was waived by the Ajou University Hospital Institutional Review Board, as this was a retrospective study analyzing student engagement and clinical performance confidence..." (p. 6) Detailed Analysis: Criterion P requires a publicly pre-registered protocol with a registration date that occurs before data collection begins. The paper explicitly states that the clinical trial number is "Not applicable" and describes the study as a "retrospective study." The paper does not provide a registry (e.g., OSF, ClinicalTrials), a registration ID, or a registration date. An internet search using the DOI and title did not identify a linked preregistration record for this study. Criterion P is not met because no pre-registered protocol is documented and none was found.