Augmented Reality–Supported Learning During a Pediatric Neurology Rotation: A Cluster-Randomized Controlled Trial

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation was conducted at the cluster (rotation-subgroup) level rather than the individual student level, satisfying the class-level RCT requirement.
    • "These rotation subgroups were then randomly assigned (at the cluster level) to the intervention or control condition."
    • Relevant Quotes: 1) "In this prospective cluster-randomized controlled pretest-posttest study, 102 students were allocated by 4 rotation subgroups (clusters) to an intervention group (standard curriculum plus ARLEs delivered via a mobile application; n = 51) or a control group (standard curriculum only; n = 51)." (Abstract, p. 1) 2) "These rotation subgroups were then randomly assigned (at the cluster level) to the intervention or control condition." (p. 3) 3) "Fourth-year medical students participate in the pediatrics rotation in 4 large subgroups in accordance with the faculty curriculum. These subgroups were regarded as the cluster units within the scope of the study... At study initiation, assignment of the subgroups to conditions was randomly determined at the cluster level in a 1:1 ratio; accordingly, 2 subgroups were assigned to the intervention condition... The remaining 2 subgroups constituted the control condition." (p. 3) Detailed Analysis: Criterion C requires that randomisation be conducted at the class level (or stronger), not at the individual student level within a single class, to prevent contamination. The paper explicitly describes a cluster-randomized design in which the unit of randomisation was the rotation subgroup (4 clusters), not the individual student. Two subgroups were randomly assigned to the intervention and two to the control. The rotation subgroup functions as a class-level (or larger) cluster, since whole intact groups of students were allocated together. This satisfies the requirement that randomisation occur at the class level or higher, and reduces within-class contamination because entire subgroups, rather than students within a single group, were assigned. Criterion C is met because randomisation was performed at the cluster (rotation-subgroup) level, which is at least class-level and avoids student-level allocation within a single class.
  • E

    Exam-based Assessment

    • The study used a custom, author-developed 30-item achievement test rather than a widely recognised standardised exam, so this criterion is not met.
    • "The AT was developed using a blueprint aligned with the rotation learning objectives to ensure coverage of each targeted competency; item content was finalized following content-validity review by pediatric neurology experts."
    • Relevant Quotes: 1) "Achievement was assessed using a 30-item multiple-choice achievement test and analyzed with a 2 × 2 mixed-design analysis of variance and Welch t test." (Abstract, p. 1) 2) "The AT was developed using a blueprint aligned with the rotation learning objectives to ensure coverage of each targeted competency; item content was finalized following content-validity review by pediatric neurology experts. Internal consistency was supported (KR-20 = 0.83), with a mean discrimination index of 0.41 and item-total correlations ranging from 0.30 to 0.90." (p. 5) 3) "In both conditions, the same 30-item multiple-choice achievement test (AT) was administered as the pretest immediately before the start of the pediatric neurology component of the pediatrics rotation and as the posttest on completion of that component." (p. 3) Detailed Analysis: Criterion E requires that the study use a standardised, widely recognised exam-based assessment that is not specially designed for the study. Here, the outcome was a 30-item multiple-choice achievement test that the authors themselves developed using a blueprint aligned with their rotation learning objectives and finalised following content-validity review by their own pediatric neurology experts. This is a custom, study-specific instrument built to measure the intervention's targeted content, not a national, state-wide, or otherwise widely recognised standardised exam. The reported KR-20 and discrimination indices establish internal reliability of the custom test but do not make it a standardised exam. Criterion E is not met because the assessment was a custom achievement test developed by the authors for this study, not a widely recognised standardised exam.
  • T

    Term Duration

    • Outcomes were measured immediately at the end of a short pediatric neurology rotation component, well under one full academic term, so this criterion is not met.
    • "Because the primary outcome was a multiple-choice achievement test administered immediately before and after the pediatric neurology component of the rotation, the findings are limited to short-term test-based cognitive outcomes."
    • Relevant Quotes: 1) "In both conditions, the same 30-item multiple-choice achievement test (AT) was administered as the pretest immediately before the start of the pediatric neurology component of the pediatrics rotation and as the posttest on completion of that component." (p. 3) 2) "Because the primary outcome was a multiple-choice achievement test administered immediately before and after the pediatric neurology component of the rotation, the findings are limited to short-term test-based cognitive outcomes." (p. 8) 3) "The same test form was administered at both time points; therefore, AT findings were interpreted as short-term test-based gains within the rotation." (p. 5) 4) "ARLEs integrated into the pediatric neurology rotation may provide a potentially useful short-term adjunct to, rather than a replacement for, face-to-face instruction." (Abstract, p. 1) Detailed Analysis: Criterion T requires that outcomes be measured at least one full academic term (approximately 3-4 months) after the intervention begins. In this study, the pretest was administered immediately before the pediatric neurology component of the pediatrics rotation and the posttest immediately on completion of that component. The pediatric neurology component is only one part of a single pediatrics rotation block, and the authors repeatedly characterise the outcomes as "short-term test-based gains within the rotation" and a "short-term adjunct." There is no indication that the interval from intervention start to outcome measurement reached a full academic term; on the contrary, the measurement occurred at the end of a short rotation component with no term-long follow-up. Criterion T is not met because outcomes were measured immediately at the end of a short pediatric neurology rotation component, far shorter than one full academic term, and the authors explicitly describe the gains as short-term.
  • D

    Documented Control Group

    • The control group's size, demographics, baseline scores, and conditions are clearly documented, meeting the requirement.
    • "An independent samplest test conducted to examine potential preintervention differences in achievement levels showed no statistically significant difference between the pretest mean scores; intervention group: 5.10±1.49, control group: 5.33±1.35; t(100)=−0.84, P=.405."
    • Relevant Quotes: 1) "A total of 102 students were included in the study; 51 were in the intervention group, in which ARLEs were added to the standard pediatric neurology curriculum, and 51 were in the control group, which received only the standard pediatric neurology curriculum." (p. 3) 2) "The mean ages of the groups were highly similar (mean ≈ 22 years in both groups), and no statistically significant between-group difference was observed at baseline (all P values > .05, indicating no statistically significant baseline differences). The gender distribution also exhibited a comparable pattern: the intervention group included 27 men (52.9%) and 24 women (47.1%), whereas the control group included 28 men (54.9%) and 23 women (45.1%)." (p. 3) 3) "Students in the control condition, during the same period, received training only within the scope of the standard curriculum and did not have access to the ARLE; after completion of the posttest assessment, access to the ARLE content was offered to all students." (p. 5) 4) "An independent samplest test conducted to examine potential preintervention differences in achievement levels showed no statistically significant difference between the pretest mean scores; intervention group: 5.10±1.49, control group: 5.33±1.35; t(100)=−0.84, P=.405." (p. 5) Detailed Analysis: Criterion D requires that the control group be well-documented, including its size, demographic characteristics, baseline performance, and the conditions (treatment) it received. The paper provides the control group's size (n = 51), demographics (gender split, mean age ≈ 22 years), baseline achievement scores (5.33 ± 1.35) with a formal baseline equivalence test, and a clear statement of what the control group received (standard pediatric neurology curriculum only, no access to the ARLE during the study). This is sufficient documentation to confirm the control group's comparability and conditions. Criterion D is met because the control group's size, demographics, baseline scores, and business-as-usual treatment are clearly documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation occurred among rotation subgroups within a single medical school rather than across multiple schools or sites, so the school-level RCT criterion is not met.
    • "First, the study was conducted at a single medical school among fourth-year students attending the pediatrics rotation, and randomization was performed at the cluster level by treating the existing rotation subgroups as the cluster units."
    • Relevant Quotes: 1) "This prospective, cluster-randomized controlled, pretest-posttest quantitative study was conducted among fourth-year medical students enrolled at a state university medical school." (p. 2) 2) "These subgroups were regarded as the cluster units within the scope of the study and were used as the primary units for forming the intervention and control conditions. At study initiation, assignment of the subgroups to conditions was randomly determined at the cluster level in a 1:1 ratio." (p. 3) 3) "First, the study was conducted at a single medical school among fourth-year students attending the pediatrics rotation, and randomization was performed at the cluster level by treating the existing rotation subgroups as the cluster units." (p. 8) Detailed Analysis: Criterion S requires that randomisation occur at the school level (the educational institution or unit implementing the intervention), among multiple schools or sites, not just classes within a single school. This study was conducted at a single medical school, and randomisation occurred at the level of rotation subgroups (clusters) within that one institution. No multiple schools or sites were randomised; the clusters are intact groups of students within a single faculty. Therefore the randomisation is at the class/cluster level within one school, not at the school level. Criterion S is not met because randomisation occurred among rotation subgroups within a single medical school, not among multiple schools or implementing sites.
  • I

    Independent Conduct

    • The same team that designed and built the intervention also delivered it, collected data, and analysed results, with no independent evaluator, so this criterion is not met.
    • "U.P. conducted data collection, performed the primary analyses, and drafted the initial manuscript."
    • Relevant Quotes: 1) "ARLEs, developed in alignment with the pediatric neurology training program, were offered to students in the intervention clusters as an additional learning component after written informed consent was obtained and in a manner that did not disrupt the routine course flow of the medical school." (p. 2) 2) "This article is derived from the master's thesis (unpublished) of the second author (Ufuk Polat, U.P.), completed under the supervision of the third author, Levent Çetinkaya (L.Ç.), with co-supervision by the first author, Turgay Cokyaman (T.Ç.). L.Ç. developed the original research idea and overall conceptual framework. Building on this framework, T.Ç. led the development of the educational content and the experimental procedures. The ARLE mobile application was designed and implemented by U.P. under the joint guidance of L.Ç. and T.Ç. U.P. conducted data collection, performed the primary analyses, and drafted the initial manuscript." (p. 9, Author Contributions) 3) "Within their assigned rotation blocks, all students completed the same standard pediatric neurology component of the pediatrics rotation; in both conditions, theoretical and practical training was delivered with identical content by the same faculty members." (p. 3) Detailed Analysis: Criterion I requires that the study be conducted independently from the authors who designed the intervention, with third-party evaluators or external oversight, to reduce bias. Here, the Author Contributions statement makes clear that the same team that conceived and developed the ARLE intervention (L.Ç. developed the conceptual framework, T.Ç. led content/procedure development, U.P. designed and implemented the app) also conducted data collection, performed the primary analyses, and drafted the manuscript. There is no external evaluation team, independent data collection agency, or blinded third-party oversight described. Training and evaluation were carried out by the same faculty members who designed the intervention. Criterion I is not met because the same authors who designed and developed the ARLE intervention also implemented it, collected the data, and performed the analyses, with no independent or third-party evaluation.
  • Y

    Year Duration

    • Outcomes were measured within a short rotation window, far less than 75% of an academic year, and criterion T is not met, so Y is not met.
    • "Because the same AT form was administered at both time points, the observed gains should be interpreted within this short-term window and may partly reflect test familiarity in addition to learning."
    • Relevant Quotes: 1) "In both conditions, the same 30-item multiple-choice achievement test (AT) was administered as the pretest immediately before the start of the pediatric neurology component of the pediatrics rotation and as the posttest on completion of that component." (p. 3) 2) "Because the same AT form was administered at both time points, the observed gains should be interpreted within this short-term window and may partly reflect test familiarity in addition to learning." (p. 8) 3) "ARLEs integrated into the pediatric neurology rotation may provide a potentially useful short-term adjunct to, rather than a replacement for, face-to-face instruction." (Abstract, p. 1) Detailed Analysis: Criterion Y requires that outcomes be measured at least 75% of one full academic year (~9-10 months) after the intervention begins. Per the prompt's criterion-specific instruction, if criterion T (Term Duration) is not met, then Y is automatically not met. Criterion T is not met here because outcomes were measured immediately at the end of a short pediatric neurology rotation component, far shorter than one term. Independently, the study itself is explicitly short-term, with pretest and posttest bracketing only the pediatric neurology component of a single rotation block, nowhere near a full academic year. Criterion Y is not met because the measurement window was a short rotation component, far less than 75% of an academic year (and T is not met).
  • B

    Balanced Control Group

    • The extra resource (the ARLE app) is the explicit treatment variable tested against an identical standard curriculum, so the business-as-usual control is appropriate and the criterion is met.
    • "We evaluated whether augmented reality–supported learning environments (ARLEs) integrated into the rotation improve medical students' cognitive achievement beyond the standard curriculum."
    • Relevant Quotes: 1) "We evaluated whether augmented reality–supported learning environments (ARLEs) integrated into the rotation improve medical students' cognitive achievement beyond the standard curriculum." (Abstract, p. 1) 2) "In the intervention condition, ARLEs were implemented as an additional learning component complementary to the standard pediatric neurology curriculum, whereas in the control condition, only the standard face-to-face training was maintained." (p. 3) 3) "Within their assigned rotation blocks, all students completed the same standard pediatric neurology component of the pediatrics rotation; in both conditions, theoretical and practical training was delivered with identical content by the same faculty members. In this way, it was intended that the primary difference between the intervention and control conditions would stem from the addition of ARLEs to the standard curriculum." (p. 3) 4) "Students in the intervention condition were encouraged to use the ARLE application during the pediatric neurology rotation in their extracurricular time, in line with their own study preferences." (p. 5) Detailed Analysis: Criterion B asks whether, when the intervention group receives extra time/budget/resources, the control group is matched, unless the additional resource is itself the explicit treatment variable being tested. Here the intervention group received an extra resource: access to the ARLE mobile application, used in extracurricular time, on top of the identical standard curriculum that both groups received. The control group received the standard curriculum only, with no comparable add-on. This is an imbalance in resources/time. However, applying the decision tree: the additional resource (the ARLE app) is precisely the treatment variable the study sets out to test. The stated aim is to "evaluate whether augmented reality–supported learning environments (ARLEs)... improve medical students' cognitive achievement beyond the standard curriculum," and the design deliberately holds the standard curriculum identical across both arms ("delivered with identical content by the same faculty members") so that the only difference is the addition of ARLEs. The ARLE add-on is therefore integral to and the explicit object of the study, and the control group is the appropriate business-as-usual baseline. Under the standard, an explicitly tested additional-resource treatment with a business-as-usual control is acceptable. Criterion B is met because the additional resource (the ARLE mobile application) is the explicit treatment variable being tested against an otherwise identical standard curriculum, making the business-as-usual control appropriate by design.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this specific pediatric neurology ARLE trial exists; only loosely related AR/VR studies in other domains are cited, so this criterion is not met.
    • "These limitations highlight the importance of future studies that are multicenter, longitudinal, and incorporate broader outcome measures."
    • Relevant Quotes: 1) "Current findings are largely consistent with the literature on the use of immersive technologies in medical education, particularly augmented reality applications. Systematic reviews and meta-analyses published in recent years indicate that AR-based learning environments are associated with significant improvements in knowledge acquisition, spatial cognition, and the development of clinical skills, often with medium-to-large effect sizes." (p. 7) 2) "studies conducted in areas such as obstetrics and clinical examination have focused on scenarios including postpartum hemorrhage management, fetal position assessment, and knee examination; these studies have reported that VR/AR-supported training, compared with standard training, may be associated with higher examination scores, shorter task completion times, and higher levels of self-confidence or student satisfaction." (p. 7) 3) "These limitations highlight the importance of future studies that are multicenter, longitudinal, and incorporate broader outcome measures." (p. 8) Detailed Analysis: Criterion R requires that this specific study be independently replicated by a different research team in a different context, published in a peer-reviewed journal. This is a newly published (2026) single-site trial of an ARLE intervention in a pediatric neurology rotation. The paper cites related AR/VR studies in other domains (obstetrics, clinical examination), but these are different interventions in different contexts, not replications of this specific pediatric-neurology ARLE trial. The authors themselves call for future replication ("multicenter, longitudinal"). An external internet search (PubMed, Google Scholar, Sage journal site, ClinicalTrials.gov) was conducted for an independent replication of this specific pediatric neurology ARLE cluster-randomized trial. No independent replication by a different research team was found; only thematically related AR/VR medical-education studies in other clinical domains (e.g., obstetrics, basic life support, septic shock simulation) exist, none of which reproduce this specific intervention, population, or design. Criterion R is not met because no independent replication of this specific pediatric neurology ARLE trial by a different research team has been published; only loosely related AR/VR studies in other domains are cited or identifiable.
  • A

    All-subject Exams

    • The study used a custom single-subject test (criterion E not met) and assessed only pediatric neurology content, so the all-subject exams criterion is not met.
    • "Achievement was assessed using a 30-item multiple-choice achievement test."
    • Relevant Quotes: 1) "Achievement was assessed using a 30-item multiple-choice achievement test." (Abstract, p. 1) 2) "Prepared ARLE content was structured around 6 core clinical topics included in the pediatric neurology curriculum: West syndrome, juvenile myoclonic epilepsy, typical absence epilepsies, self-limited epilepsy with centrotemporal spikes (SeLECTS), spinal muscular atrophy, and Duchenne muscular dystrophy." (p. 4) 3) "The AT was developed using a blueprint aligned with the rotation learning objectives to ensure coverage of each targeted competency." (p. 5) Detailed Analysis: Criterion A requires that the study measure impact across all main subjects using standardised exam-based assessments, and it explicitly depends on criterion E: if E is not met, A cannot be met. Criterion E is not met here because the outcome was a custom author-developed achievement test rather than a standardised exam. Furthermore, the study measured only a single, narrow domain (pediatric neurology content within the pediatrics rotation), not all main subjects. On both grounds the criterion fails. Criterion A is not met because the assessment was a custom single-subject achievement test (criterion E not met) and no other main subjects were assessed.
  • G

    Graduation Tracking

    • Measurement stopped immediately after the rotation with no graduation tracking, no follow-up publication exists, and criterion Y is not met, so G is not met.
    • "Because the primary outcome was a multiple-choice achievement test administered immediately before and after the pediatric neurology component of the rotation, the findings are limited to short-term test-based cognitive outcomes; clinical reasoning, applied skills, and bedside performance with real patients were not assessed separately within the scope of this study."
    • Relevant Quotes: 1) "In both conditions, the same 30-item multiple-choice achievement test (AT) was administered as the pretest immediately before the start of the pediatric neurology component of the pediatrics rotation and as the posttest on completion of that component." (p. 3) 2) "Because the primary outcome was a multiple-choice achievement test administered immediately before and after the pediatric neurology component of the rotation, the findings are limited to short-term test-based cognitive outcomes; clinical reasoning, applied skills, and bedside performance with real patients were not assessed separately within the scope of this study." (p. 8) 3) "These limitations highlight the importance of future studies that are multicenter, longitudinal, and incorporate broader outcome measures." (p. 8) Detailed Analysis: Criterion G requires follow-up tracking of participants until graduation, and per the prompt it depends on criterion Y: if Y is not met, G is automatically not met. Criterion Y is not met. Independently, this study measured outcomes only immediately at the end of a short rotation component, with no follow-up beyond the rotation and certainly no tracking to graduation from medical school. The authors explicitly note the short-term nature and call for longitudinal future work. An external internet search was conducted for any subsequent follow-up publication by these authors (Cokyaman, Polat, Çetinkaya) that tracks this cohort to graduation. No such follow-up paper was found; the article is derived from an unpublished master's thesis and reports only immediate post-rotation outcomes. Criterion G is not met because there was no follow-up to graduation (only immediate post-rotation measurement), no subsequent graduation-tracking publication exists, and criterion Y is not met.
  • P

    Pre-Registered

    • The paper reports ethics committee approval but provides no public trial pre-registration or registry ID, and none was found externally, so this criterion is not met.
    • Relevant Quotes: 1) "The study was carried out between August 1, 2023, and December 31, 2024, after obtaining ethics committee approval and in accordance with the relevant ethical principles (ethical approval no.: 2023-YÖNP-0435)." (p. 2) 2) "Ethical approval was obtained from the Ethics Committee of Çanakkale Onsekiz Mart University (ethical approval no. 2023-YÖNP-0435). The study was conducted in accordance with applicable ethical standards and institutional guidelines." (p. 9) 3) "The data sets generated and/or analyzed during the current study are available from the corresponding author on reasonable request, subject to ethical and institutional data protection requirements." (p. 9) Detailed Analysis: Criterion P requires that the full study protocol be pre-registered on a public registry (e.g., ClinicalTrials.gov, ISRCTN) before data collection began, with hypotheses, methods, and planned analyses. The paper mentions ethics committee approval and an ethics approval number, but ethics approval is not equivalent to public pre-registration of a trial protocol. No trial registry, registration ID, or pre-registration date is mentioned anywhere in the paper. An external internet search of trial registries and the publisher landing page was conducted; no public trial registration (e.g., ClinicalTrials.gov, ISRCTN, or a Turkish registry) for this study could be located. There is therefore no evidence of a pre-registered protocol. Criterion P is not met because the paper provides only ethics committee approval, with no reference to a public trial pre-registration or registry ID, and none could be found externally.