The Effect of Virtual Simulation Game on the Development of Nursing Students' Knowledge and Skills: A Randomized Controlled Trial

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation was conducted at the individual student level within a single university cohort, not at the class or school level.
    • "Seventeen students with a GPA below 2.88 and 61 students with a GPA of 2.88 or above were randomly assigned to the simulation (n = 39) and control (n = 39) groups."
    • Relevant Quotes: 1) "Random assignment was used to determine which students would be in the VSG group (those who use the VSG application) and the control group (those who use the current method of education) from our sample group." (p. 3) 2) "To limit the differences between the simulation and control groups, random assignment was determined using a stratified randomization method. Stratified randomization was performed using the Microsoft Excel program, following the students' overall weighted Grade Point Average (GPA). A total of 78 students were included in the randomization process." (p. 3) 3) "Seventeen students with a GPA below 2.88 and 61 students with a GPA of 2.88 or above were randomly assigned to the simulation (n = 39) and control (n = 39) groups." (p. 3) Detailed Analysis: The ERCT 'C' criterion requires that randomisation be conducted at the class level or stronger (school level), to prevent contamination, unless the intervention is a personal/tutoring intervention for which student-level randomisation is acceptable. Here, randomisation was performed at the individual student level: 78 individual students drawn from a single third-year cohort at one university were stratified by GPA and randomly assigned to the VSG or control group. No classes or schools were randomised as units. The exception for personal/one-to-one tutoring interventions does not apply: the VSG is a self-directed digital learning game, not a one-to-one tutoring intervention. However, because the VSG is accessed individually by each student on a personal device for 7 days, and the control group separately reviewed training presentations and videos, contamination is controlled at the individual level and the class-as-unit rationale is less central; nonetheless the standard requires class- or school-level randomisation and this design clearly randomises individuals within a single class/cohort. Criterion C is not met because randomisation was performed at the individual student level within a single university cohort, not at the class or school level, and the tutoring exception does not apply.
  • E

    Exam-based Assessment

    • The study used researcher-developed custom knowledge and skill tests rather than standardised, widely recognised exams.
    • "'Knowledge test on prevention of CAUTI', 'Skill test on prevention of CAUTI' ... All of which were developed by the researcher after reviewing the literature."
    • Relevant Quotes: 1) "The data were collected using the following forms: 'Questionnaire form on descriptive characteristics', 'Knowledge test on prevention of CAUTI', 'Skill test on prevention of CAUTI', and 'VSG application evaluation form'. All of which were developed by the researcher after reviewing the literature." (p. 3) 2) "The knowledge test consisted of 24 multiple-choice items, scored on a 100-point scale, with higher scores indicating greater knowledge of CAUTI prevention." (p. 4) 3) "The skills checklist evaluated students' performance in CAUTI prevention practices using a structured scoring system (3 = correctly performed, 2 = partially performed, 1 = not performed)." (p. 4) 4) "The development process and content validity of the instruments are described in detail in the published study protocol (Annak and Bulut 2022)." (p. 4) Detailed Analysis: The ERCT 'E' criterion requires the use of a standardised, widely recognised exam-based assessment that was not specially created by the researchers for the study. In this paper, both the knowledge test (24 multiple-choice items) and the skills checklist were explicitly "developed by the researcher after reviewing the literature." Their content validity was established via expert opinion (Lawshe CVR/CVI), but expert-validated custom instruments are not standardised national or widely recognised exams. No standardised, externally recognised exam (e.g., a national or state assessment) was used. Criterion E is not met because the outcome measures were custom instruments developed by the researchers, not standardised, widely recognised exam-based assessments.
  • T

    Term Duration

    • Outcomes were measured only 7 days after the intervention began, far short of one academic term.
    • "Debriefing was conducted after the post-test knowledge and skill assessments, which took place 7 days after the training on 28 May 2022."
    • Relevant Quotes: 1) "After randomization, the VSG group accessed VSG for 7 days." (Abstract) 2) "Students in the VSG group were allowed to play the VSG for 7 days. Students assigned to the control group were provided with training presentations and videos and were allowed to review them for the same seven-day period." (p. 5) 3) "Debriefing was conducted after the post-test knowledge and skill assessments, which took place 7 days after the training on 28 May 2022." (p. 5) 4) "Second, the sample size was relatively small, and the duration of follow-up was limited to the short term." (p. 12) Detailed Analysis: The ERCT 'T' criterion requires that outcomes be measured at least one full academic term (approximately 3-4 months) after the intervention begins. Here, the pre-test and training occurred on 21 May 2022, the intervention (VSG access) lasted 7 days, and the post-test was administered on 28 May 2022, just 7 days after the training. The entire interval from intervention start to outcome measurement was approximately one week, far shorter than a single academic term. The authors themselves acknowledge the follow-up was "limited to the short term." Criterion T is not met because outcomes were measured only 7 days after the intervention began, far short of the required one-term interval.
  • D

    Documented Control Group

    • The control group's demographics, baseline scores, size, and conditions are documented in detail in the CONSORT flow and Tables 1-3.
    • "There was no statistically significant difference between the simulation and control groups in terms of the descriptive characteristics of the students (p > 0.05) (Table 1)."
    • Relevant Quotes: 1) "This study was completed by 75 students (38 simulation and 37 control)." (Abstract) 2) "Students assigned to the control group were provided with training presentations and videos and were allowed to review them for the same seven-day period." (p. 5) 3) "The majority of students in both groups were female (91.9% in the control group and 84.2% in the VSG group). Most participants in each group were graduates of Anatolian high schools (73.0% in the control group and 73.7% in the VSG group)." (p. 5) 4) "There was no statistically significant difference between the simulation and control groups in terms of the descriptive characteristics of the students (p > 0.05) (Table 1)." (p. 5) 5) Table 1 reports age, gender, high school type, satisfaction, GPA, prior CAUTI training, etc. for the Control (n = 37) group; Table 2 and Table 3 report control-group pre-test knowledge (10.40 ± 2.14) and skill (133.81 ± 10.41) baseline scores. (pp. 7-9) Detailed Analysis: The ERCT 'D' criterion requires the control group to be well documented, including demographic information, baseline performance, and the treatment it received. The paper provides a detailed CONSORT flow diagram, a full demographic breakdown of the control group (n = 37) in Table 1, baseline pre-test knowledge and skill scores for the control group in Tables 2 and 3, and a clear description of what the control group received (training presentations and videos to review over the same 7-day period). This is thorough documentation of the control group's characteristics, size, baseline, and conditions. Criterion D is met because the control group's demographics, baseline knowledge and skill scores, size, and the materials it received are clearly documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation occurred at the individual student level within a single institution, not at the school level.
    • "First, the study was conducted at a single institution, which may limit the generalizability of the findings."
    • Relevant Quotes: 1) "The study population consisted of 242 students enrolled in the third year of the Department of Nursing during the fall semester of the 2021-2022 academic year." (p. 3) 2) "Stratified randomization was performed using the Microsoft Excel program, following the students' overall weighted Grade Point Average (GPA). A total of 78 students were included in the randomization process." (p. 3) 3) "First, the study was conducted at a single institution, which may limit the generalizability of the findings." (p. 12) Detailed Analysis: The ERCT 'S' criterion requires randomisation at the school (institutional/site) level, with multiple schools or units randomly assigned. This study was conducted at a single institution (Gazi University) and randomised individual students within one third-year nursing cohort. No schools, sites, or institutions were randomised as units; there was only one institution involved. Criterion S is not met because randomisation was at the individual student level within a single institution, not at the school level.
  • I

    Independent Conduct

    • The same authors designed the VSG intervention and also conducted and analysed the study, with no independent third-party evaluator.
    • "İnci Mercan Annak: conceptualization, formal analysis, methodology, ... investigation, project administration ... Hülya Bulut: conceptualization, formal analysis, methodology, ... supervision."
    • Relevant Quotes: 1) "This study is a doctoral thesis." (p. 1, footnote) 2) "Additionally, a VSG application for CAUTI prevention has been developed following content preparation during the study preparation phase. ... The VSG for Preventing CAUTI was developed in five phases: analysis, design, development, implementation, and evaluation." (p. 4) 3) "İnci Mercan Annak: conceptualization, formal analysis, methodology, validation, investigation, project administration, resources, visualization, writing, original draft, writing, review and editing, funding acquisition. Hülya Bulut: conceptualization, formal analysis, methodology, validation, writing, review and editing, supervision." (p. 12) 4) "The rater and statistician were blinded to control for detection bias and statistical bias." (p. 3) Detailed Analysis: The ERCT 'I' criterion requires that the study be conducted independently from the team that designed the intervention, to reduce bias. Here, the same authors (Mercan Annak and Bulut) designed and developed the VSG intervention, conducted the study, collected data, and performed the analysis; the work is the first author's doctoral thesis. While the paper notes that "the rater and statistician were blinded," there is no indication of an external, third-party evaluation team independent of the intervention designers. The blinding of a rater and statistician reduces some measurement bias but does not establish independent conduct, as the intervention designers led the trial. Criterion I is not met because the same researchers who designed and developed the VSG also conducted and analysed the study, with no independent third-party evaluator.
  • Y

    Year Duration

    • The tracking interval was about one week, far short of a year, and criterion T is not met.
    • "Debriefing was conducted after the post-test knowledge and skill assessments, which took place 7 days after the training on 28 May 2022."
    • Relevant Quotes: 1) "Students in the VSG group were allowed to play the VSG for 7 days." (p. 5) 2) "Debriefing was conducted after the post-test knowledge and skill assessments, which took place 7 days after the training on 28 May 2022." (p. 5) 3) "Second, the sample size was relatively small, and the duration of follow-up was limited to the short term." (p. 12) Detailed Analysis: The ERCT 'Y' criterion requires outcomes to be measured at least 75% of a full academic year (~9-10 months) after the intervention begins. Per the prompt's criterion Y instruction, if criterion T (Term Duration) is not met, then criterion Y is not met. Here, criterion T is not met (the interval from intervention start to measurement was only 7 days), and the total study tracking period was approximately one week, far short of a year. Criterion Y is not met because the interval from intervention start to outcome measurement was only about one week, far short of a year, and criterion T is not met.
  • B

    Balanced Control Group

    • The control group received an equivalent 7-day window of review materials, and the VSG is the explicit treatment variable tested against the standard method of education.
    • "Students in the VSG group were allowed to play the VSG for 7 days. Students assigned to the control group were provided with training presentations and videos and were allowed to review them for the same seven-day period."
    • Relevant Quotes: 1) "Students in the VSG group were allowed to play the VSG for 7 days. Students assigned to the control group were provided with training presentations and videos and were allowed to review them for the same seven-day period." (p. 5) 2) "After the pre-test assessments of knowledge and skill levels were completed and standardized CAUTI prevention education was provided to all participants, as a pre-intervention training, random assignment was conducted." (p. 3) 3) "The training ... was conducted on 21 May 21 2022 ... The two-hour session was supported by pictures, videos and visual materials and delivered using the lecture method." (p. 5) 4) "Number of games played per day 2.92 ± 2.43 ... Daily game playing duration 60.38 ± 45.94" (Table 1) Detailed Analysis: The ERCT 'B' criterion requires that the control group receive comparable time and resources unless the additional resource is the explicit treatment variable being tested. Applying the decision tree: both groups received the same standardised 2-hour CAUTI prevention training before randomisation. After randomisation, the VSG group accessed the virtual simulation game for 7 days, while the control group was provided with training presentations and videos to review for the same 7-day period ("the current method of education"). Both groups therefore had a 7-day self-study window with educational materials, and the extra resource in the intervention group is precisely the VSG itself. The VSG is the explicit treatment variable being tested (the study's stated aim is "to evaluate the effect of a VSG on the development of nursing students' knowledge and skills"), and the control group received an equivalent time window of educational review materials rather than a no-treatment baseline. Thus the design provides the control group with a comparable substitute (presentations and videos over the same period), and the additional VSG resource is integral to the intervention being tested against the current method of education. Criterion B is met because the control group received an equivalent 7-day window of educational review materials and the VSG is the explicit treatment variable tested against the standard method of education.

• Level 3 Criteria

  • R

    Reproduced

    • The study describes itself as the first of its kind and no independent replication by another team was found.
    • "The study is considered significant because it is the first study to include the care of urinary catheterized patients and contributes to the literature regarding the use of VSG practice in nursing education."
    • Relevant Quotes: 1) "However, while there are studies on the use of VSG in nursing education worldwide (Butt et al. 2018; Farsi et al. 2021; Keys et al. 2021; Luctkar-Flude et al. 2021), only one study has been found in Türkiye." (p. 2) 2) "only one study conducted with nursing students has developed a VSG for the prevention of Catheter-Associated Urinary Tract Infections (CAUTIs), and this study focused on the urinary catheterization procedure (Butt et al. 2018)." (p. 2) 3) "The study is considered significant because it is the first study to include the care of urinary catheterized patients and contributes to the literature regarding the use of VSG practice in nursing education." (p. 12) Detailed Analysis: The ERCT 'R' criterion requires that this specific study be independently replicated by a different research team in a different context, published in a peer-reviewed journal. The authors describe their study as "the first study to include the care of urinary catheterized patients" using a VSG, indicating it is novel. An internet search of available sources (Wiley Online Library, ClinicalTrials.gov, ResearchGate, PubMed/PMC) identified other VSG-in-nursing-education RCTs (e.g., Butt et al. 2018 on sterile catheterization technique; Bayram and Caliskan 2019 on tracheostomy care; Farsi et al. 2021 on CPR; Lee et al. 2023 on blood transfusion; and a multi-site sterile catheterization VR usability study by Breitkreuz et al. 2021). However, none of these is an independent replication of this specific CAUTI-prevention VSG study; each tests a different intervention, outcome, and context. No independent replication of this particular study by another research team was found. Criterion R is not met because this specific study, which the authors describe as the first of its kind, has not been independently replicated by another research team.
  • A

    All-subject Exams

    • Criterion E is not met and the study assessed only a single specialised topic, so the all-subject exam criterion fails.
    • "The primary outcome was the knowledge and skill level scores of the simulation and control groups ..."
    • Relevant Quotes: 1) "The primary outcome was the knowledge and skill level scores of the simulation and control groups, and the secondary outcome was the views of the VSG group regarding the VSG." (p. 5) 2) "The knowledge test consisted of 24 multiple-choice items ... The skills checklist evaluated students' performance in CAUTI prevention practices ..." (p. 4) Detailed Analysis: The ERCT 'A' criterion requires that impact be measured across all main subjects using standardised exam-based assessments, and per the prompt, if criterion E is not met then criterion A is not met. Criterion E is not met here (custom researcher-developed instruments rather than standardised exams), so criterion A automatically fails. Additionally, the study measured only a single, highly specialised topic (CAUTI prevention knowledge and skills) and did not assess any other subjects. Criterion A is not met because criterion E is not met and the study measured only a single specialised topic rather than all main subjects via standardised exams.
  • G

    Graduation Tracking

    • Criterion Y is not met and the study did not track participants beyond a 7-day post-test, let alone to graduation.
    • "Second, the sample size was relatively small, and the duration of follow-up was limited to the short term."
    • Relevant Quotes: 1) "Debriefing was conducted after the post-test knowledge and skill assessments, which took place 7 days after the training on 28 May 2022." (p. 5) 2) "Second, the sample size was relatively small, and the duration of follow-up was limited to the short term." (p. 12) 3) "Postgraduate studies might be designed to assess the enduring impacts of VSG on student performance." (p. 12) Detailed Analysis: The ERCT 'G' criterion requires that participants be tracked until graduation to assess long-term impact, and per the prompt, if criterion Y is not met then criterion G is not met. Criterion Y is not met here, so criterion G automatically fails. Furthermore, the study explicitly states its follow-up was limited to the short term (a single post-test 7 days after the intervention) with no tracking through graduation, and the authors recommend future studies to assess enduring impacts. An internet search for subsequent follow-up or graduation-tracking publications by the same authors (Annak and Bulut) identified only the earlier protocol paper (Annak and Bulut 2022); no follow-up paper tracking this cohort to graduation was found. Criterion G is not met because criterion Y is not met and there was no tracking of participants through graduation.
  • P

    Pre-Registered

    • The trial was registered on ClinicalTrials.gov (NCT05309317, first posted 4 April 2022) with a published protocol before data collection began on 21 May 2022.
    • "The study is registered with ClinicalTrials.gov under the number NCT05309317."
    • Relevant Quotes: 1) "The study is registered with ClinicalTrials.gov under the number NCT05309317." (Abstract / Trial Registration) 2) "A parallel group randomized controlled trial was designed and reported based on the CONSORT (Consolidated Standards of Reporting Trials) Non-Pharmacological Treatment checklist." (p. 2) 3) "The sample size calculation, inclusion, and exclusion criteria of the study are included in the study protocol (Annak and Bulut 2022)." (p. 3) 4) "All details of the VSG development phase are specified in the study protocol (Annak and Bulut 2022)." (p. 4) Detailed Analysis: The ERCT 'P' criterion requires the full study protocol to be pre-registered before data collection began, including hypotheses, methods, and planned analyses, with registration dated before data collection. The study is registered on ClinicalTrials.gov (NCT05309317) and a separate peer-reviewed protocol paper (Annak and Bulut 2022) describes the design, sample size, inclusion/ exclusion criteria, and instrument development in advance. Verifying timing on ClinicalTrials.gov (NCT05309317): the record shows the study was first submitted on 9 March 2022 and first posted on 4 April 2022, with an actual study start date of 21 May 2022 and a primary completion date of 28 May 2022. The training, randomisation, and data collection began on 21 May 2022 (Figure 3 and text). Therefore registration occurred before data collection began, satisfying the timing requirement. Criterion P is met because the trial was registered on ClinicalTrials.gov (NCT05309317; first submitted 9 March 2022, first posted 4 April 2022) and supported by a published protocol before data collection began on 21 May 2022.