Effectiveness of Kahoot Game-based Learning and Satisfaction of Newly Graduated Nurses: A Pilot Randomized Controlled Trial

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Allocation was made at the individual level using the parity of each nurse's work number within a single hospital cohort, not at the class or school level, and no tutoring exception applies.
    • "A random allocation sequence was created based on the recruits' work numbers. Participants with job numbers ending in an odd number were assigned to the traditional group, while those with job numbers ending in an even number were assigned to the KGBL group."
    • Relevant Quotes: 1) "A randomised controlled trial was used in this study to compare the KGBLgroup to the control group." (p. 13) 2) "A random allocation sequence was created based on the recruits' work numbers. Participants with job numbers ending in an odd number were assigned to the traditional group, while those with job numbers ending in an even number were assigned to the KGBL group. All participants were randomly assigned to one of the two groups." (p. 13) 3) "Eighty-seven newly graduated nurses from a tertiary hospital were randomly assigned to either the KGBL group (n = 44) or the traditional group (n = 43)." (p. 14) Detailed Analysis: Criterion C requires randomisation at the class level (or stronger school level), with entire classes assigned to conditions, unless the intervention is one-to-one tutoring. Here the unit of allocation was the individual nurse: each participant was assigned based on whether their personal work number ended in an odd or even digit. All 87 participants were drawn from a single tertiary hospital cohort and trained together, so intervention and control nurses were mixed within the same training environment rather than separated into distinct randomised classes. This is individual-level (and only quasi-random, odd/even) allocation, which carries exactly the contamination risk Criterion C is designed to prevent. The intervention is classroom gamified learning, not personal tutoring, so the tutoring exception does not apply. Criterion C is not met because randomisation was at the individual student level within a single cohort, not at the class or school level.
  • E

    Exam-based Assessment

    • Knowledge was measured with a researcher-made 50-item closed-book exam and a custom observational attention scale, not a recognised standardised exam.
    • "The assessment assessed the students' understanding of theoretical concepts through a 50-item, closed-book exam that included multiple-choice and multiple-selection questions."
    • Relevant Quotes: 1) "The assessment assessed the students' understanding of theoretical concepts through a 50-item, closed-book exam that included multiple-choice and multiple-selection questions. There was a 100-point maximum possible score." (p. 15) 2) "Based on the validated attention assessment dimensions (i.e., eye contact and body language) reported in existing literature [16, 18, 20], the observation indicators and predefined scoring standards for this study were determined through face-to-face discussions and consensus among the research team members." (p. 14) 3) "The researchers developed a questionnaire based on the work of Andres et al [22]. The final nine-item questionnaire included two additional items added by our study." (p. 15) Detailed Analysis: Criterion E requires a standardised, widely recognised exam (e.g., a national or state-wide test) rather than a test created by the researchers for the study. The primary knowledge outcome was a 50-item closed-book exam aligned to the training content; there is no statement that this is a recognised standardised exam, and it appears purpose-built for the course. The attention measure was an observational scale whose indicators and scoring standards were defined by the research team for this study, and the satisfaction instrument was a researcher-developed questionnaire. None of these qualify as a standardised exam-based assessment. Criterion E is not met because the outcomes were measured with researcher-designed instruments rather than a recognised standardised exam.
  • T

    Term Duration

    • The intervention and all outcome measurement occurred within a single five-day training program, far shorter than one academic term.
    • "The training for both groups spanned five consecutive days in late August 2023."
    • Relevant Quotes: 1) "The experimental group underwent a 5-day training program incorporating pre-class quizzes via the KGBL platform..." (Abstract) 2) "The training for both groups spanned five consecutive days in late August 2023." (p. 14) 3) "This evaluation was carried out at the end of the fifteen lessons." (p. 15) 4) "Despite limitations such as the short study duration and lack of long-term follow-up..." (Abstract) Detailed Analysis: Criterion T requires that outcomes be measured at least one full academic term (approximately 3-4 months) after the intervention begins. Here the entire program ran over five consecutive days, with the theoretical knowledge exam administered at the end of those lessons and attention observed during the sessions themselves. The authors themselves describe the "short study duration and lack of long-term follow-up" as a limitation. The interval from intervention start to measurement is therefore only days, well below one term. Criterion T is not met because outcomes were measured within a five-day window, far short of one academic term.
  • D

    Documented Control Group

    • The control group is documented with its size, training condition (traditional instruction only), and baseline demographic and educational characteristics in Table 3.
    • "The control group (n = 43) received only traditional instruction, while the KGBLgroup (n = 44) learned hospital policies and procedures through both traditional instruction and the KGBL online game."
    • Relevant Quotes: 1) "The control group (n = 43) received only traditional instruction, while the KGBLgroup (n = 44) learned hospital policies and procedures through both traditional instruction and the KGBL online game." (p. 13) 2) "In contrast, the traditional group received lectures through standard formats such as PowerPoint presentations and videos." (p. 14) 3) "Table 3. Baseline characteristics (n=87). ... Age (years), mean (SD) Control group 22.17+/-1.14 ... Gender Female 34 (79.1) ... educational status Associate Bachelor 38 (88.4) / 5 (11.6) ... KGBL experience Yes 5 (11.5) / No 38 (88.4)" (p. 17) 4) "A comparison of the participants in both groups according to age, gender, educational status, play KGBL experience, revealed no educationally relevant or significant differences." (p. 16-17) Detailed Analysis: Criterion D requires that the control group be well-documented, including size, demographic/baseline information, and the conditions it received. The paper states the control group size (n = 43) and that it received only traditional instruction via standard lecture formats (PowerPoint, videos). Table 3 reports baseline age, gender, educational status, and prior KGBL experience separately for the control group, and the text confirms no significant baseline differences between groups. This provides clear documentation of who the control group was and what they received. Criterion D is met because the control group's size, condition, and baseline characteristics are clearly documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Allocation was at the individual level within one hospital cohort, with no randomisation of schools or institutions.
    • "A random allocation sequence was created based on the recruits' work numbers."
    • Relevant Quotes: 1) "Newly graduated nurses from a tertiary hospital in China were recruited in September 2023. Participants were randomly assigned to either the experimental group (n=44) or the control group (n=43)." (Abstract) 2) "A random allocation sequence was created based on the recruits' work numbers. ... All participants were randomly assigned to one of the two groups." (p. 13) Detailed Analysis: Criterion S requires randomisation at the school / institution / implementing-unit level, with multiple such units randomly assigned to conditions. This study was conducted in a single tertiary hospital, and the unit of allocation was the individual nurse (by odd/even work number). There is only one institution and no randomisation of schools or sites. Criterion S is not met because randomisation occurred at the individual level within a single hospital, not at the school/institution level.
  • I

    Independent Conduct

    • The same research team designed the intervention and instruments and ran the trial; the only independent parties were unaffiliated observers for attention scoring, not an external evaluation of the study as a whole.
    • "Skilled nurses with no affiliation to this project served as the observers, keeping a careful eye on the students' engagement and behaviors in the classroom."
    • Relevant Quotes: 1) "Skilled nurses with no affiliation to this project served as the observers, keeping a careful eye on the students' engagement and behaviors in the classroom. We gave the observers a brief training session..." (p. 13-14) 2) "the observation indicators and predefined scoring standards for this study were determined through face-to-face discussions and consensus among the research team members." (p. 14) 3) "Author Contributions ... Minping Deng: Conceptualization, Formal Analysis, Investigation, Methodology, Visualization, Writing - original draft ... Yanfei Li: Conceptualization, Methodology, Project administration, Supervision, Writing - review & editing" (p. 20) Detailed Analysis: Criterion I requires that the study be conducted independently from those who designed the intervention, typically via a third-party evaluation team handling data collection, analysis, and conclusions. Here the same authors conceived the study, designed the attention scoring scale and questionnaire, conducted the analysis, and wrote the conclusions. While unaffiliated skilled nurses served as classroom observers for the attention measure, this is a narrow use of independent raters; it does not amount to an external/third-party evaluation of the intervention's knowledge outcomes, design, and analysis. The intervention design, exam, statistical analysis, and conclusions all remained with the authoring team. Criterion I is not met because the same team designed and evaluated the intervention, with independence limited only to classroom attention observers.
  • Y

    Year Duration

    • The study spanned only five days with no year-long tracking, and since Criterion T is not met, Y cannot be met.
    • "The training for both groups spanned five consecutive days in late August 2023."
    • Relevant Quotes: 1) "The training for both groups spanned five consecutive days in late August 2023." (p. 14) 2) "The absence of long-term follow-up was another noteworthy drawback, which prevented an analysis of KGBL long-term effects on attention and memory retention." (p. 20) Detailed Analysis: Criterion Y requires outcomes to be measured at least 75% of a full academic year after the intervention begins. The intervention and measurement occurred within five days, and the authors explicitly note the absence of long-term follow-up. Per the standard, Y also cannot be met when Criterion T (Term Duration) is not met, and here T is not met. Criterion Y is not met because the study lasted only five days, far short of an academic year, and T was not met.
  • B

    Balanced Control Group

    • Both groups followed the same curriculum, teachers, materials, and class time; the only difference was the short 2-6 minute KGBL quizzes, which are integral to the gamification intervention being tested rather than a separable extra resource.
    • "The teachers and teaching materials were consistent across both groups, with the key difference being the use of KGBL in the experimental group."
    • Relevant Quotes: 1) "Both groups followed the same curriculum based on the 2016 'Guiding Opinions on Basic Theoretical Training for New Nurses.' The training took place from 8 a.m. to 12 a.m., with three one-hour classes each morning and a 10-minute break between classes. The teachers and teaching materials were consistent across both groups, with the key difference being the use of KGBL in the experimental group." (p. 14) 2) "In the KGBL group, participants completed a short, dynamic quiz at the beginning of each class using the KGBL platform. The 2-6 minutes quizzes featured 10-15 multiple-choice questions dessigned to reinforce the course content." (p. 14) 3) "In contrast, the traditional group received lectures through standard formats such as PowerPoint presentations and videos." (p. 14) Detailed Analysis: Criterion B asks whether the intervention adds time or budget not matched in the control group, and if so whether those extra resources are the explicit treatment variable. Here both groups received the same curriculum, the same teachers, the same teaching materials, and the same overall class schedule (three one-hour classes per morning over five days). The only difference was that the KGBL group did a short 2-6 minute quiz at the start of each class instead of additional standard lecture content; this is a minor and embedded difference in time, not a substantial extra resource. Moreover, the KGBL quiz IS the gamification intervention being tested - the study explicitly compares gamified learning against traditional instruction, so the gamified component is integral to the treatment variable, not a separable confounding add-on. Following the decision tree, the extra time is negligible and, in any case, the added resource is the treatment variable itself, so the control's business-as-usual condition is appropriate. Criterion B is met because the groups had matched curriculum, teachers, materials, and time, and the only difference (the brief KGBL quizzes) is the integral treatment variable being tested.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this specific pilot trial exists; the paper cites prior Kahoot studies as background but none reproduce this study.
    • Relevant Quotes: 1) "Research on KGBL use in RN preceptor education is extremely limited." (p. 13) 2) "this study specifically explores the application value of KGBL in new nurse vocational training, a research direction that is extremely limited in existing literature..." (p. 19) 3) "This pilot study demonstrated the feasibility of KGBL in the nursing training program in hospital situations..." (Abstract) Detailed Analysis: Criterion R requires that this specific study be independently replicated by a different team in a different context and published in a peer-reviewed journal. The paper is described as a pilot study and explicitly notes that research on KGBL in new-nurse/preceptor hospital training is extremely limited and rarely involved in prior studies. The numerous cited Kahoot studies (in colleges, nursing schools, medical students, etc.) are background literature, not replications of this particular Chinese hospital new-nurse trial. An external internet search across publisher databases, PubMed, and ResearchGate found no independent reproduction of this specific Deng et al. trial; related work such as the "Integration of Game-Based Chatbot With Kahoot! for High-Alert Medication Education for Newly Graduated Nurses" (Journal of Continuing Education in Nursing, 2026) is a different intervention by a different team and does not replicate this study. Criterion R is not met because there is no independent replication of this specific pilot trial.
  • A

    All-subject Exams

    • Only theoretical knowledge for the nurse-training content was assessed (via a non-standardised exam), not all core subjects, and since Criterion E is not met, A cannot be met.
    • "The assessment assessed the students' understanding of theoretical concepts through a 50-item, closed-book exam."
    • Relevant Quotes: 1) "The training content followed the 2016 'Newly Employed Nurse Training Outline,' covering essential theoretical knowledge and practical guidelines." (Abstract) 2) "The assessment assessed the students' understanding of theoretical concepts through a 50-item, closed-book exam that included multiple-choice and multiple-selection questions." (p. 15) Detailed Analysis: Criterion A requires that all main subjects be measured using standardised exam-based assessments, and it has Criterion E as a prerequisite. This study measured only theoretical knowledge of the nurse-training content using a researcher-made exam, plus attention and satisfaction. No broad set of core subjects was assessed, and the assessment was not standardised. Since Criterion E is not met, A automatically cannot be met. Criterion A is not met because only a single domain was assessed with a non-standardised exam, and Criterion E was not satisfied.
  • G

    Graduation Tracking

    • There was no follow-up beyond the five-day program and no graduation tracking, and since Criterion Y is not met, G cannot be met.
    • "The absence of long-term follow-up was another noteworthy drawback, which prevented an analysis of KGBL long-term effects on attention and memory retention."
    • Relevant Quotes: 1) "The absence of long-term follow-up was another noteworthy drawback, which prevented an analysis of KGBL long-term effects on attention and memory retention." (p. 20) 2) "Future studies should expand the sample size, include more diverse populations, and evaluate the long-term impacts of gamified learning in this context." (p. 20) Detailed Analysis: Criterion G requires tracking participants through to graduation to assess long-term impact, and it has Criterion Y as a prerequisite. This study measured outcomes only at the end of the five-day program, with the authors explicitly acknowledging the absence of long-term follow-up. The participants are already qualified working nurses, and no longitudinal tracking was conducted. An external search for subsequent follow-up publications by the same authors tracking this cohort found none. Since Criterion Y is not met, G automatically cannot be met. Criterion G is not met because there was no long-term follow-up or graduation tracking, and Criterion Y was not satisfied.
  • P

    Pre-Registered

    • The paper contains no mention of any pre-registration on a trial registry before data collection.
    • Relevant Quotes: 1) "A randomised controlled trial was used in this study to compare the KGBLgroup to the control group. The Fifth Affiliated Hospital of Jinan University's ethics committee approved the study." (p. 13) 2) "The necessary approvals were obtained from the relevant institutions to conduct the research." (p. 16) Detailed Analysis: Criterion P requires that the full study protocol be pre-registered on a public registry (e.g., ClinicalTrials.gov, ISRCTN, ChiCTR) before data collection begins, with hypotheses and planned analyses. The paper mentions only ethics committee approval and informed consent; there is no reference to any trial registry, no registration number, and no registration date. An external search of trial registries found no pre-registration record for this study. Criterion P is not met because the paper provides no pre-registration reference or registry identifier.