Learning occupational safety through gamification: A randomized controlled field trial

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomization was performed at the individual worker level using even/odd random numbers, not at the class or school level, and the intervention is group-based OSH training rather than one-to-one tutoring.
    • "A simple randomization procedure was applied using a pre-generated sequence of random numbers classified as even or odd, which determined allocation to the experimental or control group in a 1:1 ratio." (p. 3)
    • Relevant Quotes: 1) "A convenience sample of workers was recruited for this study, with participants randomly assigned to either the experimental group (EG) or the control group (CG)." (p. 2) 2) "Participants were randomized at the beginning of each training session based on their presence at the meeting. A simple randomization procedure was applied using a pre-generated sequence of random numbers classified as even or odd, which determined allocation to the experimental or control group in a 1:1 ratio." (p. 3) 3) "The allocation sequence was generated by the research team prior to the sessions, and group assignment was performed by the investigators at the start of each session." (p. 3) Detailed Analysis: The ERCT 'C' criterion requires randomization at the class level or stronger (school level), unless the intervention is a personal one-to-one tutoring type, in which case student-level randomization is acceptable. Here, individual workers were randomized one by one using a pre-generated even/odd random number sequence based on their presence at each training session. This is student/individual-level randomization within the same training session, meaning EG and CG participants were mixed within the same sessions, creating clear contamination risk. The intervention is a group-based OSH seminar plus gaming session (games played in groups of six), not personal one-to-one tutoring, so the tutoring exception does not apply. Criterion C is not met because randomization was at the individual worker level rather than at the class or school level and the intervention is not personal tutoring.
  • E

    Exam-based Assessment

    • The study used the Q626 questionnaire, a custom instrument developed by the authors specifically for this study, not a widely recognized standardized exam.
    • "questions in the Q626 were developed based on the content of the powerpoint presentation of the seminar, as well as the issues presented in the games ... were developed by a panel of three experts on Occupational medicine and Workplace health and safety." (p. 6)
    • Relevant Quotes: 1) "The participants were assessed using an internally validated Q626 questionnaire at baseline (T0) and after the intervention (T1), with a knowledge score was computed ranged from 0 to15." (Abstract, p. 1) 2) "questions in the Q626 were developed based on the content of the powerpoint presentation of the seminar, as well as the issues presented in the games. These questions are related to health and safety issues derived from the Italian Decree 81/2008 (12), and were developed by a panel of three experts on Occupational medicine and Workplace health and safety." (p. 6) 3) "The internal reliability of the Q626 tool was moderate, with Cronbach's alpha of 0.723." (Abstract) 4) "The Q626 questionnaire exhibited moderate reliability (Cronbach's alpha=0.723) and may not comprehensively measure all aspects of occupational health and safety knowledge." (p. 10-11) Detailed Analysis: The ERCT 'E' criterion requires a standardized, widely recognized exam-based assessment that was not created specifically for the study. The Q626 instrument is the opposite: it is explicitly described as "internally validated" and developed by the authors' own three-expert panel, with questions aligned directly to the seminar's powerpoint content and the games used in the intervention. This is a custom, researcher-made instrument tightly aligned to the intervention content, which is precisely the type of assessment the criterion warns against, and its reliability is only moderate. Criterion E is not met because the outcome was measured with a custom internally validated questionnaire (Q626) rather than a recognized standardized exam.
  • T

    Term Duration

    • The intervention was a single one-day 4-hour course and the outcome was measured only five days later, far short of one full academic term.
    • "questionnaires were administered twice with a five-day interval." (p. 6)
    • Relevant Quotes: 1) "The intervention, named 'Let's play to 626!' ('Giochiamo a 626!') is structured as a one-day course lasting 4 hours." (p. 3) 2) "The Q626 questionnaire was administered at two time points: T0 (before the start of the training activities) and T1 (post-intervention)." (p. 6) 3) "questionnaires were administered twice with a five-day interval." (p. 6) 4) "The T1 assessment was designed to capture immediate post-intervention knowledge recall rather than long-term learning or retention." (p. 6) 5) "Therefore, the results should be interpreted as reflecting immediate post-training recall rather than stable learning outcomes." (p. 11) Detailed Analysis: The ERCT 'T' criterion requires that the primary outcome be measured at least one full academic term (~3-4 months) after the intervention begins, even if the intervention itself is short, so that term-long follow-up tracking is present. Here the intervention lasted a single day (4 hours) and the post-intervention Q626 was administered only five days after baseline, in fact immediately after the session. The authors explicitly state the assessment captured "immediate post-intervention knowledge recall rather than long-term learning or retention" and that they did not assess long-term retention. There is no term-long follow-up. Criterion T is not met because the outcome was measured only days after a one-day intervention, far short of a full academic term, with no term-long follow-up.
  • D

    Documented Control Group

    • The control group is documented in terms of size, baseline demographics, baseline knowledge scores, and its condition (seminar only, no games).
    • "Participants in the CG solely attended the 'Seminar' on occupational safety and health without engaging in any games." (p. 5)
    • Relevant Quotes: 1) "Participants in the CG solely attended the 'Seminar' on occupational safety and health without engaging in any games." (p. 5) 2) "The participants in the gaming experimentation group (EG) totalled 44 (57%), while those attending only the seminar (CG) numbered 33 (43%)." (p. 7) 3) "No significant differences were observed between the experimental and control groups at baseline with respect to age, sex, or occupational safety roles (p>0.05)." (p. 7) 4) "The Q626 scores did not significantly differ between the two groups at T0 (p=0.172; meanEG= 8.6 SD= 2.7; meanCG=7.7 SD=3.2)." (p. 7) 5) Table 2 provides CG-specific values: "T0 ... 7.7(3.2) (6.55-8.79); T1 ... 10.0(2.7) (9.06-10.94)." (Table 2, p. 8) Detailed Analysis: The ERCT 'D' criterion requires the control group to be well-documented, including size, baseline performance, and the conditions/treatments it received. The paper reports the CG size (n=33), confirms baseline equivalence on age, sex, and safety roles, provides the CG's baseline (T0) and post (T1) Q626 means with SDs and confidence intervals in Table 2, and explicitly states the CG received the seminar only without any games. The overall sample demographics (mean age 40.1, 52% female) and the CONSORT flow diagram further document the groups. Criterion D is met because the control group's size, baseline characteristics, baseline scores, and condition are clearly documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomization was conducted at the individual worker level, not at the school or institution level.
    • "A simple randomization procedure was applied using a pre-generated sequence of random numbers classified as even or odd, which determined allocation to the experimental or control group in a 1:1 ratio." (p. 3)
    • Relevant Quotes: 1) "Participants were randomized at the beginning of each training session based on their presence at the meeting. A simple randomization procedure was applied using a pre-generated sequence of random numbers classified as even or odd ..." (p. 3) 2) "All invited participants were employed as technical-administrative staff within the university sector (Sapienza and Mercatorum Universities) or in the office-based sector (one company)." (p. 2) Detailed Analysis: The ERCT 'S' criterion requires randomization at the level of the school/institution or implementing unit. In this study, individual workers drawn from across two universities and one company were randomized one at a time by even/odd random numbers within shared training sessions. No school, site, or institution was used as the unit of randomization; rather, individuals from the same institutions were split into EG and CG. Criterion S is not met because randomization occurred at the individual worker level, not at the school or institutional level.
  • I

    Independent Conduct

    • The same authors designed the intervention, the games, and the Q626 instrument and also conducted and analyzed the trial, with no independent or third-party evaluator.
    • "AM and GLT conceptualized the project, developed the main ideas, and outlined the proof. GC, DS, and FN contributed significantly to the technical aspects and suggested the experimental approach." (p. 12)
    • Relevant Quotes: 1) "The allocation sequence was generated by the research team prior to the sessions, and group assignment was performed by the investigators at the start of each session." (p. 3) 2) "questions in the Q626 were ... developed by a panel of three experts on Occupational medicine and Workplace health and safety." (p. 6) 3) "AM and GLT conceptualized the project, developed the main ideas, and outlined the proof. GC, DS, and FN contributed significantly to the technical aspects and suggested the experimental approach. AM and FN drafted the manuscript ... AM oversaw the project." (p. 12) 4) "This article was developed within the project 'Giochiamo a 626!' ... received funding from the 'Universitas Mercatorum' of Rome." (Funding, p. 11) Detailed Analysis: The ERCT 'I' criterion requires the study to be conducted independently from the team that designed the intervention, or at least to document third-party oversight of data collection and analysis. Here the same authors conceptualized the project, designed the "Let's play to 626!" intervention and games, developed the Q626 assessment, performed the randomization and group assignment as investigators, and analyzed and wrote up the results. There is no mention of any external evaluator, independent agency, or blinded data collection team. The intervention is also their own funded project. Criterion I is not met because the intervention designers conducted and analyzed the trial themselves with no independent oversight.
  • Y

    Year Duration

    • The intervention was a one-day course with outcome measured five days later, nowhere near 75% of an academic year, and criterion T is not met.
    • "The intervention, named 'Let's play to 626!' ... is structured as a one-day course lasting 4 hours." (p. 3)
    • Relevant Quotes: 1) "The intervention ... is structured as a one-day course lasting 4 hours." (p. 3) 2) "questionnaires were administered twice with a five-day interval." (p. 6) 3) "This study also did not assess long-term knowledge retention or behavioral changes in the workplace." (p. 11) Detailed Analysis: The ERCT 'Y' criterion requires outcomes measured at least 75% of a full academic year (~9-10 months) after the intervention begins. The intervention here lasted one day and the outcome was measured five days after baseline. The authors confirm no long-term retention was assessed. Additionally, per the prompt's rule, if criterion T is not met then Y cannot be met, and T is not met here. Criterion Y is not met because the follow-up was only a few days, far short of an academic year, and criterion T is not met.
  • B

    Balanced Control Group

    • The control group received the same seminar and the additional gaming session was the explicit treatment variable being tested against a seminar-only business-as-usual baseline.
    • "Participants in the CG solely attended the 'Seminar' on occupational safety and health without engaging in any games. The EG, on the other hand, participated in both the 'Seminar' and the 'gamification session'." (p. 5)
    • Relevant Quotes: 1) "Participants in the CG solely attended the 'Seminar' on occupational safety and health without engaging in any games. The EG, on the other hand, participated in both the 'Seminar' and the 'gamification session' ... in groups of six individuals." (p. 5) 2) "The effectiveness score ... was compared before and after the intervention (T0 and T1) and between the two groups (EG and CG) to estimate the effectiveness of the seminar alone versus the seminar combined with game sessions." (p. 6) 3) "the seminar (1.5 hours) consisted of a slide presentation on occupational health and safety ... Three games were included in the second session: gamification session (approximately 2 hours)." (p. 3) 4) "A further limitation of this study concerns the timing of the post-intervention assessment. While participants in the control group completed the questionnaire immediately after the seminar, those in the experimental group completed it after both the seminar and the gamification session." (p. 11) Detailed Analysis: Criterion B asks whether extra time/resources given to the intervention group are matched in the control, or whether those extra resources are themselves the explicit treatment variable. Following the decision tree: extra resources ARE present (the EG receives an additional ~2-hour gaming session that the CG does not get). However, the study's explicit and stated purpose is "to estimate the effectiveness of the seminar alone versus the seminar combined with game sessions" - i.e., the additional gamification session IS the treatment variable being tested. Both groups receive the same core 1.5-hour seminar (the shared business-as-usual educational input), and the gamification add-on is the integral, central element under investigation. This matches the spec's rule that when additional resources are explicitly the treatment variable tested against a business-as-usual baseline, the control may remain at that baseline. The known caveat is that the EG spent more total time, but this extra time is exactly the gamification intervention being evaluated, not a separable confounder. Note the timing difference in assessment (EG tested after both sessions) is an acknowledged limitation but does not change the resource framing. Criterion B is met because the additional gamification session was the explicit treatment variable tested against a shared seminar-only business-as-usual baseline.

• Level 3 Criteria

  • R

    Reproduced

    • This specific gamified OSH trial has not been independently replicated by another research team in a peer-reviewed publication.
    • Relevant Quotes: 1) "However, our study adds novel evidence by focusing on an analog board game format - a relatively unexplored yet accessible and cost-effective modality in the workplace training setting." (p. 10) 2) "This study can be seen as a preliminary intervention to explore the topic. Our intention is to repeat the RCT with a bigger sample ..." (p. 11) 3) "Given the exploratory nature and limited sample size, this research should be considered a pilot study aimed at assessing feasibility and preliminary efficacy." (p. 2) Detailed Analysis: The ERCT 'R' criterion requires independent replication of the specific study by a different research team in a different context, published in a peer-reviewed journal. The authors describe their analog board-game OSH intervention as novel and "relatively unexplored," call it a pilot study, and state their own intention to repeat it with a larger sample - which would not be an independent replication in any case. The paper cites related gamification and board-game studies by other teams (e.g., refs 41-47), but none of these is a replication of this specific "Let's play to 626!" intervention and design. Internet searching identified only the study's own protocol paper (La Torre et al., How can you learn general issues on occupational safety and health? A randomized controlled field trial research protocol ..., J Public Health, 2025, doi 10.1007/s10389-024-02397-7) and this paper itself, both by the same author team. No external independent replication of this particular trial by a different research team was identified. Criterion R is not met because no independent replication of this specific gamified OSH trial exists.
  • A

    All-subject Exams

    • The study measured only occupational health and safety knowledge via a custom questionnaire, not all main subjects, and criterion E is not met.
    • "the second section focused on knowledge of occupational health and safety: 15 multiple-choice questions with four possible answers, only one of which is correct." (p. 5)
    • Relevant Quotes: 1) "the second section focused on knowledge of occupational health and safety: 15 multiple-choice questions with four possible answers, only one of which is correct." (p. 5) 2) "The primary outcome of the study was the Q626 knowledge score at T1." (p. 6) Detailed Analysis: The ERCT 'A' criterion requires assessment of all main subjects using standardized exams, and it is conditional on criterion E being met. This study assessed only a single domain - occupational health and safety knowledge - using the custom Q626 questionnaire, with no assessment of any other subjects. Furthermore, per the prompt's rule, since criterion E (standardized exam) is not met, criterion A automatically cannot be met. Criterion A is not met because only a single OSH domain was assessed with a custom instrument and criterion E is not met.
  • G

    Graduation Tracking

    • The study only measured immediate post-training recall with no follow-up to graduation, and criterion Y is not met.
    • "This study also did not assess long-term knowledge retention or behavioral changes in the workplace." (p. 11)
    • Relevant Quotes: 1) "The T1 assessment was designed to capture immediate post-intervention knowledge recall rather than long-term learning or retention." (p. 6) 2) "This study also did not assess long-term knowledge retention or behavioral changes in the workplace." (p. 11) 3) "Future studies ... longer follow-up periods ... are needed to confirm these findings and to assess knowledge retention." (p. 11) Detailed Analysis: The ERCT 'G' criterion requires tracking participants through to graduation from their educational stage. This study involved adult workers (not students progressing to a graduation), measured only immediate post-training recall, and explicitly did not conduct any long-term follow-up. Internet searching for subsequent or follow-up publications by the same authors tracking this cohort found only the study's own protocol paper (J Public Health, 2025) and this paper; no follow-up or graduation/long-term tracking publication was found. The concept of graduation tracking is not applicable to adult workers and no such tracking was done. Per the prompt's rule, since criterion Y is not met, criterion G cannot be met either. Criterion G is not met because there was no follow-up beyond immediate post-training, no graduation/long-term tracking publication exists, and criterion Y is not met.
  • P

    Pre-Registered

    • The study was pre-registered on ClinicalTrials.gov (NCT06060379) in September 2023, before recruitment began in December 2023, and a protocol was published.
    • "The study was registered on the Clinicaltrials.gov on September 28th, 2023 with the identifier number NCT06060379 and it was titled 'Let's Play to 626!'. The protocol was published in 2024 (20)." (p. 2)
    • Relevant Quotes: 1) "The study was registered on the Clinicaltrials.gov on September 28th, 2023 with the identifier number NCT06060379 and it was titled 'Let's Play to 626!'. The protocol was published in 2024 (20)." (p. 2) 2) "The study was registered on ClinicalTrials.gov on September 28, 2023, under the identifier NCT06060379." (p. 7, Ethical issues) 3) "The period of recruitment was from December 2023 to April 2024." (p. 7) 4) Reference 20: "La Torre G, Capece G, Shaholli D, Nicoli F, Mannocci A. How can you learn general issues on occupational safety and health? A randomized controlled field trial research protocol for innovative training with gamification: let's play 626! J Public Health. 2025." (p. 13) Detailed Analysis: The ERCT 'P' criterion requires that the study protocol be pre-registered before data collection began, with a registry reference and a timing that precedes data collection. The paper states registration on ClinicalTrials.gov (NCT06060379) on September 28, 2023. Verification on ClinicalTrials.gov confirms the record NCT06060379 ("Giochiamo 626 - Gaming for Health and Safety in Workplaces"), with the study first submitted on September 18, 2023 and first posted on September 29, 2023 - consistent with the late-September 2023 registration the paper reports. Recruitment for the trial ran from December 2023 to April 2024; thus registration in late September 2023 clearly preceded the start of data collection (December 2023). A peer-reviewed protocol paper was also published (reference 20, J Public Health, 2025, doi 10.1007/s10389-024-02397-7). The registration date and registry ID are both provided and the timing precedes data collection. Criterion P is met because the trial was registered on ClinicalTrials.gov (NCT06060379) in September 2023, before recruitment/data collection began in December 2023, and a protocol was published.