Real-time artificial intelligence-based needle tracking for ultrasound-guided regional anesthesia training: a pilot prospective randomized controlled trial

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomization was at the individual student level, but the intervention is individualized simulator skill training with no shared-classroom contamination, so the personal-teaching exception applies and student-level assignment is acceptable.
    • "Group allocation was performed using computer-generated random numbers with a block size of four. An independent research assistant ensured allocation concealment."
    • Relevant Quotes: 1) "Twenty-three medical students with no UGRA experience were randomized into control (n=12) and YOLOv5 feedback (n=11) groups." (Abstract, p. 1) 2) "The participants were allocated to either a control group trained without feedback or a You Only Look Once version 5 (YOLOv5) group (Y group) trained with feedback from the real-time feedback system. Group allocation was performed using computer-generated random numbers with a block size of four. An independent research assistant ensured allocation concealment." (p. 3) 3) "For each puncture technique, the participants first watched a standardized instructional video, followed by a 5-minute practice session." (pp. 4-5) Detailed Analysis: Criterion C requires randomization at the class level (or stronger school level) to prevent contamination between treatment and control groups within a shared classroom. Here, randomization was performed at the individual student level using computer-generated random numbers with block size four and concealed allocation. Ordinarily student-level randomization within a single class fails criterion C. However, the standard provides an exception: when an intervention is designed for personal/individual teaching (e.g., tutoring), student-level randomization is acceptable. In this study the "intervention" is delivered as individual, one-participant-at-a-time skill practice on a physical phantom with a hardware feedback device; there is no shared classroom instruction through which a control student could absorb the treatment, and the tangible feedback system cannot be transmitted between participants. Contamination is therefore not a meaningful threat, and the design is analogous to personal/individual teaching. Criterion C is met because the individualized nature of the simulator-based skill training satisfies the personal-teaching exception that permits student-level randomization.
  • E

    Exam-based Assessment

    • Outcomes were custom study-specific measures (needle tip disappearance frequency, system-derived oscillation amplitude, and a blinded three-level safety rating), not a widely recognized standardized exam.
    • "The needle oscillation amplitude was defined as the maximum displacement amplitude in the y-axis ... of the needle coordinates automatically recorded by the YOLOv5 system."
    • Relevant Quotes: 1) "The outcomes were needle tip disappearance frequency compared between the control and YOLOv5 feedback groups and needle oscillation amplitude — the maximum lateral deviation from the ideal insertion line, automatically recorded by the system — compared between punctures classified as safe and unsafe by an independent blinded assessor." (Abstract, p. 1) 2) "The needle tip disappearance frequency was defined as the number of times the needle tip disappeared from the ultrasound image, and was compared between the two groups. The needle oscillation amplitude was defined as the maximum displacement amplitude in the y-axis (left-right direction perpendicular to the ultrasound beam) of the needle coordinates automatically recorded by the YOLOv5 system." (p. 5) 3) "An independent evaluator classified the safety of each puncture into three levels ... Level 1 (safe) ... Level 2 (moderate risk) ... Level 3 (unsafe)." (p. 5) Detailed Analysis: Criterion E requires a standardized, widely recognized exam-based assessment that was not created specifically for the study. The outcome measures here are entirely study-specific: a bespoke count of needle-tip disappearances, a novel system-derived "needle oscillation amplitude" metric computed by the authors' own YOLOv5 system, and a three-level puncture-safety rubric devised and applied by a single in-house blinded assessor. None of these are standardized, externally validated, or widely recognized examinations; they were designed by the authors for this study. Criterion E is not met because no standardized exam-based assessment was used; all outcomes were custom-built measures.
  • T

    Term Duration

    • The entire intervention and outcome measurement occurred within a single short session (5-minute practices and immediate tests), far shorter than one academic term.
    • "Participants were instructed to puncture the phantom and reach the embedded target within a 5-minute time limit."
    • Relevant Quotes: 1) "This single-center, prospective, pilot randomized controlled trial was conducted at Yamagata University Hospital between November 1 and November 30, 2024." (p. 2) 2) "Each included watching an instructional video, a 5-minute practice (with feedback for the YOLOv5 group or without for the control group), and a test puncture without feedback." (Abstract, p. 1) 3) "After the practice session, both groups performed the test without feedback ... Participants were instructed to puncture the phantom and reach the embedded target within a 5-minute time limit." (p. 5) Detailed Analysis: Criterion T requires that outcomes be measured at least one full academic term (approximately 3-4 months) after the intervention begins. In this study the intervention for each puncture technique consisted of watching a video plus a 5-minute practice, and the outcome test was performed immediately afterward within a 5-minute limit. The whole trial window spanned only November 2024, and each participant's involvement was effectively a single session. The interval from intervention start to outcome measurement is minutes, not a term. Criterion T is not met because outcomes were measured immediately after minutes-long practice sessions, far short of one academic term.
  • D

    Documented Control Group

    • The control group's size, age, sex, baseline (no prior UGRA experience), and condition (identical training without feedback) are documented.
    • "The median age was 23 years in both groups. The control group included eight males and four females, whereas the Y group included five males and six females."
    • Relevant Quotes: 1) "The study enrolled medical students with no prior experience in UGRA. ... Exclusion criteria included inability to provide informed consent and prior experience with ultrasound-guided needle procedures." (p. 3) 2) "Twenty-three participants were randomly allocated to either the control group (n=12) or the Y group (n=11). One participant in the control group was excluded before training because of equipment malfunction ... The final analysis included 22 participants: 11 and 11 in the control and Y groups, respectively. The median age was 23 years in both groups. The control group included eight males and four females, whereas the Y group included five males and six females." (p. 6) 3) "The control group practiced without feedback, whereas the Y group practiced with real-time feedback ... The researchers did not provide technical advice or verbal feedback to either group." (p. 5) Detailed Analysis: Criterion D requires the control group to be well documented, including composition, size, baseline characteristics, and the conditions it experienced. The paper reports the control group's size (n=11 analyzed, n=12 allocated with one exclusion explained), baseline homogeneity (all medical students with no prior UGRA experience), demographics (median age 23; eight males and four females), and its exact condition (identical instructional video and 5-minute practice, but without the feedback system and with no technical advice). The CONSORT flow diagram (Fig. 7) further documents allocation, exclusion, and analysis numbers. This is adequate documentation for comparison. Criterion D is met because the control group's size, demographics, baseline status, and business-as-usual condition are clearly documented.

• Level 2 Criteria

  • S

    School-level RCT

    • This was a single-center trial with randomization at the individual student level, not at the school/institution level.
    • "This single-center, prospective, pilot randomized controlled trial was conducted at Yamagata University Hospital."
    • Relevant Quotes: 1) "This single-center, prospective, pilot randomized controlled trial was conducted at Yamagata University Hospital between November 1 and November 30, 2024." (p. 2) 2) "Group allocation was performed using computer-generated random numbers with a block size of four." (p. 3) Detailed Analysis: Criterion S requires randomization at the school / institution / implementing-unit level, with multiple such units randomized. This study was conducted at a single hospital and randomized individual medical students, not schools or institutions. There is only one site and no unit-level randomization above the individual. Criterion S is not met because randomization occurred at the individual student level within a single center.
  • I

    Independent Conduct

    • The same team designed and developed the feedback system and also conducted the trial; although the outcome assessor was blinded, there was no independent third-party evaluation team.
    • "KK: Conceptualization, System development, Investigation, Data collection."
    • Relevant Quotes: 1) "We developed a YOLO-based needle tracking system designed to address the challenge of maintaining proper needle visualization among novices." (p. 2) 2) "MK: Conceptualization, Methodology, Investigation, Formal analysis, writing-original draft preparation. KK: Conceptualization, System development, Investigation, Data collection. TH: Supervision, Advice on system development. HT: Supervision, Writing - review and editing." (p. 9) 3) "The outcome assessment was independently performed by a single anesthesiologist (12 years of UGRA experience) who was blinded to the group allocation." (p. 5) Detailed Analysis: Criterion I requires that the study be conducted independently from those who designed the intervention, to reduce bias. Here the authors both developed the YOLOv5 feedback system (KK: "System development"; TH: "Advice on system development") and designed, ran, collected data for, and analyzed the trial (MK, KK: Investigation/Data collection/Formal analysis). The one element of independence is a blinded outcome assessor, but this is a single assessor within the study's own conduct, not an external, third-party evaluation team independent of the intervention developers. The design, delivery, and analysis were performed by the intervention's creators. Criterion I is not met because the intervention developers also designed, conducted, and analyzed the trial, with only internal blinded assessment rather than independent conduct.
  • Y

    Year Duration

    • Term Duration (T) is not met, so Year Duration cannot be met; the study lasted a single short session, nowhere near a year.
    • "Each included watching an instructional video, a 5-minute practice ... and a test puncture without feedback."
    • Relevant Quotes: 1) "This single-center, prospective, pilot randomized controlled trial was conducted at Yamagata University Hospital between November 1 and November 30, 2024." (p. 2) 2) "Each included watching an instructional video, a 5-minute practice (with feedback for the YOLOv5 group or without for the control group), and a test puncture without feedback." (Abstract, p. 1) Detailed Analysis: Criterion Y requires tracking of outcomes for at least 75% of a full academic year after the intervention begins. Per the criterion-specific rule, because criterion T (Term Duration) is not met, Y cannot be met. Independently, the intervention and its measurement occurred within a single session of minutes, with no longitudinal follow-up over months. Criterion Y is not met because T is not met and the study spanned only a single short session rather than an academic year.
  • B

    Balanced Control Group

    • Both groups received identical instructional videos and equal 5-minute practice time; the only difference is the real-time feedback, which is the explicit treatment variable being tested, so time and resources are balanced.
    • "The control group practiced without feedback, whereas the Y group practiced with real-time feedback provided by the developed training system. The researchers did not provide technical advice or verbal feedback to either group."
    • Relevant Quotes: 1) "For each puncture technique, the participants first watched a standardized instructional video, followed by a 5-minute practice session. During the practice, the participants were free to repeat the punctures as many times as they wished, with no limit on the number of attempts." (pp. 4-5) 2) "The control group practiced without feedback, whereas the Y group practiced with real-time feedback provided by the developed training system. The researchers did not provide technical advice or verbal feedback to either group." (p. 5) 3) "this pilot study had two objectives: (1) to preliminarily examine whether short-term training with this system improves puncture safety" (p. 2) Detailed Analysis: Criterion B compares the time, budget, and materials given to each group and asks whether any extra resource in the treatment group is matched, unless that extra resource is itself the explicit treatment variable. Both groups watched the same standardized instructional video and received the same 5-minute practice session and the same no-feedback test; instructional time and materials are therefore equivalent. The single differentiating input is the real-time visual/auditory feedback from the YOLOv5 system, which is precisely the intervention under investigation ("to examine whether short-term training with this system improves puncture safety"). Under the ERCT decision tree, when the additional resource is the integral treatment variable, the control may remain business-as-usual, and here it does so with matched time on task. No confounding extra time or budget was given beyond the feedback itself. Criterion B is met because practice time and materials were equal across groups and the only difference, the real-time feedback, is the explicit treatment variable being tested.

• Level 3 Criteria

  • R

    Reproduced

    • This is described as the first implementation of the system and a pilot study; an internet search found no independent replication by any other research team.
    • "As the first implementation of a system combining real-time quantitative needle trajectory recording with simultaneous visual and auditory feedback in UGRA training, this pilot study had two objectives."
    • Relevant Quotes: 1) "A system that combines real-time quantitative needle trajectory recording with simultaneous visual and auditory feedback has not been widely described in the context of UGRA training." (p. 2) 2) "As the first implementation of a system combining real-time quantitative needle trajectory recording with simultaneous visual and auditory feedback in UGRA training, this pilot study had two objectives." (p. 2) 3) "This pilot study provided preliminary evidence that needle oscillation amplitude has potential as a novel objective metric." (p. 9) Detailed Analysis: Criterion R requires independent replication of the specific study by a different research team in a different context, published in a peer-reviewed outlet. The authors explicitly frame this as the first implementation of the system and a preliminary pilot study. Internet searches (Google-style queries and BMC Anesthesiology / Springer listings, July 2026) for any team reproducing this YOLOv5 trajectory-feedback UGRA training trial and its needle-oscillation-amplitude metric returned no replication; only unrelated UGRA/AI simulation studies by other authors exist. Given the paper was published in May 2026 and describes a novel prototype, no independent reproduction has occurred. Criterion R is not met because the study is a first-of-its-kind pilot and no independent replication could be found.
  • A

    All-subject Exams

    • Criterion E is not met and the study measured only a single procedural-skill domain, so All-subject Exams cannot be met.
    • "The outcomes were needle tip disappearance frequency ... and needle oscillation amplitude."
    • Relevant Quotes: 1) "This pilot study had two objectives: (1) to preliminarily examine whether short-term training with this system improves puncture safety, and (2) to explore the utility of needle oscillation amplitude as a novel objective metric for assessing technical proficiency in UGRA training." (p. 2) 2) "The outcomes were needle tip disappearance frequency compared between the control and YOLOv5 feedback groups and needle oscillation amplitude." (Abstract, p. 1) Detailed Analysis: Criterion A requires that impact be measured across all main subjects using standardized exam-based assessments, and it explicitly depends on criterion E being met. Criterion E is not met here (no standardized exams were used). Moreover, the study measured a single, narrow procedural-skill domain (needle puncture safety in UGRA) and no other subjects. Criterion A is not met because E is not met and only one specialized skill domain was assessed.
  • G

    Graduation Tracking

    • Criterion Y is not met and there was no follow-up beyond the immediate post-practice test; an internet search found no subsequent graduation-tracking papers by the same authors.
    • "After the practice session, both groups performed the test without feedback, and needle position information and ultrasound images were recorded."
    • Relevant Quotes: 1) "After the practice session, both groups performed the test without feedback, and needle position information and ultrasound images were recorded." (p. 5) 2) "This study has some limitations. First, because the pilot study was conducted with a limited sample size, it may have lacked sufficient statistical power." (p. 8) Detailed Analysis: Criterion G requires that participants be tracked through to graduation to assess long-term impact, and it depends on criterion Y being met. Criterion Y is not met. Furthermore, outcomes were captured only in the immediate post-practice test within the single session; there is no follow-up over time, let alone tracking to graduation. Internet searches (July 2026) for any subsequent publication by Kurota, Kubota, and colleagues that tracks this student cohort through to graduation found none. Criterion G is not met because Y is not met and there was no longitudinal or graduation follow-up in this paper or any later publication.
  • P

    Pre-Registered

    • The trial was prospectively registered with the UMIN Clinical Trials Registry (UMIN000055602) on October 1, 2024, before data collection began on November 1, 2024.
    • "Trial registration UMIN Clinical Trials Registry (UMIN000055602), registered October 1, 2024."
    • Relevant Quotes: 1) "The research protocol and informed consent documents were approved by the Ethics Committee of the Yamagata University School of Medicine (approval number: 2024-225), and the trial was registered with the University Hospital Medical Information Network (UMIN) (UMIN000055602)." (p. 2) 2) "Trial registration UMIN Clinical Trials Registry (UMIN000055602), registered October 1, 2024." (p. 2) 3) "This single-center, prospective, pilot randomized controlled trial was conducted at Yamagata University Hospital between November 1 and November 30, 2024." (p. 2) Detailed Analysis: Criterion P requires that the study protocol be pre-registered before data collection began, on a recognized registry, with the registration date preceding data collection. The paper documents registration on the UMIN Clinical Trials Registry (UMIN000055602) on October 1, 2024, and the trial (data collection) ran between November 1 and November 30, 2024, so registration clearly precedes data collection. UMIN-CTR is an internationally recognized WHO primary registry, and an ethics-approved protocol (approval 2024-225) existed. A direct lookup of the exact UMIN000055602 record was not retrievable through open registry queries during this check, but the paper's explicit and consistent statements document prospective registration. Criterion P is met because the trial was prospectively registered on UMIN (UMIN000055602) on October 1, 2024, one month before data collection commenced on November 1, 2024.