Web-based pronunciation training and learning consonant clusters among EFL learners

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation was carried out at the individual student level within a single cohort at one university, not at the class or school level, and the study is not a personal tutoring intervention that would qualify for the exception.
    • "The participants were randomly assigned into two groups: experimental group and control group." (p. 3)
    • Relevant Quotes: 1) "These participants were randomly divided into two groups." (Abstract, p. 1) 2) "The participants were randomly assigned into two groups: experimental group and control group." (p. 3, Section 3.1 Study design) 3) "This research involved the participation of 58 Saudi students enrolled at a public university in Saudi Arabia, all of whom were native Arabic speakers... The participants were divided into two distinct groups." (p. 3, Section 3.2 Participants) 4) "During these sessions, the instructor accessed the YouGlish website, entered the target word, and projected the videos with the help of a speaker in the classroom... Each session involved the presentation of ten words, with five videos for each word." (p. 4, Section 3.6 Procedures) Detailed Analysis: The paper repeatedly describes randomisation of individual students into an "experimental group" (n=28) and a "control group" (n=30) drawn from a single pool of 58 students at one public university. There is no mention of randomising whole classes or schools; instead, individual students within what appears to be the same cohort/course were split into the two conditions. Both groups were then taught in group sessions (the instructor projected videos "in the classroom" for the experimental group, and the control group was taught via a group "listen and repeat" approach), meaning students in both conditions were physically present in group/classroom settings, raising the classic contamination concern the C criterion is designed to prevent. The exception for personal, one-to-one tutoring interventions does not apply here, since the intervention was delivered to groups of students in sessions, not as individual tutoring. Because randomisation occurred at the student level, without meeting the class/school-level requirement or the tutoring exception, criterion C is not met.
  • E

    Exam-based Assessment

    • The outcome measure was a researcher-assembled list of 20 words drawn from a course textbook and rated by two teachers on a custom binary scale, not a standardised, widely recognised exam.
    • "The study included three production assessments: a pretest, an immediate post-test, and a delayed post-test, each comprising 20 words. These words were drawn from a prescribed textbook used in an intensive English course..." (p. 3)
    • Relevant Quotes: 1) "The study included three production assessments: a pretest, an immediate post-test, and a delayed post-test, each comprising 20 words. These words were drawn from a prescribed textbook used in an intensive English course and were specifically chosen because they all featured consonant clusters, whether at the beginning, middle, or end of the word." (p. 3, Section 3.3 The pronunciation tests) 2) "Participants were instructed to orally pronounce and record these words as accurately as possible, using their mobile phones to capture the target words." (p. 3, Section 3.3) 3) "The participants' pronunciation attempts were assessed by two experienced EFL teachers who were not native speakers of the language. The evaluation process employed a binary rating system, where a score of 1 indicated correct pronunciation of the target word, and a score of 0 signified incorrect pronunciation." (p. 4, Section 3.5 Pronunciation evaluation (Rubric)) Detailed Analysis: The assessment instrument used in this study is a set of 20 target words selected by the researchers from a course textbook, specifically chosen to contain consonant clusters relevant to the study's aims. Scoring was performed by two teacher-raters using a study-specific binary correct/incorrect rubric devised by the authors, with inter-rater reliability reported via Cronbach's alpha. This is a bespoke, study-specific production test rather than a standardised, widely recognised examination such as a national curriculum test or a validated standardised language proficiency test. Because the assessment tool was custom-built for this study rather than a standardised exam, criterion E is not met.
  • T

    Term Duration

    • The intervention itself lasted only two short training sessions, and the paper does not state a clear date/interval showing outcomes were measured at least one academic term after the intervention began.
    • "The experimental group received the intervention for two sessions using YouGlish website for learning words' pronunciation. The control group received intervention for two sessions using the traditional way of teaching pronunciation." (p. 3)
    • Relevant Quotes: 1) "An experimental study design with pre-test, immediate post-test, and delayed posttest was used to achieve the aim of the study." (p. 3, Section 3.1) 2) "The experimental group received the intervention for two sessions using YouGlish website for learning words' pronunciation. The control group received intervention for two sessions using the traditional way of teaching pronunciation." (p. 3, Section 3.1) 3) "The study encompassed four distinct phases involving training implementation and data collection. In the initial stage, all participants underwent a pronunciation pre-test... Moving on to the second stage, the experimental group engaged in two training sessions... The third stage entailed the participation of all participants in an immediate post-test, while the fourth stage involved their involvement in a delayed post-test." (p. 4, Section 3.6 Procedures) 4) "The two major limitations of this study are the number of the training sessions as well as the number and location of participants. Two sessions for training may not be enough to train students with such aspects of pronunciation." (p. 6, Section 6.1) Detailed Analysis: The intervention consisted of only two training sessions per group. While the study does include a "delayed post-test" administered after the immediate post-test, no specific dates or explicit time interval (e.g., number of weeks/months between the end of the intervention and the delayed post-test) are provided anywhere in the Methods, Procedures, or Results sections. Without a quoted interval, it cannot be verified that outcomes were measured at least one academic term (~3-4 months) after the intervention began. Given the very short, session-based nature of the intervention (explicitly flagged by the authors themselves as a limitation), and the complete absence of any stated multi-month follow-up period, it is unlikely that a term-long interval was achieved, and it certainly cannot be confirmed from the text. Because no quote establishes a term-length interval between intervention start and outcome measurement, criterion T is not met.
  • D

    Documented Control Group

    • The control group's size, instructional protocol, and baseline (pretest) performance are clearly documented, allowing proper comparison with the experimental group.
    • "The second group, known as the control group, comprised 30 students. Their instruction involved listening to the instructor's pronunciation of words and then repeating after the instructor, with the instructor himself pronouncing the target words." (p. 3)
    • Relevant Quotes: 1) "The second group, known as the control group, comprised 30 students. Their instruction involved listening to the instructor's pronunciation of words and then repeating after the instructor, with the instructor himself pronouncing the target words." (p. 3, Section 3.2 Participants) 2) "Meanwhile, the control group received instruction on the same aspect (i.e., consonant cluster) in two sessions using conventional methods, which included the 'listen and repeat' approach. Participants in this group were instructed to listen to each word and then repeat it five times." (p. 4, Section 3.6 Procedures) 3) "Pretest Experimental 28 3.42 0.634 0.119 Control 30 3.20 0.761 0.138" (Table 2, p. 4) Detailed Analysis: The paper documents the control group's size (n=30), demographic profile shared with the whole sample (native Arabic speakers, ages 19-25, A2 CEFR proficiency, public university students), the exact instructional protocol they received ("listen and repeat" with the instructor pronouncing each word five times), and their baseline (pretest) performance with descriptive statistics (mean=3.20, SD=0.761) reported alongside the experimental group's baseline in Table 2. This level of detail is sufficient to assess comparability of the two groups at baseline and to understand exactly what the control condition entailed. Because the control group's composition, treatment, and baseline data are clearly documented, criterion D is met.

• Level 2 Criteria

  • S

    School-level RCT

    • The study was conducted within a single university with individual students randomised, not with randomisation across multiple schools/institutions.
    • "This research involved the participation of 58 Saudi students enrolled at a public university in Saudi Arabia..." (p. 3)
    • Relevant Quotes: 1) "This research involved the participation of 58 Saudi students enrolled at a public university in Saudi Arabia, all of whom were native Arabic speakers." (p. 3, Section 3.2 Participants) 2) "The participants were randomly assigned into two groups: experimental group and control group." (p. 3, Section 3.1) Detailed Analysis: The entire study sample of 58 students was drawn from a single public university, and randomisation occurred at the individual student level within that single institution. There is no mention of multiple schools/universities being randomised to conditions, nor any school-level unit of assignment. Because the study is a single-site, student-level design rather than a school-level RCT, criterion S is not met.
  • I

    Independent Conduct

    • There is no statement that data collection or analysis was conducted by an independent, external team; the same author team appears to have designed, implemented, and analysed the study.
    • "Yasser Alsuhaibani: Conceptualization. Hassan Saleh Mahdi: Formal analysis. Ahmed Al Khateeb: Writing – original draft. Hind Abdulaziz Al Fadda: Writing – review & editing. Hisham Alkadi: Writing – review & editing." (p. 6)
    • Relevant Quotes: 1) "Yasser Alsuhaibani: Conceptualization. Hassan Saleh Mahdi: Formal analysis. Ahmed Al Khateeb: Writing – original draft. Hind Abdulaziz Al Fadda: Writing – review & editing. Hisham Alkadi: Writing – review & editing." (CRediT authorship contribution statement, p. 6) 2) "During these sessions, the instructor accessed the YouGlish website, entered the target word, and projected the videos with the help of a speaker in the classroom." (p. 4, Section 3.6 Procedures) 3) "The participants' pronunciation attempts were assessed by two experienced EFL teachers who were not native speakers of the language." (p. 4, Section 3.5) Detailed Analysis: The CRediT statement shows the five named authors performed all core roles (conceptualization, formal analysis, writing), with no mention of an external or third-party organisation designing, delivering, or independently evaluating the intervention. The instructor who delivered the sessions and the two raters who scored pronunciation are not described as being independent of the research team, nor is there any statement of blinding or independent oversight. Because no quote indicates that an external, independent party conducted the evaluation, and the same team appears responsible for all stages of the study, criterion I is not met.
  • Y

    Year Duration

    • Because criterion T (Term Duration) is not met, per the ERCT standard's dependency rule, criterion Y (Year Duration) is automatically not met; in addition, no year-long tracking interval is described anywhere in the paper.
    • "The two major limitations of this study are the number of the training sessions as well as the number and location of participants. Two sessions for training may not be enough..." (p. 6)
    • Relevant Quotes: 1) "The experimental group received the intervention for two sessions using YouGlish website for learning words' pronunciation." (p. 3, Section 3.1) 2) "The third stage entailed the participation of all participants in an immediate post-test, while the fourth stage involved their involvement in a delayed post-test." (p. 4, Section 3.6) 3) "The two major limitations of this study are the number of the training sessions as well as the number and location of participants. Two sessions for training may not be enough to train students with such aspects of pronunciation." (p. 6, Section 6.1) Detailed Analysis: Per the ERCT criteria-specific instruction for Y, "If criterion T (Term Duration) is not met, then this criterion is not met." Since criterion T was not met (no documented term-length interval between intervention start and outcome measurement), Y cannot be met either. Independently, the paper contains no quote describing tracking of participants across a full academic year (~9-10 months, or at least 75%); the entire study apparently unfolded over a short period built around two training sessions and two/three testing points with no stated multi-month timeline. Because both the dependency rule on T and the absence of any year-long tracking evidence apply, criterion Y is not met.
  • B

    Balanced Control Group

    • Both groups received an equal number of training sessions (two) covering the same set of target words, differing only in instructional method, so the allocation of time between conditions was balanced.
    • "The experimental group received the intervention for two sessions using YouGlish website for learning words' pronunciation. The control group received intervention for two sessions using the traditional way of teaching pronunciation." (p. 3)
    • Relevant Quotes: 1) "The experimental group received the intervention for two sessions using YouGlish website for learning words' pronunciation. The control group received intervention for two sessions using the traditional way of teaching pronunciation." (p. 3, Section 3.1 Study design) 2) "During these sessions, the instructor accessed the YouGlish website, entered the target word, and projected the videos with the help of a speaker in the classroom... Each session involved the presentation of ten words, with five videos for each word." (p. 4, Section 3.6 Procedures) 3) "Meanwhile, the control group received instruction on the same aspect (i.e., consonant cluster) in two sessions using conventional methods, which included the 'listen and repeat' approach. Participants in this group were instructed to listen to each word and then repeat it five times." (p. 4, Section 3.6 Procedures) Detailed Analysis: Applying the criterion B decision tree: the two conditions received the same number of sessions (two) and covered the same set of target words (ten words per session, twenty words total), differing only in the mode of instruction — web-based videos via YouGlish for the experimental group versus instructor-led "listen and repeat" for the control group. Neither group received additional class time, extra sessions, or extra budget relative to the other; the comparison is between two instructional methods delivered within matched time allocations. Since no extra time or resources were given to one condition over the other (EXTRA_RESOURCES_PRESENT is effectively false/ negligible), the groups are considered balanced under the standard's decision tree. Because both conditions received matched instructional time and content, differing only in teaching method, criterion B is met.

• Level 3 Criteria

  • R

    Reproduced

    • A targeted internet search found no independent replication of this specific study by a different research team; the paper is itself a recent original study with no reported replications yet.
    • Relevant Quotes: (No quotes describing replication of this specific study were found in the paper; the introduction and discussion instead cite prior, distinct studies on web-based pronunciation training in general, e.g., "web-based pronunciation lessons have been shown to help EFL learners improve their articulation of problematic English sounds (Huang et al., 2023; Mompean & Fouz-Gonzalez, 2016; Nasim et al., 2022; Prahaladaiah & Andrew Thomas, 2024; Yan & Lin, 2019)." (p. 3, Section 2.3)) Detailed Analysis: This paper (published August 2024 in Acta Psychologica, DOI 10.1016/j.actpsy.2024.104459) does not report or reference any independent replication of its own specific design (YouGlish- based consonant-cluster pronunciation training compared to instructor-led listen-and-repeat, in Saudi EFL university learners). An internet search (general web search, PubMed, and ScienceDirect listings for this article and its citing works) for papers by other, independent research teams reproducing this specific experimental design and comparing results found none; the results returned only describe or summarise this same original study, or cite it as background for unrelated pronunciation-technology research. The related studies cited within the paper itself are prior work on different interventions and populations, not replications of this study's specific design and findings. Because no independent replication of this specific study was found either in the paper or via internet search, criterion R is not met.
  • A

    All-subject Exams

    • Because criterion E (Exam-based Assessment) is not met, criterion A is automatically not met; in addition, only pronunciation of consonant-cluster words was assessed, not other core subjects.
    • "The study included three production assessments: a pretest, an immediate post-test, and a delayed post-test, each comprising 20 words." (p. 3)
    • Relevant Quotes: 1) "The study included three production assessments: a pretest, an immediate post-test, and a delayed post-test, each comprising 20 words. These words were drawn from a prescribed textbook used in an intensive English course and were specifically chosen because they all featured consonant clusters..." (p. 3, Section 3.3) Detailed Analysis: Per the ERCT criteria-specific instruction for A, "If criterion E (Exam-based Assessment) is not met, then this criterion is not met either." Since criterion E was not met (the assessment was a custom, non-standardised production test), criterion A cannot be met. Furthermore, the study only measured pronunciation of consonant-cluster words in English (a narrow language-learning outcome), with no assessment of other subjects or broader academic exams. Because the E prerequisite fails and only a single, narrow outcome domain was assessed, criterion A is not met.
  • G

    Graduation Tracking

    • Because criterion Y (Year Duration) is not met, criterion G is automatically not met; an internet search also found no follow-up papers by the same author team tracking this cohort toward graduation.
    • "The third stage entailed the participation of all participants in an immediate post-test, while the fourth stage involved their involvement in a delayed post-test." (p. 4)
    • Relevant Quotes: 1) "The third stage entailed the participation of all participants in an immediate post-test, while the fourth stage involved their involvement in a delayed post-test." (p. 4, Section 3.6 Procedures) 2) "Further studies are indeed needed involving larger numbers of participants from various backgrounds with more training sessions to examine the effect effectiveness of tool in hand in a global scale." (p. 6, Section 6.1 Study limitations) Detailed Analysis: Per the ERCT criteria-specific instruction for G, "If criterion Y (Year Duration) is not met, then this criterion is not met." Since Y was not met, G cannot be met. Independently, the study's data collection ends at a "delayed post-test" shortly after the two-session intervention, with no mention of any follow-up through to the end of the participants' university studies/graduation, and no indication of planned or published follow-up papers tracking this cohort. An internet search for follow-up publications by the same five authors (Alsuhaibani, Mahdi, Al Khateeb, Al Fadda, Alkadi) tracking this same cohort of 58 students found no such papers; no subsequent graduation-tracking study referencing this 2024 cohort was located. Because the Y prerequisite fails and there is no evidence of graduation tracking either in the paper or via internet search, criterion G is not met.
  • P

    Pre-Registered

    • The paper contains no statement of pre-registration of the study protocol on any registry platform, and an internet search of common registries found no matching pre-registration record.
    • Relevant Quotes: (No quotes referencing a pre-registration platform, registry ID, or pre-registration date were found anywhere in the paper, including the Methods, Declaration of competing interest, Data availability, and Acknowledgements sections.) 1) "Data availability: Data will be made available on request." (p. 6) 2) "Declaration of competing interest: The authors have no conflict of interest." (p. 6) Detailed Analysis: A review of the Methods section and all end-matter sections (Data availability, Declaration of competing interest, Acknowledgements) reveals no mention of a study protocol registry (e.g., OSF, AsPredicted, ClinicalTrials.gov, ISRCTN), no registration ID, and no registration date. An internet search (general web search plus targeted queries referencing this study's title and authors) found no matching pre-registration record on any registry platform. There is no evidence the hypotheses, methods, or planned analyses were published before data collection began. Because no evidence of pre-registration is present in the paper or found via internet search, criterion P is not met.