Click, Watch, and Speak: The Effect of YouGlish on Kuwaiti EFL Students' Speaking Achievement

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation was carried out at the individual-student level within a single speaking course cohort, not at the class or school level, and no tutoring exception applies.
    • "They were randomly assigned to experimental and control groups, a common approach in educational research when random sampling is not feasible."
    • Relevant Quotes: 1) "In the quantitative phase, a quasi-experimental approach was implemented with 84 undergraduate students from Kuwait University, divided into experimental and control groups." (p. 275) 2) "In the quantitative phase, 84 students were selected through convenience sampling ... based on their enrolment in a speaking course and availability for the intervention. They were randomly assigned to experimental and control groups, a common approach in educational research when random sampling is not feasible (Creswell & Creswell, 2023)." (p. 275) Detailed Analysis: The unit of randomisation was the individual student, not an entire class or school. All 84 participants were drawn from the same speaking course in the Department of English Language and Literature at Kuwait University and individually assigned to experimental or control condition. There is no description of separate intact classes or sections being randomised as whole units, and the intervention is a group speaking course rather than one-to-one tutoring, so the personal-tutoring exception does not apply. Randomising individuals drawn from the same course pool creates a risk of contamination between conditions (e.g., shared materials or peer interaction outside assigned sessions), which is exactly the problem this criterion is designed to guard against. Because randomisation occurred at the student level without a valid exception, criterion C is not met.
  • E

    Exam-based Assessment

    • A researcher-designed speaking test only "adapted from" IELTS descriptors was used, not the standardised IELTS exam itself.
    • "The assessment of speaking achievement was conducted using a customized rubric adapted from the IELTS Speaking Band Descriptors."
    • Relevant Quotes: 1) "In the quantitative phase, a speaking achievement test was administered to evaluate the effectiveness of using YouGlish compared to traditional instruction (Appendix A). Adapted from the IELTS speaking test, it assessed four key components: fluency and coherence, lexical resource, grammatical range and accuracy, and pronunciation." (p. 276) 2) "The assessment of speaking achievement was conducted using a customized rubric adapted from the IELTS Speaking Band Descriptors, focusing on fluency and coherence, lexical resource, grammatical range and accuracy, and pronunciation. Drawing also on the Common European Framework of Reference (CEFR), the rubric ensured comprehensive and standardized evaluation..." (pp. 276-277) 3) "The test was rated by the researcher and two experienced IELTS examiners to ensure balanced judgment, with independent scoring applied to reduce bias." (p. 276) Detailed Analysis: The instrument used to measure the outcome was a study-specific test (Appendix A) that the authors describe as "adapted from" and "customized" using IELTS band descriptors and CEFR, not the official standardised IELTS speaking exam administered under its own protocol. Having experienced IELTS raters score the custom test strengthens rater reliability, but it does not convert a researcher-built instrument into a widely recognised, externally validated standardised exam; the prompts and content (themes such as travel, technology, friendship, professional communication) were built for this course and study. Because the assessment tool is a custom-made instrument rather than a standard, widely recognised exam, criterion E is not met.
  • T

    Term Duration

    • No clear intervention start date or measurement date is given, so the interval to at least one academic term cannot be verified from the text.
    • "Weekly lessons focused on specific themes—travel, technology, friendship, and professional communication—incorporating debates, dialogues, and presentations to promote coherence and fluency."
    • Relevant Quotes: 1) "Weekly lessons focused on specific themes—travel, technology, friendship, and professional communication—incorporating debates, dialogues, and presentations to promote coherence and fluency." (p. 276) 2) "My pronunciation improved a lot. I also feel I speak more naturally after using YouGlish in this semester." (Participant 1, pp. 279-280) Detailed Analysis: The Methodology section never states an explicit intervention start date, end date, or total duration in weeks or months, nor does it give an explicit date for when the post-test/outcome measurement occurred relative to the start. The only indirect evidence of duration is the phrase "weekly lessons" built around four rotating themes (implying at least several weeks) and a single interview participant's passing mention of "this semester." Neither of these constitutes a clear, quotable statement of the intervention's calendar length or of the interval between intervention start and outcome measurement, as the criterion procedure requires. Because the paper does not clearly document the intervention-to-measurement interval, it cannot be confirmed that it reached at least one full academic term, so criterion T is not met on documentation grounds.
  • D

    Documented Control Group

    • The control condition's activities and group size are described, though group-specific baseline demographics are not reported.
    • "The experimental group practiced by listening, recording, and tracking their speech using YouGlish, while the control group followed traditional methods such as printed materials and peer–instructor interactions."
    • Relevant Quotes: 1) "The experimental group practiced by listening, recording, and tracking their speech using YouGlish, while the control group followed traditional methods such as printed materials and peer–instructor interactions." (pp. 275-276) 2) "Descriptive statistics for overall post-test speaking achievement by group... Control 70.5476 5.47813 42" (Table 2, p. 278) 3) "They were randomly assigned to experimental and control groups..." (p. 275) Detailed Analysis: The paper describes what the control condition actually did (traditional printed materials and peer–instructor interaction, i.e. business-as-usual instruction) and documents its size (N = 42) along with post-test descriptive statistics in Table 2. This gives readers a basic picture of the control group's composition, size, and treatment. The paper does not, however, report separate baseline demographic characteristics (age, gender, prior proficiency) split by group, nor group-level pre-test descriptive statistics in the text (only the aggregate pre-test covariate is reported in the ANCOVA). This is a moderate, non-exhaustive level of documentation. On balance, since the control condition's activities and size are clearly stated, criterion D is met, albeit with the limitation that detailed baseline demographics for the control group specifically are not separately reported.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation occurred at the individual-student level within one department, not at the school level.
    • "They were randomly assigned to experimental and control groups, a common approach in educational research when random sampling is not feasible."
    • Relevant Quotes: 1) "In the quantitative phase, 84 students were selected through convenience sampling ... They were randomly assigned to experimental and control groups..." (p. 275) 2) "This study was conducted in the Department of English Language and Literature at the College of Arts, Kuwait University." (p. 275) Detailed Analysis: Criterion S requires randomisation among whole schools (or equivalent institutional units), not classes or students. Here the entire study took place within a single department of a single university, with individual students assigned to condition. This is weaker than even a class-level RCT, so it clearly does not meet the stronger school-level requirement. Criterion S is not met.
  • I

    Independent Conduct

    • The same research team designed and delivered the intervention and the lead researcher was directly involved in scoring outcomes; no independent external evaluator is described.
    • "The test was rated by the researcher and two experienced IELTS examiners to ensure balanced judgment, with independent scoring applied to reduce bias."
    • Relevant Quotes: 1) "The instructor, trained by an applied linguistics expert, used YouGlish to develop students' pronunciation, vocabulary, and fluency through exposure to authentic native speaker input." (p. 275) 2) "The test was rated by the researcher and two experienced IELTS examiners to ensure balanced judgment, with independent scoring applied to reduce bias." (p. 276) 3) "The speaking achievement test was evaluated by the researcher and two experienced IELTS raters..." (p. 276) Detailed Analysis: Criterion I requires that the evaluation be conducted independently of those who designed/delivered the intervention. Here, "the researcher" (i.e., a member of the study team, most plausibly the author who also designed the study) directly participated in scoring the outcome measure alongside two IELTS raters. There is no statement that data collection, scoring, or analysis was handed to a fully external, independent evaluation body unaware of condition assignment, unlike the qualifying exceptions in the ERCT standard (e.g., external agencies or blinded independent teams). The involvement of the researcher in scoring introduces a plausible source of bias. Because the research team itself was directly involved in outcome measurement without documented independent oversight, criterion I is not met.
  • Y

    Year Duration

    • Criterion T is not met, so criterion Y is automatically not met; no evidence of year-long tracking is provided in any case.
    • "Weekly lessons focused on specific themes—travel, technology, friendship, and professional communication..."
    • Detailed Analysis: Per the ERCT standard's dependency rule, criterion Y cannot be met when criterion T is not met. Independently of that rule, the paper contains no quote describing outcome measurement occurring 75% or more of an academic year after the intervention began; the described program appears limited to a course-length period with no explicit year-long tracking. Criterion Y is not met.
  • B

    Balanced Control Group

    • Both groups followed the same weekly course structure; the specific tool (YouGlish vs. traditional materials) is the treatment variable being compared, with no evidence of extra time or budget for the experimental group.
    • "The experimental group practiced by listening, recording, and tracking their speech using YouGlish, while the control group followed traditional methods such as printed materials and peer–instructor interactions."
    • Relevant Quotes: 1) "The experimental group practiced by listening, recording, and tracking their speech using YouGlish, while the control group followed traditional methods such as printed materials and peer–instructor interactions." (pp. 275-276) 2) "Weekly lessons focused on specific themes—travel, technology, friendship, and professional communication—incorporating debates, dialogues, and presentations to promote coherence and fluency." (p. 276) Detailed Analysis: Applying the criterion B decision procedure: the study compares two instructional materials/approaches (YouGlish videos vs. printed materials and peer–instructor interaction) delivered within what is described as the same weekly lesson schedule and the same set of themes for both groups. There is no statement that the experimental group received additional class time, extra sessions, or extra budget beyond the control group; both groups appear to receive a comparable amount of course time, differing only in the specific tool/material used. The tool itself (YouGlish) is the explicit variable being tested against a business-as-usual comparison (traditional printed materials), which is the core research question of the study. Since no extra time or budget is documented for the experimental condition beyond the standard course session, and the differing tool is the treatment variable being investigated against a comparable business-as-usual control, criterion B is met.

• Level 3 Criteria

  • R

    Reproduced

    • The authors present this as the first study of YouGlish in the Kuwaiti EFL context; a targeted internet search (Semantic Scholar, general web) found zero citing works and no independent replication of this specific study.
    • "no empirical study has yet examined its impact on speaking achievement among Kuwaiti EFL learners."
    • Relevant Quotes: 1) "Although various educational technologies have been used in EFL contexts, no research has investigated the use of YouGlish in speaking courses in the Kuwaiti EFL context." (p. 272) 2) "Although YouGlish has demonstrated strong potential in diverse EFL environments, no empirical study has yet examined its impact on speaking achievement among Kuwaiti EFL learners." (p. 273) Detailed Analysis: The authors explicitly frame this study as filling a previously unaddressed gap, i.e., the first study of YouGlish's effect on speaking achievement among Kuwaiti EFL learners. No mention is made in the paper of an independent replication of this specific study by another research team, and given the paper's own claim of novelty, no such replication could plausibly exist yet. The other YouGlish studies cited (e.g., Fu & Yang, 2019; Sahboun et al., 2023; Quispe Vargas et al., 2024) are prior, different studies in other contexts that this paper itself builds on, not independent replications of this particular Kuwaiti study. Independent internet verification: the article was published 12 June 2026 (DOI 10.6007/IJARBSS/v16-i6/28217). A Semantic Scholar lookup for this DOI on 2026-07-28 returned a citation count of 0 and an empty citing-papers list, and general web searches for independent replications of this specific Kuwaiti-university YouGlish speaking-achievement study returned no results. Given the paper is only about six weeks old as of this check, an independent peer-reviewed replication is not expected to exist yet. Criterion R is not met.
  • A

    All-subject Exams

    • Criterion E is not met, so criterion A is automatically not met; only a single skill domain (speaking) was assessed in any case.
    • "The speaking achievement test was used to collect the quantitative data before and after the intervention..."
    • Detailed Analysis: Per the ERCT standard's dependency rule, criterion A cannot be met when criterion E is not met, since A requires standardised exam-based assessment across all main subjects. Independently, the study measured only English speaking achievement (fluency and coherence, lexical resource, grammar, pronunciation) via a researcher-built test, with no assessment of other core subjects such as mathematics or science. Criterion A is not met.
  • G

    Graduation Tracking

    • Criterion Y is not met, so criterion G is automatically not met; the study only reports immediate post-test results with no long-term follow-up, and no subsequent publication tracking this cohort was found.
    • "Future research should examine the long-term effects of YouGlish and similar technology-based tools on EFL learners' speaking development..."
    • Relevant Quotes: 1) "Future research should examine the long-term effects of YouGlish and similar technology-based tools on EFL learners' speaking development, focusing on sustained improvements in pronunciation, fluency, and grammatical accuracy." (p. 283) Detailed Analysis: Per the ERCT standard's dependency rule, criterion G cannot be met when criterion Y is not met. Independently, the paper reports only pre- and post-intervention speaking scores with no tracking of students beyond the course period, and the authors explicitly recommend future research into long-term effects, confirming that no such follow-up was performed in this study. Independent internet verification: a Semantic Scholar lookup for this paper's DOI on 2026-07-28 shows zero citing works, and no follow-up publication by Tazang or Adi Kasuma tracking this same cohort to graduation was found. Given the six-week gap since publication, this is expected rather than informative. Criterion G is not met.
  • P

    Pre-Registered

    • No mention of pre-registration of the study protocol appears anywhere in the paper, and no registry record was located.
    • Detailed Analysis: A full review of the Methodology, Context and Sample, Data Collection, and ethics-related passages found no reference to a trial registry, a pre-registration platform, a registration ID, or a registration date. The only procedural safeguards mentioned are institutional approval and informed consent, which do not constitute pre-registration of hypotheses, methods, and planned analyses. Independent internet verification: searches for a pre-registration record (e.g., OSF, AsPredicted) linked to this study or its authors returned no results, consistent with the paper's own silence on pre-registration. Since no quote referencing pre-registration was found, criterion P is not met.