The Efficacy of Phonics-Based Instruction of English as a Second Language in an Italian High School: A Randomised Controlled Trial

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Students from various classes were individually randomly assigned to the trial or control cohort, not entire classes or schools, and the exception for one-to-one tutoring does not apply since this was group classroom instruction.
    • "The students came from a variety of different classes and were randomly assigned to the trial lessons by the English teacher organising the course and the school administrative staff." (p. 40)
    • Relevant Quotes: 1) "The study groups were initially between 17–24 (both trial n= 24 and control n=17), 17-18 year-olds from the same school, randomly assigned to each group by the school." (p. 40) 2) "The students came from a variety of different classes and were randomly assigned to the trial lessons by the English teacher organising the course and the school administrative staff." (p. 40) 3) "Trial lessons consisted of 10 two-hour extra-curriculum sessions." (p. 40) Detailed Analysis: The ERCT 'C' criterion requires randomisation at the class (or stronger school) level, with an exception only for one-to-one tutoring interventions. The paper explicitly states that individual students, drawn "from a variety of different classes," were separately assigned to the trial or control cohort. This is student-level randomisation within a single school, not class-level or school-level randomisation. The intervention itself was a group extra-curriculum course (10 two-hour sessions taught to a group of up to 24 students), not personal one-to-one tutoring, so the tutoring exception described in the standard does not apply. Because randomisation occurred at the individual student level rather than at the class or school level, and no tutoring exception applies, this criterion is not satisfied.
  • E

    Exam-based Assessment

    • The orthography, pronunciation, and interview tests were all custom-designed by the researchers specifically for this trial, rather than being widely recognised standardised assessments.
    • "Calini Preliminary Language Test Phonics: Teacher's copy." (Appendix 2, p. 62)
    • Relevant Quotes: 1) "Calini Preliminary Language Test Phonics: Teacher's copy." (Appendix 2, p. 62) 2) "The orthography test consisted in an examiner reading out aloud twice, a series of 32 sentences, each of which contained three or four elements of one phonics pattern." (p. 43) 3) "The pronunciation test consisted of each student reading sentences aloud, and was assessed on the same series of 32 sentences used for the orthography test." (p. 45) 4) "The interview was not a duplicate of the FCE but was a shorter interview based on personal information, which is available to the general public on the FCE website (Cambridge, 2016), so it cannot be claimed that the results would predict final FCE test results." (p. 50) Detailed Analysis: The ERCT 'E' criterion requires a widely recognised, standardised exam rather than a test custom-built for the study. Here, the orthography and pronunciation tests are a bespoke "Calini Preliminary Language Test Phonics," composed of 32 sentences specifically constructed by the researchers to target the exact phonics patterns taught in the trial course. This is not a standard, externally validated instrument. The interview, while loosely modelled on Cambridge FCE marking categories, is explicitly stated by the authors not to replicate the FCE oral exam and not to predict FCE outcomes, confirming it is also a custom, non-standardised measure. Because none of the outcome measures are widely recognised standardised exams, this criterion is not met.
  • T

    Term Duration

    • Outcomes were measured immediately at the end of the 10-week, 20-hour course, well short of the minimum one-term interval required from intervention start to measurement.
    • "Trial lessons consisted of 10 two-hour extra-curriculum sessions." (p. 40)
    • Relevant Quotes: 1) "A trial group of 24, 17-18 year-olds followed a short 10-week, 20-hour trial course..." (Abstract, p. 29) 2) "Trial lessons consisted of 10 two-hour extra-curriculum sessions." (p. 40) 3) "In a total final cohort of 38 high school students preparing for the FCE exam during a 20-hour preparatory course, the trial teaching methodology was shown to be statistically effective..." (p. 52) Detailed Analysis: The ERCT 'T' criterion requires that outcomes be measured at least one full academic term (roughly 3-4 months) after the intervention begins. The paper describes a compact 10-week, 20-hour extra-curriculum course, with pre- and post-tests administered at the start and immediately at the end of this short course. There is no indication of any follow-up assessment beyond the immediate post-test, and 10 weeks falls short of the 3-4 month term benchmark, with no term-long tracking after the intervention concluded. Because the interval from intervention start to final measurement was only about 10 weeks with no further tracking, this criterion is not met.
  • D

    Documented Control Group

    • The paper documents the control group's size, demographics, ability level, dropout, and the standard FCE exam-practice content it received.
    • "A trial group of 24, 17-18 year-olds followed a short 10-week, 20-hour trial course and were compared to a control (14 students) preparing for the Cambridge First Certificate exam." (Abstract, p. 29)
    • Relevant Quotes: 1) "The study groups were initially between 17–24 (both trial n= 24 and control n=17), 17-18 year-olds from the same school, randomly assigned to each group by the school. There was a slight majority of female students and all students were pre-tested at B2 level..." (p. 40) 2) "As the study classes were optional, attendance was not compulsory and there was a certain degree of random dropout (final numbers: trial n= 23, control n=14)." (p. 40) 3) "The content of the control course was dictated by the content of the exam practice book and on-line material available (Cambridge ESOL, 2016) for the FCE exam." (p. 49) 4) "Both control and trial teachers had passed through the schools' selection process, had the same amount of experience teaching these courses." (p. 41) Detailed Analysis: The ERCT 'D' criterion requires clear documentation of the control group's characteristics, size, and the treatment it received. The paper reports the control group's initial and final sample sizes (n=17, final n=14), its shared demographic profile with the trial group (age, sex ratio, B2 English level, same school), and the exact nature of its "business as usual" instruction (standard FCE exam practice material, taught by an equally experienced native-speaker teacher). Because the control group's size, characteristics, and instructional content are clearly described, this criterion is met.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation occurred at the individual student level within one school, not at the school level, so the stronger school-level requirement is not satisfied.
    • "The students came from a variety of different classes and were randomly assigned to the trial lessons by the English teacher organising the course and the school administrative staff." (p. 40)
    • Relevant Quotes: 1) "The study groups were initially between 17–24 (both trial n= 24 and control n=17), 17-18 year-olds from the same school, randomly assigned to each group by the school." (p. 40) 2) "The students came from a variety of different classes and were randomly assigned to the trial lessons by the English teacher organising the course and the school administrative staff." (p. 40) Detailed Analysis: The ERCT 'S' criterion requires randomisation at the school level, i.e. entire schools assigned to condition. This study took place at a single school, with individual students from different classes separately allocated to the trial or control cohort. There is no cluster of multiple schools and no school-level unit of randomisation. Because only a single school was involved and randomisation occurred at the student level within it, this criterion is not met.
  • I

    Independent Conduct

    • No statement in the paper indicates that data collection or analysis was conducted by an external, independent team separate from the researchers who designed the intervention.
    • Relevant Quotes: 1) "Both the trial and the control lessons were taught by two NS teachers. Both control and trial teachers had passed through the schools' selection process, had the same amount of experience teaching these courses." (p. 41) 2) "All statistic calculations used the applicative program (primer) Statistica per Discipline Biomediche..." (p. 40) 3) The Authors section describes Robert Coates, Judith Gorham and Richard Nicholas as the paper's designers, writers and (implicitly, through first-person "we") the researchers who designed the trial materials, conducted the statistical analysis, and wrote up the results. (p. 60) Detailed Analysis: The ERCT 'I' criterion requires that the study be conducted independently of the people who designed the intervention, typically evidenced by an external evaluation team handling data collection or analysis. Nowhere in the methods, discussion, or acknowledgements does the paper state that an independent third party collected data, ran the statistical tests, or otherwise evaluated the trial. The first-person plural ("we") is used throughout for designing the course, the tests, and performing the ANOVA/t-test analysis, strongly suggesting the same author team who created the phonics intervention also ran and analysed the trial themselves. Because there is no evidence of independent conduct or external evaluation, this criterion is not met.
  • Y

    Year Duration

    • Since criterion T (Term Duration) is not met, and the 10-week course also falls far short of 75% of an academic year, criterion Y cannot be met.
    • "Trial lessons consisted of 10 two-hour extra-curriculum sessions." (p. 40)
    • Relevant Quotes: 1) "A trial group of 24, 17-18 year-olds followed a short 10-week, 20-hour trial course..." (Abstract, p. 29) 2) "Trial lessons consisted of 10 two-hour extra-curriculum sessions." (p. 40) Detailed Analysis: Per the ERCT standard's criteria-specific instructions, if criterion T (Term Duration) is not met, criterion Y (Year Duration) is automatically not met. Independently, the 10-week course with immediate post-testing is also far short of 75% of an academic year (~9-10 months), since there is no indication of any tracking beyond the end of the short course. Because criterion T is not met, and the actual duration is far below the year-long requirement, criterion Y is not met.
  • B

    Balanced Control Group

    • Both cohorts received the same total instructional time (ten two-hour sessions); the trial group's phonics content substituted for, rather than added to, the control's exam-practice time.
    • "In the trial class, phonics techniques replaced the oral exam practice components and consisted of at least 60 minutes of the two-hour lessons." (p. 41)
    • Relevant Quotes: 1) "Trial lessons consisted of 10 two-hour extra-curriculum sessions. All students followed their regular English lessons during class hours." (p. 40) 2) "In the trial class, phonics techniques replaced the oral exam practice components and consisted of at least 60 minutes of the two-hour lessons. It should be noted that all students were still given other exam preparation tasks." (p. 41) 3) "Furthermore, both cohorts received approximately the same amount of time spent on the oral skills inherent in the pronunciation and interview FCE tests." (p. 41) Detailed Analysis: Applying the ERCT criterion B decision tree: the first question is whether the intervention group received extra time or budget compared to control. Here, both cohorts attended the identical ten two-hour extra-curriculum sessions (twenty hours total each); within the trial lessons, phonics techniques simply replaced the oral exam-practice component rather than being added on top of it, and both groups received comparable time on pronunciation/interview practice. Since no extra time or budget was given to the trial group (only a substitution of content within the same session length, with the control cohort if anything receiving more direct exam practice), the EXTRA_RESOURCES_PRESENT condition is false and the criterion is met at the first branch of the decision tree without needing to assess whether the substituted content was integral to the treatment. Because the intervention and control groups received equal instructional time and resources, differing only in content, this criterion is met.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this study by a different research team was found in any searched source; this appears to remain a single, small pilot study.
    • Relevant Quotes: 1) "We believe that the results could be the basis of more widespread phonics/prosody based empirical tests for a wider L2 population." (p. 52) 2) "This would need to be examined, in more extensive and intensive research than this current work was designed to fulfil." (p. 49) Detailed Analysis: The ERCT 'R' criterion requires that the specific study be independently replicated by a different research team in a different context, published in a peer-reviewed journal. The authors themselves frame this as a small pilot study whose findings "would need to be verified on a larger scale," and explicitly call for future, larger-scale research rather than pointing to any existing replication. Internet searches (web search across academic indexes, Semantic Scholar, ResearchGate, and the publisher's own journal site) for citing works, replication studies, or subsequent trials of this specific "Calini Preliminary Language Test Phonics" protocol with Italian L1 high-school ESL students found no independent replication by a different research team. The only records located are the original journal record (Gist Education and Learning Research Journal, DOI 10.26817/16925777.390) and its mirrored listing under an alternate title in the same journal's system, both by the same author team (Coates, Gorham, Nicholas). No other authors were found to have replicated this trial's design or findings. Because no independent replication of this study exists in any searched source, this criterion is not met.
  • A

    All-subject Exams

    • Only English language outcomes (orthography, pronunciation, interview) were assessed, not other main subjects, and criterion E (a prerequisite) is also not met.
    • "Results showed that the trial-group significantly improved in both orthography (p=0.048) and pronunciation (p=0.000)..." (Abstract, p. 29)
    • Relevant Quotes: 1) "Results showed that the trial-group significantly improved in both orthography (p=0.048) and pronunciation (p=0.000), in particular in the long vowel and digraph categories." (Abstract, p. 29) 2) "Furthermore, they significantly improved in a shortened interview category (p=0.024), for lexis, discourse and pronunciation." (Abstract, p. 29) Detailed Analysis: Per the ERCT standard's criteria-specific instructions, criterion A cannot be met unless criterion E is met, and E was found not met above (no standardised exam was used). Independently, the study's outcome measures (orthography, pronunciation, and an interview covering grammar, discourse, lexis and pronunciation) are all confined to the single subject area of English language learning; no other core school subjects (e.g. maths, science) were assessed. Because criterion E is not met and only a single subject area was assessed, this criterion is not met.
  • G

    Graduation Tracking

    • The authors explicitly state they did not follow up students after the trial, criterion Y (a prerequisite) is also not met, and no subsequent graduation-tracking publication by the same authors was found.
    • "We did not follow-up the students to understand their real FCE results as their final result would have involved too many other variables to be relevant for this research." (p. 50)
    • Relevant Quotes: 1) "We did not follow-up the students to understand their real FCE results as their final result would have involved too many other variables to be relevant for this research." (p. 50) 2) "In a total final cohort of 38 high school students preparing for the FCE exam during a 20-hour preparatory course..." (p. 52) Detailed Analysis: Per the ERCT standard's criteria-specific instructions, if criterion Y (Year Duration) is not met, criterion G cannot be met either, and Y was found not met above. Independently, the authors explicitly confirm that no follow-up was conducted after the short course ended, with tracking stopping at the immediate post-test. Internet searches for subsequent publications by Robert Coates, Judith Gorham, or Richard Nicholas tracking this same cohort of Italian high-school students toward graduation (via Semantic Scholar, Google-style search, and the authors' institutional profiles at Università Bocconi) did not locate any follow-up paper reporting graduation outcomes for these students. No such quotes could be found, and none are fabricated here. Because criterion Y is not met, the authors state no follow-up occurred, and no later graduation- tracking publication was found, this criterion is not met.
  • P

    Pre-Registered

    • There is no statement anywhere in the paper of a pre-registered protocol, registry platform, or registration date, and no internet search located a pre-registration record for this trial.
    • Relevant Quotes: No quotes referencing pre-registration, a trial registry (e.g. ClinicalTrials.gov, ISRCTN, OSF), or a published pre-specified analysis plan were found anywhere in the methodology, discussion, or references sections of the paper. Detailed Analysis: The ERCT 'P' criterion requires explicit evidence that the full study protocol, including hypotheses, methods and planned analyses, was publicly registered before data collection began. The paper contains no mention of any pre-registration platform or registration date; the only related procedural detail is that "Each student signed an informed written consent agreement," which pertains to ethics consent, not protocol pre-registration. An internet search for a pre-registration record under the authors' names (Coates, Gorham, Nicholas) or the study's protocol (Calini Preliminary Language Test Phonics, Italian high school phonics RCT) on common registries and OSF found no matching record. Because no evidence of pre-registration is present in the paper or found via internet search, this criterion is not met.