Differential effects of instruction on the development of second language comprehensibility, word stress, rhythm, and intonation: The case of inexperienced Japanese EFL learners

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Two pre-existing ("intact") university classes were assigned to conditions with no described randomisation procedure, so this is a quasi-experiment, not a class-level RCT.
    • "20 Japanese first-year university students from two intact classes participated as EFL learners"
    • Relevant Quotes: 1) "the current article reports on a quasi-experimental study, which investigated whether and to what degree a single-semester, suprasegmental-based instructional treatment could affect the comprehensibility, word stress, rhythm, and intonation development of inexperienced Japanese students of English as a foreign language (EFL)." (p. 590) 2) "In total, one experienced Japanese speaker participated as an instructor, 20 Japanese first-year university students from two intact classes participated as EFL learners, and four native-speaking English teachers participated as experienced raters." (p. 594) 3) "The two classes were assigned to serve as the experimental group (n = 10) and the control group (n = 10), respectively." (p. 594) Detailed Analysis: The authors explicitly label their own design a "quasi-experimental study" (p. 590), not a randomised controlled trial. The two intervention conditions correspond to two pre-existing, intact university classes that were "assigned" to the experimental and control conditions; no quote anywhere in the Method section describes a random-assignment mechanism (e.g. coin flip, random number generator, or lottery) at either the class, student, or school level. "Intact classes" is standard terminology for pre-formed groups that were not created by randomisation. Because there is no documented randomisation procedure at all, the exception for one-to-one tutoring does not rescue the criterion (this is classroom-based group instruction, not personal tutoring), and the stronger school-level criterion (S) is also not met (see below), so it cannot flow up to satisfy C either. Criterion C is not met because the study is explicitly a quasi-experiment using intact, non-randomised classes.
  • E

    Exam-based Assessment

    • Outcomes were measured via researcher-elicited read-aloud speech samples rated with a custom comprehensibility slider and acoustic/phonetic coding, not a standardised exam.
    • "Speech data for this study were elicited from a paragraph reading task, in which students read two different presentation introductions"
    • Relevant Quotes: 1) "Speech data for this study were elicited from a paragraph reading task, in which students read two different presentation introductions: a trained text (Text A) and an untrained text (Text B)." (p. 597) 2) "All speech samples were played in a randomized order using computer-based software developed in previous research (Saito, Trofimovich, & Isaacs, 2015). Upon listening to each speech sample only once, the raters used a free-moving slider to assess comprehensibility." (p. 598) 3) "The effect of form-focused instruction on learners' suprasegmental development was analysed via the acoustic measures used in Isaacs and Trofimovich (2012). A linguistically trained coder analysed word stress and rhythm via auditory impressions, and intonation by listening and visuals (spectrograms) via Praat (Boersma & Weenink, 2012)." (pp. 598-599) Detailed Analysis: Criterion E requires a widely recognised, standardised exam-based assessment rather than an instrument built specifically for the study. Here, outcomes were derived from bespoke read-aloud passages (Texts A and B, written by the researchers) rated by four hand-picked raters using a purpose-built slider-based rating tool, plus phonetic coding of stress/rhythm/intonation errors by a trained coder using Praat. None of these instruments is a standardised exam (e.g. a national or state exam); they are custom research measures designed by the same research group specifically to capture suprasegmental and comprehensibility gains from this intervention. Criterion E is not met because assessment relied entirely on researcher-designed speech-elicitation and rating instruments rather than a standardised exam.
  • T

    Term Duration

    • The interval from the start of the instructional treatment (around Week 5) to the post-test (Week 12) was only about seven weeks, well short of a full academic term.
    • "For the first 30 minutes of Weeks 5, 6, 7, 8, and 9, the instructor started with form-focused instruction on word stress and intonation in turn"
    • Relevant Quotes: 1) "From Week 5 onwards, 10 students in the experimental group received approximately 30 minutes of form-focused instruction on the full/weak vowel distinction (word stress, rhythm) and tonal melody (intonation)." (pp. 594-595) 2) "Ten students in the experimental group received a total of three hours of instruction on suprasegmentals (six times) over six weeks. For the first 30 minutes of Weeks 5, 6, 7, 8, and 9, the instructor started with form-focused instruction on word stress and intonation in turn ... Rhythm ... was introduced in Week 11." (p. 596) 3) "All of the participating students took the post-test in Week 12." (p. 595) 4) "Over the 14 weeks of the semester, the students watched model presentations on the attached DVD..." (p. 597) Detailed Analysis: The suprasegmental instruction ran from Week 5 through Week 11 (six weeks, three hours total), and the post-test (Week 12) occurred immediately after the instructional treatment ended, roughly seven weeks after the treatment began. Although the overall course spans a 14-week semester, the interval specifically between the start of the intervention and the outcome measurement is approximately seven weeks (about 1.5-1.75 months), far short of the required one full academic term (roughly 3-4 months). There is no later follow-up measurement reported that would extend this interval. Criterion T is not met because outcomes were measured only about seven weeks after the intervention began, short of a full academic term.
  • D

    Documented Control Group

    • The control group's composition, baseline proficiency, and the activities they received in place of the intervention are described in detail.
    • "The 10 students in the control group received meaning-oriented lessons that were comparable in terms of duration, but lacked any focus on English suprasegmentals."
    • Relevant Quotes: 1) "The two classes were assigned to serve as the experimental group (n = 10) and the control group (n = 10), respectively." (p. 594) 2) "Comparatively, the 10 participants in the control group received meaning-oriented lessons on English presentation skills, in which they carried out similar activities on presentation structure and effective visuals with no particular focus on suprasegmentals." (p. 597) 3) "The 10 students in the control group received meaning-oriented lessons that were comparable in terms of duration, but lacked any focus on English suprasegmentals." (p. 595) 4) "The participants' scores ranged from 420 to 435, with no significant group differences according to an independent sample t-test (p > .05)." (p. 595) 5) "The two groups were also comparable in their pronunciation performance at the outset of the project. A set of independent sample t-tests found no significant group differences in any contexts at a p < .05 level." (p. 596) Detailed Analysis: The paper provides detailed documentation of the control group: its size (n = 10), the specific activities its members received (meaning-oriented presentation-skills lessons lacking suprasegmental focus), the duration of those activities relative to the experimental group, and baseline comparability with the experimental group on both proficiency (TOEIC scores) and pre-test pronunciation performance, confirmed via statistical tests. This level of detail satisfies the documentation requirement, independent of the randomisation and standardisation issues that affect other criteria. Criterion D is met because the control group's makeup, baseline characteristics, and treatment are clearly documented and shown to be comparable to the experimental group at baseline.

• Level 2 Criteria

  • S

    School-level RCT

    • Only two intact classes at a single university were involved, with no school-level (or any) randomisation described.
    • "20 Japanese first-year university students from two intact classes participated as EFL learners"
    • Relevant Quotes: 1) "20 Japanese first-year university students from two intact classes participated as EFL learners" (p. 594) 2) "The two classes were assigned to serve as the experimental group (n = 10) and the control group (n = 10), respectively." (p. 594) Detailed Analysis: Criterion S requires randomisation among whole schools (or equivalent institutional units), not classes or students. Here, the study involves a single institution, and the unit compared is two intact classes within that one institution, with no randomisation procedure described at any level. There is no evidence of multiple schools being involved or randomised. Criterion S is not met because the study used two intact classes at a single institution rather than randomised whole schools.
  • I

    Independent Conduct

    • The intervention was designed and delivered by one of the authors, who also served as the sole instructor for both groups.
    • "Both classes were taught by the first author, a Japanese EFL teacher with near-native English proficiency and post-graduate education in applied linguistics and L2 pronunciation teaching."
    • Relevant Quotes: 1) "Both classes were taught by the first author, a Japanese EFL teacher with near-native English proficiency and post-graduate education in applied linguistics and L2 pronunciation teaching." (p. 595) 2) "At the beginning of each lesson, a set of rules about the target suprasegmentals were briefly introduced. The students first carefully listened to and repeated the instructor's model pronunciation..." (p. 597) 3) "We are grateful to the journal editor, Frank Boers, and Language Teaching Research reviewers for providing constructive comments on earlier versions of this article. We gratefully acknowledge Jeff Broderick who helped data analyses..." (p. 605, Acknowledgements) Detailed Analysis: Criterion I requires that the evaluation be conducted independently of the people who designed and delivered the intervention. In this study, the first author both designed the form-focused instructional treatment and personally taught both the experimental and control classes. The acknowledgements mention only assistance with data analysis (Jeff Broderick) and editorial feedback, not an independent evaluation team responsible for implementation, data collection, or interpretation. There is no statement of third-party oversight or an external evaluator blind to condition. Criterion I is not met because the same author who designed the intervention also personally delivered it to both groups and there is no independent evaluation team.
  • Y

    Year Duration

    • Because criterion T (Term Duration) is not met, criterion Y is automatically not met per the ERCT dependency rule.
    • "All of the participating students took the post-test in Week 12."
    • Relevant Quotes: 1) "All of the participating students took the post-test in Week 12." (p. 595) 2) "Over the 14 weeks of the semester, the students watched model presentations..." (p. 597) Detailed Analysis: The entire study, from Week 1 (project explanation) through the Week 12 post-test, spans well under a full academic year and even under the 14-week single semester referenced in the paper; there is no follow-up at a later point in the academic year. Per the ERCT standard's dependency rule, since the weaker Term Duration (T) criterion is already not met (the intervention-to-measurement interval was only about seven weeks), the stronger Year Duration (Y) criterion cannot be met either. Criterion Y is not met because criterion T is not met and the observed tracking period (about 11 weeks total) is far short of a full academic year.
  • B

    Balanced Control Group

    • The control group received meaning-oriented lessons explicitly matched in duration to the experimental group's lessons, isolating the suprasegmental-instruction content as the treatment variable.
    • "The 10 students in the control group received meaning-oriented lessons that were comparable in terms of duration, but lacked any focus on English suprasegmentals."
    • Relevant Quotes: 1) "From Week 5 onwards, 10 students in the experimental group received approximately 30 minutes of form-focused instruction ... Following this, they engaged in meaning-oriented lessons aimed at improving their presentation skills." (pp. 594-595) 2) "The instruction was provided over six weeks and totaled three hours. The 10 students in the control group received meaning-oriented lessons that were comparable in terms of duration, but lacked any focus on English suprasegmentals." (p. 595) 3) "Comparatively, the 10 participants in the control group received meaning-oriented lessons on English presentation skills, in which they carried out similar activities on presentation structure and effective visuals with no particular focus on suprasegmentals." (p. 597) Detailed Analysis: Applying the criterion B decision procedure: the first check is whether the intervention adds any extra time or budget relative to the control group. Here it does not - the paper explicitly states that the total instructional duration was matched between groups ("comparable in terms of duration"), and the control group engaged in an active, comparable set of presentation-skills activities during that time rather than receiving nothing. The additional resource in the experimental group is therefore not extra time or budget but a difference in instructional focus/ content within equal time, with the control group serving as an active comparison condition that controls for mere exposure and test-retest effects (part of the authors' stated rationale for including the control group). Since no extra time or budget is present at all (the first branch of the criterion B decision tree), the criterion is met without needing to invoke the "integral treatment variable" exception. Criterion B is met because the paper explicitly documents that the control group's lessons were of comparable duration to the experimental group's, with both groups actively engaged for equivalent time, so no extra resource imbalance exists to be assessed.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this specific study is mentioned in the paper, and an internet search found no subsequent replication by other research teams.
    • Relevant Quotes: 1) "Motivated by a growing number of studies on L2 suprasegmental instruction (typically involving intermediate-to-advanced ESL learners) (e.g. Derwing et al., 1998, 2014), the current study examined the effectiveness of form-focused instruction on developing the word stress, rhythm, and intonation abilities of inexperienced Japanese EFL learners." (p. 604) 2) "more studies with a larger sample size and longitudinal design are necessary to generalize the findings of EFL suprasegmental development to a larger context." (p. 604) Detailed Analysis: The paper situates itself as testing the generalisability of prior suprasegmental-instruction research (e.g. Derwing et al.) to a new population (inexperienced Japanese EFL learners), but this is a different study population and design, not a replication of the present study itself. The authors explicitly call for future studies with larger samples to further test generalisability, indicating that, as of this paper, no independent replication of this specific study (same design, same population) had yet been conducted. No quotes describing a subsequent independent replication were found within the paper's own text. An internet search (web search for citing/related work on this specific 2017 Saito & Saito study and its design) did not surface any independent replication of this particular quasi-experimental design (two intact Japanese university EFL classes, same suprasegmental treatment, same outcome measures) conducted by a different research team in a different context. Later related work by the same and other authors on L2 suprasegmental/pronunciation instruction was found, but none of it constitutes a reproduction of this specific study's design and sample. Criterion R is not met because no independent replication of this specific study is reported in the paper or evident from an internet search.
  • A

    All-subject Exams

    • Because criterion E (Exam-based Assessment) is not met, and only suprasegmental/comprehensibility pronunciation outcomes (not other core subjects) were assessed, criterion A is not met.
    • "The effect of form-focused instruction on learners' suprasegmental development was analysed via the acoustic measures used in Isaacs and Trofimovich (2012)."
    • Relevant Quotes: 1) "The current study examined in depth the effects of suprasegmental-based instruction on the global (comprehensibility) and suprasegmental (word stress, rhythm, and intonation) development of Japanese learners of English as a foreign language (EFL)." (p. 589) 2) "The effect of form-focused instruction on learners' suprasegmental development was analysed via the acoustic measures used in Isaacs and Trofimovich (2012)." (pp. 598-599) Detailed Analysis: Per the ERCT standard, criterion A first requires that criterion E be met (only standardised exam-based assessments count), which is not the case here. In addition, the study exclusively measured pronunciation outcomes (comprehensibility, word stress, rhythm, intonation) within an EFL presentation-skills course; no other core subjects (e.g. mathematics, science, general language arts) were assessed, and no exception rationale for a specialised vocational/upper-secondary focus is offered (this is a general EFL pronunciation study, not such a specialised case). Criterion A is not met both because criterion E is not met and because only pronunciation outcomes in a single course were assessed, with no all-subject coverage.
  • G

    Graduation Tracking

    • Because criterion Y (Year Duration) is not met, criterion G is automatically not met, and no follow-up tracking toward graduation is reported or found via internet search.
    • "All of the participating students took the post-test in Week 12."
    • Relevant Quotes: 1) "All of the participating students took the post-test in Week 12." (p. 595) 2) "Future research should overcome this issue by including extemporaneous speech samples which are representative of natural speech (Lee et al., 2015)." (p. 604, Limitations) Detailed Analysis: The study's data collection ends with the Week 12 post-test; there is no mention anywhere in the paper of any further follow-up testing, let alone tracking students until graduation. The Limitations section discusses several methodological caveats (task type, rater selection, sample size) but does not mention any planned or completed long-term follow-up. Per the ERCT dependency rule, since Y (Year Duration) is not met, G cannot be met either. An internet search for later publications by Yukie Saito or Kazuya Saito that might track this same cohort of 20 Japanese university students toward graduation did not find any such follow-up study; the authors' subsequent publications address different samples, designs, or theoretical questions rather than long-term tracking of this cohort. Criterion G is not met because tracking ended at the Week 12 post-test with no graduation follow-up, the prerequisite Y criterion is also not met, and no follow-up publication tracking this cohort was found.
  • P

    Pre-Registered

    • No statement of pre-registration of the study protocol appears anywhere in the paper, and an internet search found no registry entry for this study.
    • Relevant Quotes: 1) "In Week 1, the EFL students were given an explanation of the study's purpose, signed a consent form and filled in a background survey containing their bio-data and English learning experience." (p. 594) 2) No mention of any trial registry, protocol pre-registration platform, or registration ID appears anywhere in the Method, Results, Discussion, or Acknowledgements/Funding sections of the paper. Detailed Analysis: The Method section describes obtaining informed consent from participants but contains no reference to a pre-registered study protocol, hypotheses, or analysis plan filed with a registry (e.g. OSF, AsPredicted, or a trials registry) prior to data collection. The Funding statement ("This research received no specific grant from any funding agency...") and Acknowledgements likewise make no mention of pre-registration. An internet search for a pre-registration record of this study (e.g. on OSF, AsPredicted, or a clinical/education trials registry) associated with these authors and this 2017 Language Teaching Research paper did not return any matching registration. Criterion P is not met because there is no evidence in the paper, or from an internet search, of a pre-registered protocol prior to data collection.