The Effectiveness of Content-Based Instruction in English Language Teaching

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • The paper's own methodology section explicitly describes a "non-randomized" design, with the experimental group formed by ranking students on achievement scores rather than by random assignment, so no valid class-level (or stronger) randomisation is documented.
    • "In the present study, the non-randomized control group pretest, post-test design was adopted." (p. 45)
    • Relevant Quotes: 1) "In the present study, the non-randomized control group pretest, post-test design was adopted." (p. 45) 2) "The subjects of the two groups were selected and the application of randomness led to the classification of the control and experimental groups." (p. 45) 3) "The experimental group was formed on the basis of the academic achievement scores of the students. Thirty learners were grouped into five teams with six members in each team based on the scores of the first term examination of the English language. The first five highest scorers formed the first members of the five teams and the remaining scorers were distributed to each team as per the procedure of distribution." (p. 45) 4) "The control group consisted of 30 learners studying in the same class of the same college." (p. 45) Detailed Analysis: The abstract claims "A randomized pretest, post-test, control group and experimental group design were employed," but this directly contradicts the Methodology section, which explicitly states the design was "non-randomized." The description of how the experimental group was formed (ranking students by achievement score and distributing them across five teams to balance ability) is a deliberate, stratified assignment procedure, not random allocation. The control group was simply "the same class of the same college," with no described procedure for randomly allocating students between control and experimental conditions. No quote anywhere in the paper describes randomisation at the class, school, or even student level being genuinely carried out; the concrete account given is of a non-randomized, quasi- experimental design using pre-formed, ability-matched groups. The intervention is not a personal tutoring program, so the student-level exception does not apply either. Because the study explicitly self-describes as "non-randomized" and no valid random-assignment procedure at any level is documented, this raises serious doubt about whether the paper qualifies as a genuine RCT at all, and criterion C is not met.
  • E

    Exam-based Assessment

    • The pre- and post-tests were "self-made achievement tests" constructed by the researcher for this specific course content, not a recognised standardised exam.
    • "The investigator's self-made achievement tests were used for the pre-tests and post-tests of both the groups."
    • Relevant Quotes: 1) "The investigator's self-made achievement tests were used for the pre-tests and post-tests of both the groups." (p. 46) 2) "The same question paper was used for both the groups to evaluate pupils' skills in Optional I English covering selected topics of the content." (p. 46) 3) "In order to increase the reliability and validity of the post-test performance and to eliminate the testing effect of the pretest, two other achievement tests were constructed. These tests were a slight modification of the Pretest." (p. 46) 4) "The content of both the tests were validated by a team of English language specialists." (p. 46) Detailed Analysis: The outcome measures were tests custom-built by the researcher herself, covering specific syllabus units (5 and 6) of the "Optional I English" course. Although the tests were reviewed by a panel of English language specialists for content validity, and test-retest reliability coefficients are reported, they are not standardised, widely recognised examinations (e.g. a national or board exam); they are researcher- constructed instruments closely aligned to the exact content taught in the intervention. This is precisely the scenario the ERCT standard flags as a bias risk, since a custom test tailored to what was taught may inflate the apparent effectiveness of the intervention. Final sentence: Criterion E is not met because the assessments used were self-made, non-standardised achievement tests rather than a recognised standardised exam.
  • T

    Term Duration

    • No calendar dates or explicit interval are given for the intervention or the post-tests; the only concrete time interval mentioned in the paper (two weeks) refers to reliability piloting, not the actual study.
    • "Test-retest method was used to check the reliability. First, a test was administered to them and it was repeated on the same group after two weeks."
    • Relevant Quotes: 1) "Contents of units 5 and 6 in Optional I English were selected for the administration of the pretest." (p. 46) 2) "Other important contents of unit 5 and 6 were selected for administering the other two tests." (p. 46) 3) "A pilot group of 30 students was randomly selected from the population of the study who were excluded from the sample. Test-retest method was used to check the reliability. First, a test was administered to them and it was repeated on the same group after two weeks." (p. 46) Detailed Analysis: The paper never states the calendar start date of the CBI intervention, nor the dates on which Post-test I or Post-test II were administered, so the interval from intervention start to outcome measurement cannot be verified from the text. The only explicit "two week" interval mentioned anywhere in the paper concerns the test-retest reliability piloting of the instrument with an excluded pilot group, not the timing of the main study. Given that the intervention covers only two specific curriculum units within a single course, and there is no documented evidence of a term-long tracking window, the paper provides no basis to conclude outcomes were measured at least one academic term after the intervention began. Final sentence: Criterion T is not met because the paper does not document an interval of at least one academic term between intervention start and outcome measurement, and available evidence suggests a much shorter, single-unit intervention.
  • D

    Documented Control Group

    • The control group's size, institutional context, instructional condition, and baseline achievement scores (shown statistically equivalent to the experimental group) are documented.
    • "The control group consisted of 30 learners studying in the same class of the same college. The group was exposed to the traditional method instruction and no novel treatment was given to this group."
    • Relevant Quotes: 1) "The control group consisted of 30 learners studying in the same class of the same college. The group was exposed to the traditional method instruction and no novel treatment was given to this group." (p. 45) 2) "Both the groups did not differ in their pre-test mean achievement scores as testified by the t-value of 0.19 which is not significant at the 0.05 level." (p. 47) 3) Table 1: "Pretest - Comparison between Control and Experimental Groups / Control 30 56.4 8.5 0.19 / Experimental 30 56.8 7.2" (p. 47) Detailed Analysis: The paper documents the control group's size (30 learners), its institutional context (drawn from the same B.Ed. class/college population as the experimental group), the instructional condition it received ("traditional method instruction," with "no novel treatment"), and its baseline achievement performance via Table 1, which shows near-identical pretest means and a statistically non-significant t-value between the control and experimental groups. This gives a reasonably clear, quoted picture of who the control group was, how comparable it was to the experimental group at baseline, and what treatment (business-as-usual instruction) it received. Final sentence: Criterion D is met because the paper documents the control group's size, baseline comparability, and the ordinary instruction it received.

• Level 2 Criteria

  • S

    School-level RCT

    • Both groups were drawn from a single college with non-random, achievement-based group formation, not randomisation across multiple schools or institutions.
    • "The two groups were selected from the College of Education in the Salem District."
    • Relevant Quotes: 1) "The two groups were selected from the College of Education in the Salem District." (p. 45) 2) "The control group consisted of 30 learners studying in the same class of the same college." (p. 45) 3) "In the present study, the non-randomized control group pretest, post-test design was adopted." (p. 45) Detailed Analysis: Both groups came from a single institution (one College of Education), and allocation between conditions was explicitly non-random and based on achievement scores, not conducted at the level of whole schools/colleges. There is no evidence in the paper of multiple schools or colleges being randomly assigned to intervention versus control conditions. Final sentence: Criterion S is not met, since this is a single-institution, non-randomized study rather than a school-level RCT.
  • I

    Independent Conduct

    • The single-author researcher designed, delivered, and evaluated her own intervention, with no independent third party conducting the trial.
    • "The investigator's self-made achievement tests were used for the pre-tests and post-tests of both the groups."
    • Relevant Quotes: 1) "The researcher has identified the need for a professional development activity to support the student teachers' understanding... She conducted research in the strategy of teaching learning process with the goal of having the desired impact on teacher understanding." (p. 37) 2) "The investigator's self-made achievement tests were used for the pre-tests and post-tests of both the groups." (p. 46) 3) "The content of both the tests were validated by a team of English language specialists... The remarks and suggestions of the team were taken into consideration and the researcher made the necessary modification before its application." (p. 46) Detailed Analysis: The paper is authored by a single researcher (Dr. Evangelin Arulselvi), who designed the CBI intervention, constructed the assessment instruments, administered the study, and analysed the results herself. An external panel of language specialists reviewed the tests for content validity, but this is a limited instrument-validation step, not independent conduct of the trial's implementation, data collection, or analysis. No third-party evaluator or independent research team is mentioned as overseeing or running the study. Final sentence: Criterion I is not met because the same researcher who designed the intervention also delivered it and analysed the results, without independent oversight.
  • Y

    Year Duration

    • Criterion T is not met and no evidence of year-long (or even multi-month) tracking is provided, so Y is not met.
    • Detailed Analysis: As established under criterion T, the paper does not document the calendar timing of the intervention or outcome measurement, and the only concrete duration mentioned (two weeks) pertains to reliability piloting, not the main study. There is no quote anywhere in the paper describing tracking of students over 75% of an academic year, or indeed over any extended period at all. Per the ERCT rule that criterion Y cannot be met if criterion T is not met, and given the complete absence of any indication of a year-long follow-up window, criterion Y is not met. Final sentence: Criterion Y is not met because criterion T is not met and no evidence of a year-length tracking period exists in the paper.
  • B

    Balanced Control Group

    • The CBI method was implemented as a substitute teaching technique with no additional class time, materials, or budget relative to the control group's regular instruction.
    • "The result indicated that CBI method can be used successfully for students of diverse abilities and it can be easily used as a modification to instruction with no extra time or effort required of the teacher."
    • Relevant Quotes: 1) "The result indicated that CBI method can be used successfully for students of diverse abilities and it can be easily used as a modification to instruction with no extra time or effort required of the teacher." (p. 50) 2) "One lesson plan using Content-based instruction method has built-in peer tutoring and support within the heterogeneous class groupings, which eliminates the requirement for several different plans to meet the needs of all students." (p. 50) 3) "The control group consisted of 30 learners studying in the same class of the same college. The group was exposed to the traditional method instruction and no novel treatment was given to this group." (p. 45) 4) "The time allotted for answering was one hour." (applies to testing of both groups, p. 46) Detailed Analysis: The CBI intervention is a change in teaching technique applied within the existing instructional time for the same course content (Optional I English, units 5 and 6). The authors themselves state it "can be easily used as a modification to instruction with no extra time or effort required of the teacher," indicating no additional class time, materials, or budget was given to the experimental group relative to the control group's normal instruction. Both groups underwent identical testing conditions (same time allotted). No extra resources of any kind are described for the experimental condition, so there is nothing that needs to be balanced. Applying the ERCT decision tree: since the intervention does not add extra time or budget to the experimental group, criterion B is trivially met. Final sentence: Criterion B is met because the study involved no additional time, materials, or budget for the intervention group beyond ordinary instruction.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this specific study by another research team was found in the paper or in a dedicated internet search of citing literature.
    • Relevant Quotes: 1) "Yoon (Christina) Heo (2005) conducted research on Content-based instruction (CBI)... Her research paper has reviewed general information about the features of CBI, including its theoretical foundations and models." (p. 39) 2) "Silvia Pessoa and Heather Hendry (2007) conducted research on testing the effects of CBI in the foreign language classrooms." (p. 40) Detailed Analysis: This is a small, single-site, single-author study. The literature review cites several earlier and contemporaneous CBI studies by other authors (Heo, 2005; Pessoa & Hendry, 2007; Warrington, 2008; Iosif, 2011; Omoto, 2013; Addison & Walker, 2000), but these are general prior studies on CBI in different contexts and designs, not independent replications of this specific paper's design, population, and results (most of them predate this paper). No quote in the paper, nor any external evidence apparent from the text, indicates that this specific study (comparing CBI versus conventional instruction for Optional I English among B.Ed. students in Salem District) has been independently replicated by a different research team. An internet search (ERIC, Google Scholar-style web search, and general web search) for citations of "The Effectiveness of Content-Based Instruction in English Language Teaching" (Arulselvi, 2016, Excellence in Education Journal) found the original ERIC record (EJ1210158) and numerous unrelated CBI studies from other authors and contexts, but no independent replication study reproducing this specific paper's design, population, and results by a different research team was located. Final sentence: Criterion R is not met because no independent replication of this specific study was identified in the paper or via internet search.
  • A

    All-subject Exams

    • Only one subject (English) was assessed using non-standardised, researcher-made tests, and since criterion E is not met, criterion A is automatically not met.
    • "The same question paper was used for both the groups to evaluate pupils' skills in Optional I English covering selected topics of the content."
    • Relevant Quotes: 1) "The same question paper was used for both the groups to evaluate pupils' skills in Optional I English covering selected topics of the content." (p. 46) 2) "The investigator's self-made achievement tests were used for the pre-tests and post-tests of both the groups." (p. 46) Detailed Analysis: Only English-language achievement (specifically, content from units 5 and 6 of Optional I English) was assessed; no other core subjects were measured. Per the ERCT specification, criterion E (standardised exam-based assessment) is a prerequisite for criterion A, and since E is not met here (the tests were self-made, not standardised), criterion A cannot be met either, independent of subject coverage. Final sentence: Criterion A is not met because only one subject was assessed with non-standardised tests, and criterion E is not met.
  • G

    Graduation Tracking

    • No follow-up beyond the immediate post-tests is reported, no follow-up publication tracking this cohort was located, and since criterion Y is not met, criterion G is automatically not met.
    • Detailed Analysis: The study concludes with the administration of Post-test II, immediately followed by discussion of results; there is no mention anywhere in the paper of any further follow-up, and certainly no tracking of students through to graduation from the B.Ed. program or any other educational stage. An internet search for subsequent papers by Evangelin Arulselvi tracking this same cohort of College of Education students in Salem District through to graduation did not locate any such follow-up publication; her other identifiable publications concern different teaching methods (e.g. task-based language teaching, the Silent Way) rather than a longitudinal follow-up of this 2016 CBI cohort. Additionally, per the ERCT rule that criterion G cannot be met if criterion Y is not met, and since Y is not met here, criterion G is automatically not met. Final sentence: Criterion G is not met because no follow-up beyond the immediate post-tests is described in the paper or found via internet search, and criterion Y is not met.
  • P

    Pre-Registered

    • No pre-registration statement, registry link, or registration date is provided anywhere in the paper.
    • Detailed Analysis: There is no mention anywhere in the paper of the study protocol being registered on any public trial registry (e.g. ClinicalTrials.gov, ISRCTN, AEA RCT Registry, or similar) prior to data collection, nor any registration ID or date. No quote referencing pre-registration exists in the text, so no external registry search was needed per the verification instructions (only papers that mention pre-registration require registry lookup). Final sentence: Criterion P is not met because no evidence of pre-registration is present in the paper.