Research on the Influence of CBI Theme-Based Instructional Model on Metallurgy Major English Classroom

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation was carried out at the individual student level (82 students randomly split into two groups) rather than by randomising entire pre-existing classes, and the paper is not a one-to-one tutoring intervention that would qualify for the exception.
    • "这些学生被随机分为两组：实验组(41 人，标记为实验班 A)和对照组 (41 人，标记为控制班 B)。" (p. 827)
    • Relevant Quotes: 1) "本研究以江西理工大学 2022 级的 82 名冶金专业学生为研究对象。 这些学生被随机分为两组：实验组(41 人，标记为实验班 A)和对照组 (41 人，标记为控制班 B)。" (p. 827) [Translation: This study takes 82 metallurgy-major students of the 2022 cohort at Jiangxi University of Science and Technology as subjects. These students were randomly divided into two groups: an experimental group (41, labelled experimental class A) and a control group (41, labelled control class B).] 2) "The study adopts an experimental design, dividing participants into experimental and control groups, where the experimental group receives instruction based on the CBI theme-based model." (Abstract, p. 826) Detailed Analysis: Criterion C requires that randomisation be performed at the class level or higher, i.e., that entire intact classes (or schools) be randomly assigned to conditions. In this study the unit of randomisation is the individual student: 82 students were randomly divided into two groups which were then labelled class A and class B. No pre-existing classes were randomised; with only one class per condition there is no class-level random assignment at all. The paper also gives no description of the randomisation mechanism (no mention of how the random allocation was generated or concealed). The intervention is whole-class CBI instruction, not personal tutoring or one-to-one teaching, so the tutoring exception does not apply. Although delivering the two conditions in separate classrooms reduces within-classroom contamination, the paper never states that intact classes were the randomised unit, and the design cannot separate teacher/class effects from the treatment. Criterion C is not met because randomisation was performed at the individual student level with no class-level random assignment and no applicable exception.
  • E

    Exam-based Assessment

    • Outcomes were measured with the national College English Test Band 4 (CET-4), a widely recognised standardised exam, with Gaokao English scores used as the standardised baseline.
    • "前测采用学生入学时的高考英语成绩作为基准数据(见表 1)，后测则 使用实验结束后的大学英语四级考试成绩。" (p. 828)
    • Relevant Quotes: 1) "为了评估实验效果，本研究进行了前测和后测。前测采用学生入学 时的高考英语成绩作为基准数据(见表 1)，后测则使用实验结束后的 大学英语四级考试成绩。所有数据均通过 SPSS22.0 软件进行分析处理。" (p. 828) [Translation: To evaluate the experimental effect, this study conducted a pre-test and a post-test. The pre-test used the students' college entrance examination (Gaokao) English scores at enrolment as baseline data (see Table 1); the post-test used the College English Test Band 4 (CET-4) scores after the experiment ended. All data were analysed with SPSS 22.0.] 2) "笔试成绩总分 控制组 41 412.73 48.513 −2.841 80 0.006 / 实验组 41 447.22 60.727" (Table 2, p. 828) [CET-4 total written score, listening, reading and writing sub-scores compared between groups.] Detailed Analysis: Criterion E requires that outcomes be measured with a standard, widely recognised standardised exam rather than a researcher-made test. The primary quantitative outcome here is the score on the College English Test Band 4 (CET-4, 大学英语四级考试), which is a national standardised English examination administered across Chinese universities, with standard sub-scores (listening, reading, writing) analysed in Table 2. The baseline measure is the national college entrance examination (Gaokao) English score, also a national standardised exam. Neither instrument was created for this study. The accompanying attitude questionnaire is a custom instrument, but it is supplementary; the achievement outcome is the standardised CET-4. Criterion E is met because the study's academic outcome was measured with the national standardised CET-4 examination.
  • T

    Term Duration

    • The paper never states the intervention start date, its length or the interval to the CET-4 measurement, and the authors themselves describe the experimental period as relatively insufficient, so a term-long interval cannot be verified.
    • "未发现显著差异究其原因在于教学方法直接影响写作技能加之实验周期 相对不足。" (p. 829)
    • Relevant Quotes: 1) "前测采用学生入学时的高考英语成绩作为基准数据(见表 1)，后测 则使用实验结束后的大学英语四级考试成绩。" (p. 828) [Pre-test: Gaokao English scores at enrolment; post-test: CET-4 scores after the experiment ended.] 2) "至于作文，未发现显著差异究其原因在于教学方法直接影响写作技 能加之实验周期相对不足。" (p. 829) [Translation: As for writing, no significant difference was found; the reason lies in the teaching method directly affecting writing skills, plus the experimental period being relatively insufficient.] 3) "听力技能的提高需要长时间打磨训练，短时间内无法立竿见影。" (p. 829) [Translation: Improvement in listening skills requires long training over time and cannot show immediate results in a short period.] Detailed Analysis: Criterion T requires that outcomes be measured at least one full academic term (about 3-4 months) after the intervention begins, with quotable start and measurement dates. This paper provides no explicit intervention start date, no end date, and no statement of how long the CBI instruction lasted. The only timing information is that the pre-test was the Gaokao score at enrolment (2022 cohort) and the post-test was the CET-4 taken after the experiment ended; the actual instructional window is never dated. While the course may plausibly have run for a semester, the standard requires the interval to be clearly documented, and the authors themselves twice characterise the experiment as short ("实验周期相对不足", "短时间内无法立竿见影"), suggesting the period was limited. With no documented interval of at least one term from intervention start to outcome measurement, the criterion cannot be verified as satisfied. Criterion T is not met because the intervention timeline is not documented and the authors describe the experimental period as relatively insufficient.
  • D

    Documented Control Group

    • The control group's size (41 students), cohort, baseline Gaokao score distribution and mean, and its business-as-usual traditional instruction are all clearly documented.
    • "对照组则采用传统教学法，按照常规方式进行教学。" (p. 827)
    • Relevant Quotes: 1) "这些学生被随机分为两组：实验组(41 人，标记为实验班 A)和对照 组(41 人，标记为控制班 B)。在实验过程中，实验组接受内容依托式 主题教学法(CBI)的指导，而对照组则接受传统教学法。" (p. 827) [The 82 students were randomly divided into an experimental group (41) and a control group (41); the control group received traditional teaching methods.] 2) "对照组则采用传统教学法，按照常规方式进行教学。" (p. 827) [Translation: The control group used traditional teaching methods, taught in the conventional way.] 3) "Table 1. Analysis of college entrance examination scores of experimental group and control group... 90 分以下 3 3 / 90~110 分 13 10 / 110~130 分 19 23 / 130 分以上 7 6 / 平均分 462.81 463.82" (p. 828) 4) "笔试成绩总分 控制组 41 412.73 48.513" (Table 2, p. 828) Detailed Analysis: Criterion D requires clear documentation of the control group's composition, baseline performance and the treatment it received. The paper specifies the control group's size (41 students), its population (2022-cohort metallurgy majors at Jiangxi University of Science and Technology), its baseline academic performance (Table 1 gives the distribution of Gaokao English scores in bands and the group mean of 463.82, comparable to the experimental group's 462.81), and states that it received conventional traditional instruction with no special treatment. Table 2 further reports the control group's outcome means and standard deviations for each CET-4 component. Demographic detail beyond cohort and major is thin, but the essentials required by the criterion - who the control group is, baseline scores, group size, and what instruction they received - are documented. Criterion D is met because the control group's size, baseline performance and business-as-usual condition are clearly documented.

• Level 2 Criteria

  • S

    School-level RCT

    • The study randomised individual students within a single university programme; no schools or institutions were randomised.
    • "本研究以江西理工大学 2022 级的 82 名冶金专业学生为研究对象。 这些学生被随机分为两组" (p. 827)
    • Relevant Quotes: 1) "本研究以江西理工大学 2022 级的 82 名冶金专业学生为研究对象。 这些学生被随机分为两组：实验组(41 人，标记为实验班 A)和对照组 (41 人，标记为控制班 B)。" (p. 827) [82 metallurgy-major students at one university were randomly divided into two groups.] Detailed Analysis: Criterion S requires randomisation among schools or institution-level units implementing the intervention. This study took place entirely within a single institution (Jiangxi University of Science and Technology) and randomised individual students of one major into two groups. No multiple schools, campuses or sites were involved, and nothing was randomised at any institutional level. Criterion S is not met because randomisation occurred at the student level within a single university, not among schools.
  • I

    Independent Conduct

    • The authors, English instructors at the university, designed the CBI intervention and conducted the teaching, testing and analysis themselves with no independent evaluator mentioned.
    • "为了深入了解学生对英语学习的态度... 笔者基于 Gardner (1985)的 态度动机测试理论，设计了包含 14 个封闭式问题和 1 个开放式问题的 调查问卷。" (p. 829)
    • Relevant Quotes: 1) "骆 迎，邹晓萍* 江西理工大学外国语学院，江西 赣州" (p. 825) [Authors: Ying Luo, Xiaoping Zou, School of Foreign Languages, Jiangxi University of Science and Technology, Ganzhou, Jiangxi.] 2) "实验组采用 CBI 的主题教学模式，结合《冶金工程英语课程》的 特点，运用 Stoller 提出的 6-T 法...来组织教学。" (p. 827) [The experimental group's teaching was organised by applying the 6-T method within the CBI theme-based model.] 3) "笔者基于 Gardner (1985)的态度动机测试理论，设计了包含 14 个 封闭式问题和 1 个开放式问题的调查问卷。" (p. 829) [Translation: The author designed a questionnaire containing 14 closed questions and 1 open question based on Gardner's attitude-motivation test theory.] 4) "基金项目 江西理工大学校级教改课题..." (p. 830) [Funded by a university-level teaching-reform project of the same university.] Detailed Analysis: Criterion I requires the study to be conducted independently of those who designed the intervention, or at least documented third-party oversight of data collection and analysis. Here the two authors are faculty of the School of Foreign Languages at the same university where the experiment ran; they designed the CBI/6-T instructional intervention, organised the teaching, designed the questionnaire themselves ("笔者...设计"), and analysed all data. There is no mention of any external evaluation team, independent enumerators, blinded assessors, or third-party oversight. The funding is an internal teaching-reform grant of the same university. The use of an externally administered exam (CET-4) provides some objectivity in scoring, but the study's conduct, data handling and analysis remained entirely with the intervention designers. Criterion I is not met because the intervention designers themselves conducted and analysed the study with no documented independent oversight.
  • Y

    Year Duration

    • Criterion T is not met and the paper documents no intervention timeline, so tracking covering at least 75% of an academic year cannot be established.
    • "未发现显著差异究其原因在于教学方法直接影响写作技能加之实验周期 相对不足。" (p. 829)
    • Relevant Quotes: 1) "前测采用学生入学时的高考英语成绩作为基准数据(见表 1)，后测 则使用实验结束后的大学英语四级考试成绩。" (p. 828) 2) "至于作文，未发现显著差异究其原因在于教学方法直接影响写作技 能加之实验周期相对不足。" (p. 829) [The authors attribute null writing effects partly to the relatively insufficient experimental period.] Detailed Analysis: Criterion Y requires outcomes to be measured at least 75% of a full academic year (roughly 9-10 months) after the intervention begins, with clear start and end dates. Per the prompt's rule, if criterion T (term duration) is not met then Y cannot be met; T failed here because no intervention start date, duration or measurement interval is documented anywhere in the paper. The authors additionally describe the experimental period as "relatively insufficient", indicating a short study rather than a year-long one. There is no quoted evidence of tracking approaching an academic year. Criterion Y is not met because criterion T is not met and no year-long (or 75% of a year) tracking interval is documented.
  • B

    Balanced Control Group

    • Both groups took the same Metallurgy Engineering English course in regular class time and differed only in teaching method (CBI 6-T versus traditional), with no extra time, materials or budget given to the experimental group.
    • "实验组接受内容依托式主题教学法(CBI)的指导，而对照组则接受传统 教学法。" (p. 827)
    • Relevant Quotes: 1) "在实验过程中，实验组接受内容依托式主题教学法(CBI)的指导， 而对照组则接受传统教学法。" (p. 827) [During the experiment the experimental group received CBI theme-based instruction while the control group received traditional teaching methods.] 2) "实验组采用 CBI 的主题教学模式，结合《冶金工程英语课程》的 特点，运用 Stoller 提出的 6-T 法(主题 Theme、文本 Text、话题 Topic、线索 Thread、任务 Task、过渡 Transition)来组织教学。" (p. 827) 3) "对照组则采用传统教学法，按照常规方式进行教学。" (p. 827) [The control group used traditional teaching methods in the conventional way.] Detailed Analysis: Criterion B requires that intervention and control groups be balanced in educational time and resources unless extra resources are themselves the treatment variable. Here the intervention is a change of pedagogy inside the existing Metallurgy Engineering English course: the experimental class was taught with the CBI theme-based model organised by the 6-T method (themes, professional texts, topics, threads, tasks, transitions), while the control class received the same course via traditional instruction. Nothing in the paper indicates the experimental group received additional class hours, tutoring, technology, materials budget or other supplementary inputs; both classes of 41 students followed the same course with different instructional organisation. Following the decision tree, EXTRA_RESOURCES_PRESENT is false (the intervention adds no extra time or budget, only a different teaching method within regular instruction), so the balance requirement is trivially satisfied. The selection of metallurgy-related English texts is the instructional content of the method itself rather than a separable resource add-on. Criterion B is met because the intervention changed only the teaching method within the same course time, with no additional time or resources given to the experimental group.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this specific CBI metallurgy-English experiment is reported or found; related prior CBI studies involve one of the same authors or different designs and contexts, and an internet citation search found zero papers citing or replicating this study.
    • Relevant Quotes: 1) "廖根福, 邹晓萍. CBI 理念下的 6-T 主题模式对大学生英语听力理 解能力发展的影响研究[J]. 中国外语, 2016, 13(5): 66-74." (Reference [1], p. 830) [A 2016 study of the 6-T theme model on listening comprehension, co-authored by Xiaoping Zou - the same author as the present paper.] 2) "陈曦蓉(2019)探索了基于慕课与雨课堂的线上线下混合式教学在商 务英语教学中的应用，验证了 CBI 理念在促进学生商务交流能力方面的 有效性。" (p. 827) [A different CBI-related study, in business English with a blended-learning design, not a replication of this trial.] Detailed Analysis: Criterion R requires that this specific study (or its central experimental claim in the same design and context) be independently replicated by a different research team and published in a peer-reviewed journal. The paper itself, being newly published (May 2024), reports no replication. The prior 6-T/CBI study it cites (Liao & Zou, 2016) shares author Xiaoping Zou, so it is not independent, and it addresses listening comprehension in general college English rather than the metallurgy-English CBI experiment reported here. Other cited CBI works (e.g., Chen 2019 on business English blended learning) are different interventions in different contexts, not replications of this trial. As part of this verification, a Semantic Scholar citation lookup on the paper's DOI (10.12677/ae.2024.145772) returned zero citing papers, and Google Scholar / Bing searches for authors Ying Luo and Xiaoping Zou combined with the CBI metallurgy-English topic likewise returned no matching independent replication studies. No independent peer-reviewed replication of this CBI theme-based metallurgy-English RCT was identified. Criterion R is not met because no independent replication of this specific study by a different research team is reported or found, and a dedicated internet search for citing or replicating papers found none.
  • A

    All-subject Exams

    • Only English proficiency (CET-4) was assessed with a standardised exam; the professed professional-knowledge outcome and other subjects were never measured with standardised tests and no exception rationale is given.
    • "后测则使用实验结束后的大学英语四级考试成绩。" (p. 828)
    • Relevant Quotes: 1) "后测则使用实验结束后的大学英语四级考试成绩。" (p. 828) [The post-test used only CET-4 English scores.] 2) "旨在提高学生在冶金专业英语课堂中的语言能力和专业知识水平。" (Abstract, p. 825) [Translation: aiming to improve students' language ability and professional knowledge level in the metallurgy English classroom.] 3) "CBI 主题依托模式在冶金专业英语课堂中对促进学生的学科知识和 语言能力的整合发展有显著性影响" (p. 830) [Conclusion claims impact on the integrated development of subject knowledge and language ability.] Detailed Analysis: Criterion A requires standardised exam-based measurement across all main subjects taught at the education level, or a clearly justified exception for a highly specialised intervention. Criterion E is met (CET-4), so A is not automatically failed, but the study measures only English. Although the paper explicitly claims the intervention promotes both language ability and metallurgical subject knowledge, no standardised assessment of metallurgy or any other core subject of the students' degree programme is reported - the only additional instrument is a custom attitude questionnaire. The paper offers no rationale for restricting measurement to English despite claiming cross-domain benefits, and possible negative or positive effects on the students' other subjects are unexamined. The specialised-intervention exception in the standard applies to upper secondary or vocational education with an explained rationale, which is not provided here. Criterion A is not met because only English was assessed and no justification is given for omitting the other main subjects of the programme, including the professional knowledge the study claims to improve.
  • G

    Graduation Tracking

    • Measurement stopped at the post-experiment CET-4 with no follow-up, criterion Y is not met, and an internet search for subsequent papers by the same authors tracking this cohort found none.
    • "后测则使用实验结束后的大学英语四级考试成绩。" (p. 828)
    • Relevant Quotes: 1) "后测则使用实验结束后的大学英语四级考试成绩。" (p. 828) [The post-test was the CET-4 taken after the experiment ended; no later measurement is described.] 2) "未来研究可以进一步探讨 CBI 教学法在听力和写作技能培养方面的 应用及优化策略。" (p. 829) [Translation: Future research can further explore the application and optimisation of CBI for listening and writing skills - i.e., further work is only proposed, not conducted.] Detailed Analysis: Criterion G requires participants to be tracked until graduation from their educational stage. Per the prompt's rule, G cannot be met when criterion Y is not met, and Y failed here. Substantively, the study's final measurement is the CET-4 taken after the experiment ended; the students of the 2022 cohort would not graduate until around 2026, and no follow-up through to degree completion is described or planned. A Semantic Scholar citation lookup on this paper's DOI returned zero citing papers, and Google Scholar / Bing searches for follow-up publications by Ying Luo or Xiaoping Zou tracking this same 2022-cohort metallurgy-English group found no such papers. Criterion G is not met because outcome measurement ended at the post-experiment CET-4 with no tracking to graduation, and no follow-up publication tracking this cohort was found.
  • P

    Pre-Registered

    • The paper contains no mention of any trial registry, pre-registration ID or pre-specified protocol, and no such registration was found through internet search.
    • Relevant Quotes: 1) "为了评估实验效果，本研究进行了前测和后测... 所有数据均通过 SPSS22.0 软件进行分析处理。" (p. 828) [The methods describe the tests and SPSS analysis with no reference to any registration.] 2) "基金项目 江西理工大学校级教改课题“外语教师 TPACK 能力结构、 翻转课堂及翻转学习的整合研究”(XJG-2019-1)。" (p. 830) [Only a university teaching-reform grant is acknowledged; no registry entry.] Detailed Analysis: Criterion P requires the study protocol - hypotheses, methods and planned analyses - to be pre-registered on a public registry before data collection, with quotable evidence of the registry and its timing. This paper contains no mention of any registry (e.g., ClinicalTrials.gov, ChiCTR, OSF, AEA registry), no registration ID, and no protocol publication. The hypotheses are stated only within the paper itself. This is a small, internally funded teaching-reform study (funded by a university-level teaching-reform grant, XJG-2019-1) of a type not typically pre-registered, and no external search located any pre-registration record for it. There is therefore no evidence that any pre-registration occurred before data collection began. Criterion P is not met because no pre-registration statement, registry link or registration date appears anywhere in the paper, and no registry record was found via internet search.