An Account of Teaching Vocabulary to Indonesian EFL Learners through Web-Based Language Instruction (WLI): Attitude in Focus

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation was performed at the individual student level within a single school, not at the class or school level, and no tutoring exception applies.
    • "Students were randomly assigned to one of two groups: the Experimental group (EG = 30) ... and the control group (CG = 30), which got regular classroom teaching every session."
    • Relevant Quotes: 1) "Students were randomly assigned to one of two groups: the Experimental group (EG = 30), which used free vocabulary learning webpages on the IELTS English language learning site each day for 12 weeks, and the control group (CG = 30), which got regular classroom teaching every session." (Abstract) 2) "To ensure homogeneity among the students, a modified Michigan test was administered to 120 preintermediate students from MA DDI Sendana in Majene Regency on the island of West Sulawesi, Indonesia, to select 60 male and female students with the same level of proficiency. After that, they were separated into two groups: EG (EG = 30) and one CG (CG = 30), each of which had 30 participants." (p. 3) Detailed Analysis: The paper describes randomisation of individual students drawn from a single school (MA DDI Sendana), not of entire classes or the school itself. There is no quote describing assignment by class or by school; instead, 60 individually-screened students (selected from a pool of 120 via a proficiency test) were split into an experimental and a control group. No exception for personal tutoring is stated: the EG condition is a self-study web-based vocabulary tool used independently at home, and the CG condition is "regular classroom teaching," implying a classroom setting rather than one-to-one tutoring. Because randomisation was performed at the individual student level within a single institution, and no class- or school-level unit of randomisation or tutoring exception is documented, this criterion is not satisfied. Criterion C is not met because randomisation occurred at the individual student level within a single school, with no class-level or school-level randomisation described and no valid tutoring exception stated.
  • E

    Exam-based Assessment

    • The outcome assessment is an unnamed 40-item multiple-choice test with no evidence of being a recognised standardised exam.
    • "During the pretest phase, the participants were required to complete the vocabulary test, which consisted of 40 multiple-choice questions."
    • Relevant Quotes: 1) "To ensure homogeneity among the students, a modified Michigan test was administered to 120 preintermediate students ... to select 60 male and female students with the same level of proficiency." (p. 3) 2) "Regarding the acquisition of receptive vocabulary, during the pretest phase, the participants were required to complete the vocabulary test, which consisted of 40 multiple-choice questions." (p. 3) 3) "The receptive vocabulary size was measured using the posttest." (p. 4) Detailed Analysis: The "modified Michigan test" is used only as a screening instrument to select a homogeneous sample of 60 students from an initial pool of 120; it is not described as the outcome measure. The actual pre- and post-test used to evaluate vocabulary gain is described simply as "the vocabulary test, which consisted of 40 multiple-choice questions," with no citation, publisher, or name identifying it as a recognised, standardised assessment. Nor is it stated that this instrument is the (unmodified) Michigan test or any other named standardised exam. Given the lack of evidence that the outcome assessment is a widely recognised standardised test, this criterion is not satisfied. Criterion E is not met because the outcome test is an unnamed, seemingly researcher-selected 40-item multiple-choice vocabulary test with no confirmed standardisation, distinct from the screening-only "modified Michigan test."
  • T

    Term Duration

    • The intervention and measurement window spanned only eight to twelve weeks, short of a full academic term.
    • "Over eight weeks, the EG students studied their course vocabulary using free vocabulary study tools on the IELTS website."
    • Relevant Quotes: 1) "the Experimental group (EG = 30), which used free vocabulary learning webpages on the IELTS English language learning site each day for 12 weeks..." (Abstract) 2) "Over eight weeks, the EG students studied their course vocabulary using free vocabulary study tools on the IELTS website." (pp. 3-4) 3) "After the course, EGs and CGs took a posttest." (p. 4) Detailed Analysis: The paper is internally inconsistent about the intervention length: the abstract states the EG used the web-based tool "each day for 12 weeks," while the Methodology section states the study ran "over eight weeks." Neither figure reaches even the lower bound of one academic term (approximately 3-4 months / roughly 13-16 weeks); the higher of the two reported durations (12 weeks, about 2.5-3 months) is still short of a full term, and the posttest was administered immediately "after the course" with no delayed follow-up. Given that the reported duration under either figure falls short of a term, and the paper does not clarify the discrepancy, this criterion is not satisfied. Criterion T is not met because the intervention lasted only eight to twelve weeks (with the paper itself inconsistent on the exact figure), short of the required one-term interval from intervention start to outcome measurement.
  • D

    Documented Control Group

    • The control group's size, baseline scores, and the "regular classroom teaching" it received are clearly documented and statistically compared to the treatment group.
    • "There are 30 learners in EG and 30 in CG. The mean for EG is 11.2000, and the mean for CG is 11.2667... both groups were at the same level of vocabulary knowledge before the treatment."
    • Relevant Quotes: 1) "To ensure homogeneity among the students, a modified Michigan test was administered to 120 preintermediate students ... to select 60 male and female students with the same level of proficiency." (p. 3) 2) "Table 1 contains essential information concerning group comparisons, such as sample size (n), mean, standard deviation, and standard error. As can be seen, there are 30 learners in EG and 30 in CG. The mean for EG is 11.2000, and the mean for CG is 11.2667." (p. 4) 3) "The p value of Levene's test is printed as '0.863' ... so we conclude that the variance in vocabulary pretest of EG is not significantly different than that of CG. The researchers concluded that both groups were at the same level of vocabulary knowledge before the treatment." (p. 4) 4) "Every week, the CG students were given the same teaching process." (p. 4) Detailed Analysis: The paper reports the control group's size (n=30), baseline vocabulary pretest mean, standard deviation and standard error (Table 1), and an explicit statistical test (Levene's test, independent-samples t-test) confirming baseline equivalence between EG and CG. It also describes what the control group received ("regular classroom teaching every session" / "the same teaching process"). While detailed demographic breakdowns (e.g., age, gender split per group) are not separately reported by condition, the baseline performance data, group size, and description of the "business as usual" instruction the control group received are documented with reasonable clarity. Criterion D is met because the paper documents the control group's size, baseline vocabulary performance (with statistical comparison to the treatment group), and the nature of the instruction it received.

• Level 2 Criteria

  • S

    School-level RCT

    • Only a single school was involved, with individual students allocated to conditions, not a randomisation of multiple schools.
    • "a modified Michigan test was administered to 120 preintermediate students from MA DDI Sendana in Majene Regency ... to select 60 male and female students..."
    • Relevant Quotes: 1) "a modified Michigan test was administered to 120 preintermediate students from MA DDI Sendana in Majene Regency ... to select 60 male and female students with the same level of proficiency." (p. 3) Detailed Analysis: The entire study was conducted within a single school (MA DDI Sendana), with individual students recruited and randomly split into EG and CG within that one institution. There is no randomisation of multiple schools to conditions; only one school is involved at all, so a school-level RCT design (comparing whole schools assigned to different conditions) is not present. Criterion S is not met because the study involves a single school with student-level allocation, not randomisation across multiple schools.
  • I

    Independent Conduct

    • No statement of independent, third-party data collection or evaluation is provided; the authors appear to have designed and conducted the study themselves.
    • Relevant Quotes: 1) "Correspondence should be addressed to Muhammad Aswad; aswad@unsulbar.ac.id and Zeynab Rezaei Gashti; zeynabrg2010@gmail.com" (Author list) 2) "The authors declare that they have no conflicts of interest." (Conflicts of Interest section, p. 6) Detailed Analysis: The paper contains no statement that data collection, implementation, or analysis were carried out by an independent third-party evaluator separate from the authors who designed the study. The methods section describes the authors' own team administering the screening test, running the intervention, and analysing outcomes, with no mention of an external evaluation agency, blinded administrators, or independent oversight body. The "no conflicts of interest" disclosure addresses financial/personal conflicts, not independence of conduct from the study designers. Criterion I is not met because no evidence indicates that an independent, third-party team (rather than the study's own authors) conducted the data collection, implementation, or analysis.
  • Y

    Year Duration

    • The study lasted only eight to twelve weeks, far short of 75% of an academic year, and criterion T is not met.
    • "Over eight weeks, the EG students studied their course vocabulary using free vocabulary study tools on the IELTS website."
    • Relevant Quotes: 1) "Over eight weeks, the EG students studied their course vocabulary using free vocabulary study tools on the IELTS website." (pp. 3-4) 2) "the Experimental group (EG = 30), which used free vocabulary learning webpages on the IELTS English language learning site each day for 12 weeks..." (Abstract) Detailed Analysis: As noted under criterion T, the intervention and outcome measurement together spanned only eight to twelve weeks, which is far short of 75% of an academic year (approximately 9-10 months / about 27-30 weeks). Since the weaker Term Duration criterion (T) is already not met, the stronger Year Duration criterion cannot be met either, per the criteria-specific instruction that Y requires T to be met. Criterion Y is not met because the study duration (eight to twelve weeks) is far shorter than 75% of an academic year, and criterion T is also not met.
  • B

    Balanced Control Group

    • The extra resources (laptop, internet access) given to the treatment group are the explicit treatment variable being tested against a business-as-usual classroom control.
    • "The independent variable is the use of the Internet as a collaborative tool, while the dependent variable is the improvement in the vocabulary level of the students."
    • Relevant Quotes: 1) "The independent variable is the use of the Internet as a collaborative tool, while the dependent variable is the improvement in the vocabulary level of the students." (p. 3) 2) "Over eight weeks, the EG students studied their course vocabulary using free vocabulary study tools on the IELTS website. ... Computers were available to students 24 hours a day. Each student received a laptop and instructions on using it at home to study for the IELTS vocabulary test independently. Every week, the CG students were given the same teaching process." (pp. 3-4) Detailed Analysis: The EG received additional resources not given to the CG: a personal laptop, 24-hour computer/internet access, and a specific web-based (IELTS) vocabulary tool for independent, self-paced study. Applying the criterion B decision procedure: extra resources are present (laptop, internet access); however, the paper explicitly frames "the use of the Internet as a collaborative tool" as the independent variable itself -- i.e., the very thing being tested is whether web-based instruction (WLI), including its associated hardware and internet access, outperforms conventional classroom teaching. These resources are therefore INTEGRAL to the treatment variable being tested, not a supplementary, separable add-on. The CG, by contrast, received "the same teaching process" it would ordinarily receive (regular classroom instruction), representing the business-as-usual baseline against which the web-based tool is compared. Because the additional resources are integral to and inseparable from the treatment variable under investigation, the study is not required to give the CG equivalent hardware/internet access. Criterion B is met because the additional resources (laptop and internet-based vocabulary tool) given to the EG are the explicit treatment variable being tested against a business-as-usual classroom-teaching control condition.

• Level 3 Criteria

  • R

    Reproduced

    • The authors propose replication only as future work; no independent replication of this study has been reported or found through internet search.
    • "This experiment may be replicated to see if the same findings are achieved."
    • Relevant Quotes: 1) "This experiment may be replicated to see if the same findings are achieved." (p. 6, Discussion and Conclusion) Internet Search Findings: A search was conducted for independent replications of this specific study (Aswad, Yassi, Pammu, Nasmilah, and Rezaei Gashti, 2022, "An Account of Teaching Vocabulary to Indonesian EFL Learners through Web-Based Language Instruction (WLI): Attitude in Focus," Education Research International). The paper is cited as background literature by several later EFL/vocabulary papers (e.g., studies on CALL, crossword puzzles, and digital reading in EFL vocabulary instruction), but no paper was found that independently reproduces this specific study's WLI intervention, design, and setting with a different research team. No independent replication of this particular study was identified. Detailed Analysis: The authors themselves note that the study "may be replicated" as a suggestion for future research, which confirms that no replication had occurred as of publication. No quotes in the paper reference a prior or subsequent independent replication of this specific WLI-vocabulary intervention by a different research team, and the internet search above did not locate any external replication of this particular study. Criterion R is not met because the authors explicitly propose replication as future work rather than reporting an existing independent replication, and no independent replication was found via internet search.
  • A

    All-subject Exams

    • Only vocabulary outcomes in a single subject area were measured, and criterion E is not met.
    • "The vocabulary of EFL students improved as a result of WBL education."
    • Relevant Quotes: 1) "Regarding the acquisition of receptive vocabulary, during the pretest phase, the participants were required to complete the vocabulary test..." (p. 3) 2) "The vocabulary of EFL students improved as a result of WBL education." (Abstract) Detailed Analysis: The study measures only vocabulary/language outcomes within the single subject area of EFL vocabulary; no other core subjects (mathematics, science, etc.) were assessed. In addition, because criterion E (Exam-based Assessment) is not met, per the criteria-specific instructions criterion A cannot be met either. Criterion A is not met because only vocabulary outcomes in a single subject were assessed, and criterion E is also not met.
  • G

    Graduation Tracking

    • No follow-up tracking beyond the immediate posttest is reported or found through internet search, and criterion Y is not met.
    • "After the course, EGs and CGs took a posttest."
    • Relevant Quotes: 1) "After the course, EGs and CGs took a posttest." (p. 4) 2) "Lastly, in the final session, an attitude questionnaire was given to students..." (p. 4) Internet Search Findings: A search was conducted for follow-up publications by the same author team (Muhammad Aswad and colleagues, Universitas Hasanuddin / Universitas Sulawesi Barat) tracking the same cohort of 60 students from MA DDI Sendana toward graduation. No such follow-up study was found; subsequent identifiable work by Aswad and co-authors addresses unrelated topics (e.g., teachers' talk and student attitudes, pair-interaction models), not continued tracking of this cohort. Detailed Analysis: Data collection ended with a posttest and an attitude questionnaire administered in the "final session" immediately after the eight-to-twelve-week intervention. There is no mention of any further follow-up, let alone tracking students until graduation, and no follow-up publication was located. Since criterion Y (Year Duration) is not met, per the criteria-specific instructions criterion G cannot be met either. Criterion G is not met because data collection stopped immediately after the intervention with no follow-up tracking located via internet search, and criterion Y is also not met.
  • P

    Pre-Registered

    • No pre-registration of the study protocol is mentioned anywhere in the paper, and none was found through internet search.
    • Relevant Quotes: (No quotes found referencing pre-registration.) Internet Search Findings: Because the paper contains no reference to a pre-registration platform, registry ID, or protocol publication, no registry lookup was applicable. A general search for a trial registration associated with this study or its authors did not locate any pre-registration record. Detailed Analysis: The paper contains no statement of a study protocol, hypotheses, or analysis plan being registered on any public registry (e.g., a trial registry) prior to data collection. Only a "Data Availability" and "Conflicts of Interest" statement are provided; neither mentions pre-registration. Criterion P is not met because no evidence of pre-registration is present anywhere in the paper or identifiable via internet search.