English Vocabulary with a Picture Application for Enhancing Thai EFL Students' Daily English Vocabulary Memorization

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Two intact classes (not individual students within one class) were randomly assigned to the experimental and control conditions, satisfying the class-level randomisation requirement despite the very small number of clusters.
    • "Two intact classes were randomly assigned to the control and the experimental group, with 24 students serving as the control group and 24 students as the experimental group (EVP)." (p. 718)
    • Relevant Quotes: 1) "The research was a quasi-experimental study designed by a randomized pre-test and post-test control group design (McMillan & Schumacher, 2010) with 2 intact groups: a control group and an experimental group." (p. 718) 2) "These subjects were selected through Simple Random Sampling. Two intact classes were randomly assigned to the control and the experimental group, with 24 students serving as the control group and 24 students as the experimental group (EVP)." (p. 718) 3) "At the beginning of the study, forty-eight participants were two intact groups that were equally and randomly divided into one control group and one experimental group." (p. 721) 4) "The participants were 48 undergraduate students who were randomly assigned to the experimental and the control group." (Abstract, p. 712) Detailed Analysis: The methods section states explicitly that the unit of allocation was the intact class: "Two intact classes were randomly assigned to the control and the experimental group." Although the abstract loosely says students "were randomly assigned," the more detailed methodology clarifies that whole pre-existing classes, not individual students within a single class, were randomised to conditions. This design keeps the treatment and control participants in separate classrooms, which is exactly the contamination protection that criterion C targets. Weaknesses remain: the paper self-describes as "quasi-experimental," only two clusters were randomised (so class and condition are fully confounded), and the mechanics of the random assignment are not described in detail. However, the ERCT procedure asks whether the paper clearly states that randomisation was at the class level or stronger, and the quoted statements do state class-level random assignment. Criterion C is met because the paper explicitly states that two intact classes were randomly assigned to the experimental and control conditions, i.e., randomisation at the class level.
  • E

    Exam-based Assessment

    • Outcomes were measured with a researcher-made 70-item multiple-choice vocabulary test built from the study's own target words, not a widely recognised standardised exam.
    • "The daily English vocabulary pre-test required participants to choose the appropriate words in an English sentence, with four choices for each question and a total of seventy questions. The target words were the vocabulary items to be learned with EVP from activities 1-10." (p. 720)
    • Relevant Quotes: 1) "The main instruments included EVP, the daily English vocabulary pre-and post-test, and a questionnaire of students' opinions toward learning vocabulary with EVP." (p. 719) 2) "The daily English vocabulary pre-test required participants to choose the appropriate words in an English sentence, with four choices for each question and a total of seventy questions. The target words were the vocabulary items to be learned with EVP from activities 1-10. For construct test validity, the test items were sent to three English language experts to evaluate the appropriateness of CVI, which must not be less than 0.80 (Di lorio, 2005). The CVI of pre-test was 0.94 which showed the appropriate standard to employ." (p. 720) 3) "To measure participants' vocabulary memorization gains achieved through EVP, the daily English vocabulary post-test consisted of the same 70 target words measured in the pre-test." (p. 721) 4) "Their English proficiency level was low-intermediate, a score ranging from 21-40, according to Ordinary National Educational Test (Thai standardized test for university admission)." (p. 718) Detailed Analysis: The outcome instrument was a custom pre/post multiple-choice test constructed by the researchers around the 70 target words taught in the EVP application itself. This is precisely the situation the E criterion warns against: a bespoke test tightly aligned to the intervention content, which can inflate apparent effectiveness. Expert content-validity checks (CVI = 0.94) do not make the instrument a widely recognised standardised exam. The only standardised test mentioned, the Thai Ordinary National Educational Test, was used solely to characterise participants' baseline proficiency, not as an outcome measure. Criterion E is not met because the outcome was assessed with a custom researcher-made vocabulary test rather than a standardised exam.
  • T

    Term Duration

    • The intervention consisted of ten application activities run in August 2021 with the post-test given immediately upon completion of the treatment, far short of the one-term interval between intervention start and outcome measurement.
    • "However, a caveat is the time limitation of this research. The study should be reassessed after the completion of an experiment in three months and six months to investigate how much learners' vocabulary memorization can be retained in a certain period of time." (p. 726)
    • Relevant Quotes: 1) "EVP was employed during the first semester in August 2021 as shown in the detail of the application's instruction." (p. 721) 2) "EVP application includes ten activities with a 50-minute time duration of practice per activity. Seventy target words are displayed in the application with seven words per activity." (p. 720) 3) "Upon the completion of the treatment, the daily English vocabulary post-test was given to both the control group and the experimental group without prior notice..." (p. 721) 4) "However, a caveat is the time limitation of this research. The study should be reassessed after the completion of an experiment in three months and six months to investigate how much learners' vocabulary memorization can be retained in a certain period of time." (p. 726) Detailed Analysis: The T criterion requires at least one full academic term (roughly 3-4 months) between the start of the intervention and the primary outcome measurement. Here the intervention comprised ten activities (about 50 minutes of practice plus a 40-minute supplementary exercise each) delivered in August 2021, and the post-test was administered immediately "upon the completion of the treatment," with no delayed follow-up. The authors themselves acknowledge the "time limitation" of the research and recommend re-assessment at three and six months, confirming that no term-long tracking interval occurred. The elapsed time from intervention start to measurement is on the order of weeks at most, well below one term. Criterion T is not met because outcomes were measured immediately at the end of a short intervention, with no term-long interval from the intervention start.
  • D

    Documented Control Group

    • The control group's size, gender composition, baseline equivalence on the pre-test, and the traditional instruction it received are all explicitly documented.
    • "Instead, they were taught vocabulary through a traditional method used in many Thai schools for teaching English to students. The vocabulary was taught by a teacher, and participants had time to memorize the words and write them down on the paper." (p. 721)
    • Relevant Quotes: 1) "...with 24 students serving as the control group and 24 students as the experimental group (EVP)." (p. 718) 2) "...while the participants in the control group included five males (20.83%) and nineteen females (79.17%)." (p. 718) 3) "They were all native speakers of Thai, and they were homogeneous in terms of English learning experiences with the same number of years they had been exposed to EFL instruction. None of them had lived in an English-speaking country prior to conducting the study. Their English proficiency level was low-intermediate, a score ranging from 21-40, according to Ordinary National Educational Test..." (p. 718) 4) "The participants in the control group did not have experience working with EVP. Instead, they were taught vocabulary through a traditional method used in many Thai schools for teaching English to students. The vocabulary was taught by a teacher, and participants had time to memorize the words and write them down on the paper. The teacher then repeated the words and encouraged them to visualize the words and meaning. At the end of the class, the participants were given dictation practice." (p. 721) 5) "...the insignificant vocabulary gains (t = -1.35, p = .190) were demonstrated by the participants with a mean score 28.29 (SD = 3.65) in the pre-test (before treatment) and 29.42 (SD = 5.71) in the post-test (after treatment)." (p. 723) 6) "...the mean scores of pre-test of both groups were very similar to each other (the experimental group = 28.17, the control group = 28.29). It is assumed that both groups had similar knowledge about the target daily vocabulary before they were exposed to the treatment." (pp. 723-724) Detailed Analysis: The paper documents the control group in reasonable detail: its size (n = 24), gender breakdown (Table 1), shared demographic and language-background characteristics with the treatment group, baseline pre-test performance (Table 3), and a concrete description of the condition it experienced (traditional teacher-led vocabulary instruction with memorisation, writing, repetition, visualisation prompts, and dictation). Baseline comparability is explicitly checked via the very similar pre-test means (28.17 vs 28.29). This satisfies the D requirement of documented composition, baseline performance, and treatment received. Criterion D is met because the control group's size, demographics, baseline scores, and instructional condition are clearly documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation involved only two intact classes within a single university; no schools or institutions were randomised.
    • "Two intact classes were randomly assigned to the control and the experimental group, with 24 students serving as the control group and 24 students as the experimental group (EVP)." (p. 718)
    • Relevant Quotes: 1) "The subjects of this study comprised 48 Thai undergraduate students at Rajamangala University of Technology Tawan-ok." (p. 718) 2) "Two intact classes were randomly assigned to the control and the experimental group, with 24 students serving as the control group and 24 students as the experimental group (EVP)." (p. 718) Detailed Analysis: The S criterion requires randomisation among schools or equivalent implementing institutions/sites. This study took place entirely within one university, and the unit of randomisation was the intact class (two classes in total). There is no mention of multiple institutions, sites, or any school-level assignment procedure anywhere in the paper. Criterion S is not met because randomisation occurred between two classes within a single university, not among schools or sites.
  • I

    Independent Conduct

    • The same two authors developed the EVP application and conducted, analysed, and reported the evaluation themselves, with no independent evaluation team or third-party oversight.
    • "...the purpose of this study was to develop EVP based on the picture mnemonic technique and the generative theory of multimedia learning as computer-assisted vocabulary learning material and examine the effectiveness of EVP used in the classroom." (p. 715)
    • Relevant Quotes: 1) "...the purpose of this study was to develop EVP based on the picture mnemonic technique and the generative theory of multimedia learning as computer-assisted vocabulary learning material and examine the effectiveness of EVP used in the classroom." (p. 715) 2) "EVP application used in this study was developed and designed systematically using the ADDIE model comprising 5 steps; analysis, design, development, implementation, and evaluation (McGriff, 2000)." (p. 719) 3) "For each target word's picture and meaning shown on the application screen, the researchers provided pictures and the Thai meanings of the words to be used as target words in EVP application..." (p. 720) 4) "This study was granted by the University fund budget revenue from Rajamangala University of Technology Tawan-ok and approved by The Research Ethics Review Committee for Research Involving Human Research Subjects (COA No. 018/RMUTTO REC No. 036/2021)." (p. 727) Detailed Analysis: Criterion I requires that the study be conducted independently of the intervention's designers, or at least that data collection and analysis be handled by parties with no stake in the intervention. Here the two authors both developed the EVP application (via the ADDIE process, selecting words and pictures) and then implemented the trial, administered the tests, and analysed the results at their own institution. No external evaluators, independent test administrators, or third-party oversight of data collection or analysis are mentioned anywhere in the paper; ethics approval and university funding do not constitute independent conduct. Criterion I is not met because the intervention designers themselves conducted and analysed the evaluation without any documented independent oversight.
  • Y

    Year Duration

    • The study ran only a short set of activities in August 2021 with immediate post-testing, nowhere near 75% of an academic year, and criterion T already fails.
    • "EVP was employed during the first semester in August 2021 as shown in the detail of the application's instruction." (p. 721)
    • Relevant Quotes: 1) "EVP was employed during the first semester in August 2021 as shown in the detail of the application's instruction." (p. 721) 2) "Upon the completion of the treatment, the daily English vocabulary post-test was given to both the control group and the experimental group without prior notice..." (p. 721) 3) "However, a caveat is the time limitation of this research. The study should be reassessed after the completion of an experiment in three months and six months..." (p. 726) Detailed Analysis: The Y criterion requires outcome measurement at least 75% of an academic year (roughly 9-10 months) after the intervention begins. This study delivered ten application-based activities within a single month (August 2021) and measured outcomes immediately upon completion of the treatment. The authors explicitly flag the study's time limitation and call for 3- and 6-month follow-ups, which were not conducted. Additionally, per the instructions, because criterion T (Term Duration) is not met, criterion Y cannot be met either. Criterion Y is not met because the interval from intervention start to measurement was weeks, far below 75% of an academic year, and the prerequisite criterion T also fails.
  • B

    Balanced Control Group

    • Both groups received vocabulary instruction on the same 70 target words during the study period; the control had an active traditional teacher-led condition, and the EVP application itself was the integral treatment variable being tested against that business-as-usual method.
    • "Instead, they were taught vocabulary through a traditional method used in many Thai schools for teaching English to students. The vocabulary was taught by a teacher, and participants had time to memorize the words and write them down on the paper. The teacher then repeated the words and encouraged them to visualize the words and meaning. At the end of the class, the participants were given dictation practice." (p. 721)
    • Relevant Quotes: 1) "During the treatment period, EVP was used in the experimental group, and none of the programs were used in the control group." (p. 721) 2) "The participants in the control group did not have experience working with EVP. Instead, they were taught vocabulary through a traditional method used in many Thai schools for teaching English to students. The vocabulary was taught by a teacher, and participants had time to memorize the words and write them down on the paper. The teacher then repeated the words and encouraged them to visualize the words and meaning. At the end of the class, the participants were given dictation practice." (p. 721) 3) "EVP application includes ten activities with a 50-minute time duration of practice per activity. Seventy target words are displayed in the application with seven words per activity." (p. 720) 4) "A forty-minute supplementary exercise provided at the end of each activity is composed of a Quiz Game... and a Timer Game..." (p. 720) 5) "...this experimental study aimed to develop English Vocabulary with a Picture Application (EVP) for improving students' daily English vocabulary memorization and examine the effectiveness of EVP used between an experimental group and a control group." (Abstract, p. 712) Detailed Analysis: Applying the criterion B decision tree from the current ERCT specification: extra resources are present (the EVP application's structured practice and games are not something the control group also received), so the first branch does not apply. However, these resources are the treatment variable itself, explicitly framed as such ("examine the effectiveness of EVP used between an experimental group and a control group"), which satisfies the RESOURCES_ARE_TREATMENT branch and is sufficient on its own for "met," regardless of whether the control group's time exactly matches. Independently, the control group was also not left untreated: it received an active, teacher-led traditional vocabulary instruction condition covering the same 70 target words during the same study period, including presentation, memorisation time, writing, repetition, visualisation prompts, and dictation practice, so both groups received structured educational time on the target content. The paper does not report minute-by- minute time matching between conditions (EVP activities were 50 minutes of practice plus a 40-minute exercise each), which is a documentation weakness, but the technology and its practice format are integral to the EVP intervention being compared against business-as-usual traditional instruction, matching the standard's exception examples (e.g., DPL tool vs teach-as- usual). The additional resource (the application with its games and self-paced practice) is clearly noted here as an integral part of the tested intervention. Criterion B is met because the EVP application's added technology and practice format are integral to the treatment variable being tested, and the control group additionally received an active traditional instruction condition on the same target vocabulary rather than no intervention at all.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this specific EVP study by a different research team is reported in the paper or found via internet citation search of the literature.
    • "Further study could investigate the achievement of EVP in teaching vocabulary for different purposes in various contexts." (Abstract, p. 712)
    • Relevant Quotes: 1) "As reviewed above, the English vocabulary learning application designed by the integration of visual mnemonic devices, especially picture presentation with multimedia learning has not been comprehensively studied." (p. 717) 2) "Further study could investigate the achievement of EVP in teaching vocabulary for different purposes in various contexts." (Abstract, p. 712) 3) "Further study should be made to compare the effectiveness of using the application with other mnemonic devices or other approaches to determine which method works better." (p. 726) Detailed Analysis: Criterion R requires independent replication of this specific study by a different research team in a different context, published in a peer-reviewed journal. The paper explicitly positions itself as filling a gap ("has not been comprehensively studied") and calls for further studies, indicating the work is novel rather than replicated. The related studies it cites (e.g., Farrokh et al., 2021 on visual mnemonics; Hirschel & Fritz, 2013 on CALL; Alfadil, 2020 on VR vocabulary games) test different interventions and are not replications of the EVP application trial. For this verification, an internet search of the Semantic Scholar citation graph for DOI 10.61508/refl.v30i3.268078 (accessed 2026-07-27) was performed, since the ERCT standard allows evidence of reproduction to appear in papers that cite the original study. It returned 4 citing works: "Enhancing high school students' writing proficiency through academic writing instruction: a Cambridge FCE exam-based study" (Ochoa, Soto, & Novillo Sarmiento, 2025); "A Systematic Review on the Effectiveness of Digital Game-Based Learning in the Development of Vocabulary Skills" (Morales, Castillo, & Cuarte, 2024); "Can Picture Dictionaries Unlock Vocabulary Mastery? Unveiling Insights Through Quasi-Experimental Research" (Menggo, Amul, & Ndiung, 2024); and "Contextual Crosswords: A Gamified Approach to Slang Acquisition and Critical Thinking Skills" (Fitri, Agustina, & Setiawan, year unspecified). Based on their titles and topics (writing proficiency assessment, a review article, picture dictionaries rather than the EVP application, and crossword-based slang learning), none of these is an independent replication of this specific EVP picture-mnemonic application study in a new context; they cite it as background literature only. No verbatim quotes could be extracted from these citing papers' full texts within the scope of this verification, so none are fabricated or presented here; only their bibliographic identification is reported. No other independent replication of this 2023 EVP study was identified. Criterion R is not met because no independent replication of this specific EVP study was found in the paper or via citation search.
  • A

    All-subject Exams

    • Only English vocabulary was assessed, with a custom test, so neither the all-subject coverage nor the prerequisite standardised-exam criterion E is satisfied.
    • "To measure participants' vocabulary memorization gains achieved through EVP, the daily English vocabulary post-test consisted of the same 70 target words measured in the pre-test." (p. 721)
    • Relevant Quotes: 1) "To measure participants' vocabulary memorization gains achieved through EVP, the daily English vocabulary post-test consisted of the same 70 target words measured in the pre-test." (p. 721) 2) "...the learners' ability in daily English vocabulary memorization was measured by means of the multiple-choice daily English vocabulary pre-and post-tests..." (p. 722) Detailed Analysis: Criterion A requires standardised exam-based assessment across all main subjects. This study measured a single narrow outcome: memorisation of 70 daily English vocabulary items, using a researcher-made test. No other subjects were assessed, and because criterion E is not met (the assessment is custom-made, not standardised), criterion A automatically fails as well per the instructions. Although the participants are university students in a general English course, the paper offers no justification framing this as a specialised-programme exception with standardised related-subject exams. Criterion A is not met because only English vocabulary was measured with a custom test and prerequisite criterion E fails.
  • G

    Graduation Tracking

    • Measurement stopped immediately after the treatment with no follow-up at all, let alone tracking of participants to graduation, and the prerequisite criterion Y fails.
    • "The study should be reassessed after the completion of an experiment in three months and six months to investigate how much learners' vocabulary memorization can be retained in a certain period of time." (p. 726)
    • Relevant Quotes: 1) "Upon the completion of the treatment, the daily English vocabulary post-test was given to both the control group and the experimental group without prior notice..." (p. 721) 2) "However, a caveat is the time limitation of this research. The study should be reassessed after the completion of an experiment in three months and six months to investigate how much learners' vocabulary memorization can be retained in a certain period of time." (p. 726) Detailed Analysis: Criterion G requires tracking participants until graduation from their educational stage. This study ended with an immediate post-test; the authors acknowledge the absence of any delayed follow-up and merely recommend 3- and 6-month re-assessments for future work. There is no mention of tracking the undergraduate participants to degree completion. For this verification, an internet search (Crossref, accessed 2026-07-27) for other works by the same authors, Danuchawat Suwanasilp and Nunpaporn Durongbhandhu, was conducted to check for follow-up publications tracking this same cohort. It found "Effectiveness of Multimodal Glossing Reading Program on English Vocabulary Acquisition" (Durongbhandhu & Suwanasilp, 2021, English Language Teaching, 14(6), 62) and "Fostering Rajamangala University Students' English Reading and Writing Proficiency Through Cultural Content-Based Language Instruction" (Durongbhandhu, Yodmanee, & Suwanasilp, 2025, The EUrASEANs, 5(54), 617-627). Neither title indicates a follow-up of the 2021 EVP cohort, and no other publication tracking these 48 participants toward graduation was found. In addition, per the instructions, criterion G cannot be met when criterion Y is not met. Criterion G is not met because no follow-up beyond the immediate post-test was conducted, no graduation-tracking follow-up publication by the same authors was found via internet search, and prerequisite criterion Y fails.
  • P

    Pre-Registered

    • The paper contains no mention of any pre-registration of the study protocol on a registry before data collection, and no matching registration was found via internet search.
    • Relevant Quotes: 1) "This study was granted by the University fund budget revenue from Rajamangala University of Technology Tawan-ok and approved by The Research Ethics Review Committee for Research Involving Human Research Subjects (COA No. 018/RMUTTO REC No. 036/2021)." (p. 727) Detailed Analysis: Criterion P requires the full study protocol to be pre-registered on a public registry (with hypotheses, methods, and planned analyses) before data collection begins. The only procedural approval mentioned is institutional ethics review; ethics approval is not pre-registration. No registry platform (e.g., ClinicalTrials.gov, OSF, ISRCTN), registration ID, or registration date appears anywhere in the paper. A search of the OSF registries (accessed 2026-07-27) for "English Vocabulary with a Picture Application" returned no matching registration, consistent with the paper's own silence on pre-registration. Criterion P is not met because no pre-registration statement, registry entry, or registration date is provided in the paper or found via internet search.