Effects of Memrise on Vietnamese EFL Students' Vocabulary: A Case Study at a College in a Rural Area

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • No random assignment of classes or students to conditions is described anywhere in the paper; two intact classes were selected because their baseline scores were similar.
    • "these two groups (Group 2: Traditional teaching method; Group 4: Memrise) were eligible to participate in the intervention." (p. 4)
    • Relevant Quotes: 1) "the researcher sent invitations to 148 first-year students, equivalent to six classes." (p. 4) 2) "The number of students in each group in order was 26 (Group 1), 24 (Group 2), 24 (Group 3), 23 (Group 4), 25 (Group 5), and 25 (Group 6)." (p. 4) 3) "There was a significant difference in the lexicon knowledge of the groups, except for Group 2 and Group 4, when compared with each other. Therefore, these two groups (Group 2: Traditional teaching method; Group 4: Memrise) were eligible to participate in the intervention." (p. 4) Detailed Analysis: The study used six pre-existing intact classes. Two of them (Groups 2 and 4) were selected for the comparison because their baseline vocabulary screening-test scores happened to be statistically similar, not because of any randomisation procedure. There is no quote anywhere in the Methods section describing a random number generator, coin flip, lottery, or any comparable random-allocation method used to assign a class to the "Memrise" or "traditional teaching method" condition. The tutoring exception does not apply, since this is whole-class vocabulary instruction, not one-to-one tutoring. Assignment therefore appears to be a matched, non-random (quasi-experimental) selection of intact classes. Because no randomisation of classes (or students) to condition is documented, criterion C is not met.
  • E

    Exam-based Assessment

    • The outcome measure was a custom vocabulary test built by the researchers themselves, not a standardised, widely recognised exam.
    • "The screening test served as a critical instrument for data collection and was designed to equitably compare the efficacy of the two treatment conditions." (p. 5)
    • Relevant Quotes: 1) "The screening test served as a critical instrument for data collection... 72 words that were answered incorrectly by students in both groups were identified. These words were written on individual pieces of paper, folded, and placed in a cardboard box. Team members then took turns drawing these pieces of paper until a total of 50 words were selected." (p. 5) 2) "a pilot test was conducted on a subset of 30 students... Three experts also reviewed the questions for appropriateness and relevance to the study's objectives... The Cronbach's alpha value was found to be 0.87." (p. 5) 3) "The posttest was designed to assess students' memorisation of these 50 words through four types of questions including: 1) word completion exercises (10 words); 2) word completion exercises (10 words); 3) multiple-choice questions (15 words); and 4) word selection tasks (15 words)." (p. 5) Detailed Analysis: The vocabulary test was constructed specifically for this study: words were hand-picked from the students' own textbook and drawn from a pool of items the target sample had answered incorrectly on a screening test, and the final items were reviewed only by the researchers' own experts and piloted on a small convenience sample. The reported Cronbach's alpha of 0.87 demonstrates internal reliability of this custom instrument, not that it is a widely recognised, externally validated standardised exam (e.g., a national curriculum test). No such standard exam is used anywhere in the study. Because the assessment is a purpose-built, custom test rather than a standard, widely recognised exam, criterion E is not met.
  • T

    Term Duration

    • The 15-week intervention period plus the following posttest stage covers an interval consistent with at least one academic term.
    • "15 weeks While-intervention ... 2 weeks Postintervention Posttest" (Table 1, pp. 6-7)
    • Relevant Quotes: 1) "2 weeks Preintervention Pretest" (Table 1, p. 6) 2) "15 weeks While-intervention" (Table 1, p. 6) 3) "2 weeks Postintervention Posttest ... At the end of the treatment, a posttest was run to check the students' vocabulary development in the two groups." (Table 1, p. 7) Detailed Analysis: Table 1 shows the "while-intervention" teaching stage ran for 15 weeks, and the posttest measuring the primary outcome was administered in the following 2-week post-intervention stage, i.e. after the full 15-week teaching period. The interval from intervention start to outcome measurement is therefore roughly 15-17 weeks (about 3.5-4 months), which is consistent with the ERCT definition of a term (approximately 3-4 months). Although the paper never uses the word "term" explicitly, the reported duration meets or exceeds this threshold. Criterion T is met because the interval from intervention start to measurement covers at least one academic term.
  • D

    Documented Control Group

    • The control group's size, baseline score, demographic profile, and exact instructional treatment at each stage are documented in the paper.
    • "In the second stage, the students were instructed to work with handouts to review the words they had learned. To start learning with the handouts, the teacher introduced the words for learning with their provided Vietnamese meanings, sound, and parts of speech." (Table 1, p. 7)
    • Relevant Quotes: 1) "The number of students in each group in order was 26 (Group 1), 24 (Group 2), 24 (Group 3), 23 (Group 4)..." (p. 4) 2) "the average scores of groups from Group 1 to Group 6 were 60, 30, 52, 29, 73, and 80, respectively. There was a significant difference in the lexicon knowledge of the groups, except for Group 2 and Group 4..." (p. 4) 3) "Participants were students studying at a Medical college in the Southwest of Vietnam, specifically majoring in Nursing. Their average age was around 19 years old, and their level of English was generally intermediate according to the CEFR." (p. 4) 4) "In the second stage, the students were instructed to work with handouts to review the words they had learned. To start learning with the handouts, the teacher introduced the words for learning with their provided Vietnamese meanings, sound, and parts of speech." (Table 1, p. 7) Detailed Analysis: The control group (Group 2, n=24) is documented with respect to sample size, baseline vocabulary score (mean 30, not significantly different from the experimental Group 4's score of 29), the shared demographic profile of the overall sample (~19-year-old nursing majors, CEFR-intermediate English), and a stage-by-stage description of exactly what instructional treatment the control condition received (identical Steps 1, 2.1, 3 and 4 as the experimental group, with handout-based rather than Memrise-based review in Step 2.2). This level of detail is sufficient for a reader to judge whether the control group is a fair, comparable group. Criterion D is met because the control group's composition, baseline data, and treatment are clearly documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Only one institution was involved, and there is no school-level (or any-level) randomisation described.
    • "Participants were students studying at a Medical college in the Southwest of Vietnam" (p. 4)
    • Relevant Quotes: 1) "Participants were students studying at a Medical college in the Southwest of Vietnam, specifically majoring in Nursing." (p. 4) 2) "the researcher sent invitations to 148 first-year students, equivalent to six classes." (p. 4) Detailed Analysis: The entire study was conducted within a single college, using two of six intact classes at that one institution; no other schools or institutions were involved, and (as established under criterion C) there is no randomisation at any level, let alone at the school level. Criterion S is not met because only a single institution participated and no school-level randomisation occurred.
  • I

    Independent Conduct

    • The same research team designed the intervention, supervised its delivery, and analysed the results, with no independent evaluator involved.
    • "The research team designed all the lesson plans used in this current study to reduce the fact that the differences in the teachers' teaching styles would affect the research results." (p. 5)
    • Relevant Quotes: 1) "Two volunteer lecturers were also invited as collaborators of the study." (p. 4) 2) "Before the intervention was implemented, the research team and these two teachers, together, had several sessions to discuss lesson plans that they would use to teach the students. The research team designed all the lesson plans used in this current study..." (p. 5) 3) "Then, two teachers taught demos under the observation of the research team to evaluate and adjust their teaching to best serve the purpose of this study." (p. 5) Detailed Analysis: The same research team that designed the study also designed all lesson plans, selected the two participating teachers, closely supervised and adjusted their teaching ("under the observation of the research team"), and prepared and validated the vocabulary tests. There is no mention of an independent, external evaluator or third-party agency conducting data collection or analysis. Criterion I is not met because the intervention design, implementation oversight, and evaluation were all handled by the same research team.
  • Y

    Year Duration

    • The total study duration of about 19 weeks falls well short of 75% of an academic year.
    • "15 weeks While-intervention" (Table 1, p. 6)
    • Relevant Quotes: 1) "2 weeks Preintervention Pretest ... 15 weeks While-intervention ... 2 weeks Postintervention Posttest" (Table 1, pp. 6-7) Detailed Analysis: The entire study, from pretest through posttest, spans approximately 19 weeks (2 + 15 + 2), far short of 75% of an academic year (roughly 27-30 weeks out of a ~36-40 week academic year). Criterion Y requires the tracking period from intervention start to measurement to cover at least 75% of a full academic year, and the reported timeline here is well under half of that. Criterion Y is not met because the study duration is far shorter than the required 75% of an academic year.
  • B

    Balanced Control Group

    • The control group received a closely matched substitute activity (handout review of the same words, same time and content) for the same instructional steps, isolating the review medium as the tested variable.
    • "In the second stage, the students were instructed to work with handouts to review the words they had learned." (Table 1, p. 7)
    • Relevant Quotes: 1) "In the second stage, the students were instructed to work with Memrise to review the words they had learned. To start learning with Memrise, the teacher introduced the words for learning with their provided Vietnamese meanings, sound, and parts of speech." (Table 1, p. 7) 2) "In the second stage, the students were instructed to work with handouts to review the words they had learned. To start learning with the handouts, the teacher introduced the words for learning with their provided Vietnamese meanings, sound, and parts of speech." (Table 1, p. 7) 3) "Step 1... Step 2.1 Teaching vocabulary... Step 3 Checking students' understanding of vocabulary... Step 4 Using the words." (Table 1, pp. 6-7) 4) "After the treatment, nine students were invited for the interview, which was conducted via Zoom." vs "x — 'X' means to do nothing." (Table 1, p. 7) Detailed Analysis: Steps 1, 2.1, 3 and 4 of the lesson sequence are identical for both groups, occupying the same allotted class time. Even Step 2.2 (the review stage) is matched in structure and content: both groups review the exact same word list with the same provided information (Vietnamese meaning, sound, part of speech), the only difference being the medium (Memrise app vs. paper handouts) within the same lesson slot. The intervention therefore adds no extra class time or budget relative to the control condition; the review medium itself is the treatment variable under investigation, and the handout condition is a well-matched "business-as-usual" comparator providing an equivalent educational substitute. The one asymmetry - only the experimental group's students were invited to post-treatment interviews - is a qualitative data-collection activity, not an educational resource or instructional-time advantage, so it does not create a resource imbalance. Criterion B is met because no additional time or budget was given to either group, and the control group received a time- and content-matched substitute activity for the review medium being tested.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this specific study is reported or referenced in the paper, and none was found through internet citation searches.
    • Detailed Analysis: The paper does not report, nor does it reference, any independent replication of this specific study (i.e., the Memrise-vs-handout comparison among Vietnamese nursing college students in a rural setting) conducted by a different research team. The literature review cites other Memrise studies (e.g., Fadhilawati, 2016; Fathi, Alipour and Saeedian, 2018; Tyas and Nurdiawati, 2019) as prior, related work rather than as replications of this particular study, and this paper positions itself as filling a research gap rather than replicating an earlier Vietnamese study. Internet Search: A Google Scholar citation search shows this paper has been cited by 14 subsequent works as of the check date. Reviewing the citing works (covering topics such as Memrise and learner motivation, Memrise for TOEFL preparation, general mobile-assisted vocabulary learning, and gamification in language teaching more broadly), none of them constitutes an independent replication of this specific study's design (Memrise vs. handout review, rural Vietnamese nursing college students). No other independent reproduction of this exact study was located. Criterion R is not met because no independent replication of this specific study is documented in the paper or found through internet search.
  • A

    All-subject Exams

    • Only vocabulary outcomes in one subject (English) were measured, and criterion E (a prerequisite) is not met.
    • "attempting to investigate the vocabulary development of college students" (Abstract)
    • Relevant Quotes: 1) "this study employed a mixed method of quantitative and qualitative design attempting to investigate the vocabulary development of college students..." (Abstract) 2) "The posttest was designed to assess students' memorisation of these 50 words..." (p. 5) Detailed Analysis: The study measures only English vocabulary outcomes; no other core subjects (e.g., mathematics, science, or other language-arts domains) are assessed. Criterion A also explicitly requires criterion E (standardised exam-based assessment) to be met as a prerequisite, and criterion E is not met here since a custom test was used. Criterion A is not met because only one subject was assessed and the exam-standardisation prerequisite fails.
  • G

    Graduation Tracking

    • The authors themselves state that only short-term outcomes were measured, with no long-term or graduation follow-up; criterion Y is also not met, which independently rules out criterion G.
    • "the study investigated the students' vocabulary performance shortly afterward the treatment. Therefore, the study only evaluated the students' memory in constant time, not for vocabulary retention." (p. 10)
    • Relevant Quotes: 1) "In addition, the study investigated the students' vocabulary performance shortly afterward the treatment. Therefore, the study only evaluated the students' memory in constant time, not for vocabulary retention." (p. 10) 2) "such as limited time for the treatment, small sample size of participants and incapability of measuring the potential of this app in students' vocabulary learning" (p. 10) Detailed Analysis: The authors explicitly acknowledge in their limitations section that outcomes were measured only shortly after treatment ended and that long-term retention was not assessed. There is no mention of any follow-up tracking of participants, let alone tracking through to graduation. Consistent with the rule that G cannot be met if Y is not met, and Y is not met here, criterion G fails on both grounds. Internet Search: A Google Scholar citation search (14 citing works as of the check date) and general web search for follow-up publications by any of the seven authors (Anh Thi Nguyen, Thu Thuy Nguyen, Thanh Thao Le, Hoang Yen Phuong, Trut Thuy Pham, Thi Anh Thu Huynh, Huong Tra Nguyen) tracking this same cohort's longer-term or graduation outcomes did not locate any such subsequent paper. Criterion G is not met because there is no follow-up beyond the immediate posttest, either in this paper or in any located subsequent publication, and criterion Y is not met.
  • P

    Pre-Registered

    • No pre-registration statement, registry link, or registration date is mentioned anywhere in the paper, and none was found through internet search.
    • Detailed Analysis: There is no mention anywhere in the Methods, Procedures, or References sections of a study protocol being registered on any public registry prior to data collection. No registration ID, registry name, or registration date is provided anywhere in the text. Internet Search: A search of the Open Science Framework (OSF) and general web search for a pre-registration of this study (by title, authors, or institution) did not locate any registered protocol. Criterion P is not met because no pre-registration evidence is present in the paper or found through internet search.