Metodología activa y digitalizada en L2: Efecto en el rendimiento académico (Active Learning and Digitalized Methodology in L2: Effect in Academic Results)

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • The authors explicitly declare a quasi-experimental design using two pre-existing intact classes, so a properly implemented class-level RCT is not demonstrated.
    • "Esta investigación utiliza un diseño cuasi-experimental debido a que la asignación de los grupos no es al azar, sino que están conformados por las aulas de clases" (p. 59)
    • Relevant Quotes: 1) "un Centro Educativo de Madrid (España) que son asignados aleatoriamente a un grupo control y un grupo experimental" (Abstract, p. 52) 2) "Todos los estudiantes fueron divididos aleatoriamente en dos grupos que conforman las aulas de clase desde primer curso de Educación Secundaria Obligatoria." (p. 58) 3) "La selección de la muestra fue intencional a través de muestreo no probabilístico por accesibilidad y la asignación del grupo control y experimental a las aulas fue aleatoria." (p. 58) 4) "Esta investigación utiliza un diseño cuasi-experimental debido a que la asignación de los grupos no es al azar, sino que están conformados por las aulas de clases, lo que amenaza la validez interna porque los grupos pueden ser diferentes antes del tratamiento." (p. 59) Detailed Analysis: The paper is internally contradictory about randomisation. The abstract claims students "son asignados aleatoriamente" (were randomly assigned) to control and experimental groups, but the design section explicitly states the study is quasi-experimental ("diseño cuasi-experimental") because group assignment "no es al azar" (is not random): the two groups are pre-existing intact classes formed when students entered secondary school years before the study. The only random element for this study is which of the two intact classrooms received the experimental condition ("la asignación del grupo control y experimental a las aulas fue aleatoria"). With only two intact clusters in a single school, no described randomisation procedure for the classes themselves, and the authors' own explicit self-designation of the design as quasi-experimental, this cannot be considered a properly implemented class-level RCT under the ERCT standard. The intervention is whole-class gamified teaching, not personal tutoring, so the tutoring exception does not apply. Because the study itself is not a properly documented RCT (see rct field), this also affects the top-level randomised controlled trial classification. Criterion C is not met because the authors explicitly describe a quasi-experimental design with two pre-existing intact classes rather than a properly implemented and clearly described class-level randomised controlled trial.
  • E

    Exam-based Assessment

    • Outcomes were measured with ad hoc, school-made trimester exams, not a widely recognised standardised test.
    • "los instrumentos utilizados para la gamificación y exámenes que están creados ad hoc" (p. 68)
    • Relevant Quotes: 1) "Exámenes de 1ª y 2ª evaluación. Estas pruebas son las mismas tanto para el grupo de control como el experimental. Se estructuran de manera idéntica, aunque con distintos contenidos." (p. 59) 2) "El grupo de control es evaluado de forma sumativa con un examen único al final del trimestre." (p. 60) 3) "siendo una de sus principales limitaciones el tamaño muestral junto con los instrumentos utilizados para la gamificación y exámenes que están creados ad hoc, por lo que cualquier generalización a otras poblaciones ha de realizarse con precaución." (p. 68) Detailed Analysis: The outcome measure is the school's own second-trimester English exam (reading comprehension, writing, use of English, lexis), designed within the school for this course. The authors themselves acknowledge in the limitations that the exams "están creados ad hoc" (are created ad hoc). Although the exam format is loosely modelled on the Spanish university entrance evaluation and aligned with CEFR B2 content, no recognised standardised external test (e.g., Cambridge, EOI, national exam) was used. Under the ERCT standard, custom or teacher-made assessments do not satisfy the requirement for a widely recognised standardised exam. Criterion E is not met because outcomes were measured with ad hoc school trimester exams rather than a recognised standardised test.
  • T

    Term Duration

    • Only about two months elapsed from intervention start (January 2020) to outcome measurement (early March 2020), which is shorter than one full academic term.
    • "La experiencia educativa se puso en marcha ... desde enero hasta finales de febrero ... la primera semana de marzo de 2020 se tuvo que realizar los exámenes de la 2ª evaluación" (p. 62)
    • Relevant Quotes: 1) "Esta investigación analiza el efecto en el rendimiento académico que tiene una metodología activa digitalizada durante dos meses en Educación Secundaria Obligatoria." (Abstract, p. 52) 2) "La experiencia educativa se puso en marcha durante el segundo trimestre del curso 2019/2020; concretamente, desde enero hasta finales de febrero, puesto que debido a la situación del COVID-19 la primera semana de marzo de 2020 se tuvo que realizar los exámenes de la 2ª evaluación por la subsiguiente suspensión de clases y el curso completo." (p. 62) 3) "Se realizaron veinte sesiones con una duración aproximada de cincuenta minutos con una periodicidad de tres sesiones a la semana." (p. 62) Detailed Analysis: The intervention started in January 2020 and ran until the end of February 2020, with the outcome exam administered in the first week of March 2020 because COVID-19 forced early testing and subsequent suspension of classes. The interval from intervention start (January) to outcome measurement (early March) is approximately two months, and the abstract itself describes the intervention as lasting "dos meses" (two months). The ERCT standard requires at least one full academic term (approximately 3-4 months) between intervention start and outcome measurement. Two months falls short of a full term. Criterion T is not met because only about two months elapsed between the start of the intervention in January 2020 and the outcome exam in the first week of March 2020.
  • D

    Documented Control Group

    • The control group's size, demographics, baseline exam scores and traditional-instruction conditions are documented in detail.
    • "el primer grupo, que es el grupo control, está conformado por 37% de chicos y 63% de chicas con una media de edad de 15,2 años" (p. 58)
    • Relevant Quotes: 1) "Cada grupo está formado por 30 estudiantes, el primer grupo, que es el grupo control, está conformado por 37% de chicos y 63% de chicas con una media de edad de 15,2 años" (p. 58) 2) "El grupo control recibe una enseñanza tradicional centrada en clase magistral" (Abstract, p. 52) 3) "En cuanto al grupo control, durante las diferentes sesiones, se realizaron las actividades proporcionadas por el libro de texto (Rogers, 2005). La profesora dirigió todas las lecciones al gran grupo." (p. 62) 4) "Los contenidos de la asignatura de Inglés del segundo trimestre son los mismos para los dos grupos (control y experimental) y la profesora que impartió la docencia fue la misma para eliminar cualquier sesgo" (p. 62) 5) "Nota 1ª evaluación 7.10 1.47" (control group baseline mean and SD, Table 1, p. 64) Detailed Analysis: The control group is described in reasonable detail: its size (n=30), gender composition (37% boys, 63% girls), mean age (15.2 years), inclusion criteria, and its condition (traditional teacher-centred instruction using the textbook Prepare 7, same content and same teacher as the experimental group, no graded activities during the term). Baseline academic performance is documented via the first-evaluation exam scores in Table 1 (M=7.10, SD=1.47), and an inferential test confirmed no significant baseline difference between groups (p=0.064). This documentation allows a proper comparison between the groups. Criterion D is met because the control group's size, demographics, baseline performance and instructional conditions are clearly documented.

• Level 2 Criteria

  • S

    School-level RCT

    • The study involved only two classes within one single school, so there was no school-level randomisation.
    • "El tamaño muestral de esta investigación es de 60 estudiantes ... de un centro educativo concertado-privado católico de línea 2 de Madrid Capital (España)." (p. 58)
    • Relevant Quotes: 1) "El tamaño muestral de esta investigación es de 60 estudiantes de cuarto de Educación Secundaria Obligatoria de un centro educativo concertado-privado católico de línea 2 de Madrid Capital (España)." (p. 58) 2) "la asignación del grupo control y experimental a las aulas fue aleatoria." (p. 58) Detailed Analysis: The study took place in a single school in Madrid, with two intact classes of the same grade assigned to the two conditions. There is no randomisation of multiple schools or institutional units; the whole experiment is contained within one school. School-level randomisation requires several schools (or comparable implementing units) to be randomly allocated between conditions, which is not the case here. Criterion S is not met because the study was conducted within a single school with no school-level randomisation.
  • I

    Independent Conduct

    • The authors designed the intervention, created the instruments and analysed the data themselves, with no independent third-party evaluation.
    • "las autoras prepararon todos los materiales e instrumentos para la enseñanza activa y digitalizada que tendría el grupo experimental" (p. 62)
    • Relevant Quotes: 1) "Durante el mes de diciembre de 2019, las autoras prepararon todos los materiales e instrumentos para la enseñanza activa y digitalizada que tendría el grupo experimental y que pondría en práctica una profesora del centro." (p. 62) 2) "En primer lugar, el Departamento de Inglés aceptó la propuesta de aplicación de una metodología activa y digitalizada para los estudiantes de cuarto curso" (p. 61) 3) "El análisis de datos se realiza con el programa informático IBM® SPSS® Statistics 25." (p. 63) Detailed Analysis: The two authors designed the intervention, prepared all the materials and instruments, and conducted the data analysis themselves. Although the day-to-day teaching was delivered by a teacher of the school, that teacher acted as the implementer of the authors' intervention, not as an independent evaluator, and no external evaluation team, third-party data collection, or independent oversight of the analysis is mentioned anywhere in the paper. Under the ERCT standard, the same team designing the intervention and conducting the evaluation without documented independent oversight fails this criterion. Criterion I is not met because the intervention designers (the authors) also prepared all instruments and analysed the data, with no independent evaluation team involved.
  • Y

    Year Duration

    • The study lasted only about two months, far below the required 75% of an academic year.
    • "La experiencia educativa se puso en marcha durante el segundo trimestre del curso 2019/2020; concretamente, desde enero hasta finales de febrero" (p. 62)
    • Relevant Quotes: 1) "Esta investigación analiza el efecto en el rendimiento académico que tiene una metodología activa digitalizada durante dos meses en Educación Secundaria Obligatoria." (Abstract, p. 52) 2) "La experiencia educativa se puso en marcha durante el segundo trimestre del curso 2019/2020; concretamente, desde enero hasta finales de febrero" (p. 62) 3) "sería recomendable ampliar el tamaño muestral, extender la duración de la aplicación de gamificación de todo el curso escolar" (p. 68) Detailed Analysis: The intervention and outcome tracking spanned roughly two months (January to early March 2020), which is far below 75% of an academic year (approximately 9-10 months). The authors themselves recommend extending the gamification to the whole school year in future research, confirming the current study did not cover a year. Additionally, per the ranking instructions, criterion T (Term Duration) is not met, so criterion Y cannot be met. Criterion Y is not met because the study covered only about two months, far short of 75% of an academic year, and criterion T is already not met.
  • B

    Balanced Control Group

    • Both groups had identical class time, teacher and content; the gamified digital activities were integral to the tested methodology, so resources were balanced.
    • "Los contenidos de la asignatura de Inglés del segundo trimestre son los mismos para los dos grupos (control y experimental) y la profesora que impartió la docencia fue la misma" (p. 62)
    • Relevant Quotes: 1) "Se realizaron veinte sesiones con una duración aproximada de cincuenta minutos con una periodicidad de tres sesiones a la semana." (p. 62) 2) "En cuanto al grupo control, durante las diferentes sesiones, se realizaron las actividades proporcionadas por el libro de texto (Rogers, 2005)." (p. 62) 3) "Los contenidos de la asignatura de Inglés del segundo trimestre son los mismos para los dos grupos (control y experimental) y la profesora que impartió la docencia fue la misma para eliminar cualquier sesgo en los resultados debido a las características del profesor." (p. 62) 4) "Las horas en las que tienen las clases de Inglés ambos grupos son por la mañana durante la segunda hora, unos días el grupo control y otros días el grupo experimental, para intentar evitar el sesgo del momento del día en el que reciben clase." (p. 62) 5) "La enseñanza fue impartida en las aulas habituales de los estudiantes con óptimas condiciones de iluminación y sonoridad." (p. 63) Detailed Analysis: Applying the criterion B decision tree: the gamified intervention was delivered inside the regular English class sessions (twenty 50-minute sessions in the same trimester), with the same teacher, the same curricular content, and the same morning time slot for both groups (alternating between groups to avoid time-of-day bias). The control group spent the equivalent class time on textbook activities directed by the same teacher. No extra instructional hours, staff, or budget beyond the regular class time were added for the experimental group; what differed was the teaching methodology itself (gamified, cooperative, digitalized activities vs. traditional lectures and textbook exercises), which is precisely the treatment variable under test. The digital tools (Kahoot, Quizizz, Socrative, LearningApp, GoAnimate, QR codes) and the formative grading of activities are integral components of the active digitalized methodology being tested rather than separable add-on resources; the authors note some apps could also be replayed outside class, but this self-directed practice is inherent to the gamified methodology itself and is not a resource withheld from the control group's business-as-usual instruction. Because instructional time, teacher, content and setting were matched, and the methodological differences are integral to the intervention contrast being tested, the groups are considered balanced. Criterion B is met because both groups received the same amount of class time, the same teacher and the same content, with the digital gamified activities being integral to the methodology being tested rather than an unmatched extra resource.

• Level 3 Criteria

  • R

    Reproduced

    • The study is presented as a first exploratory work, and a citation-database search found only one unrelated citing paper and no independent replication.
    • "Este estudio es el primero que relaciona las actividades realizadas en la enseñanza gamificada con el rendimiento académico" (p. 69)
    • Relevant Quotes: 1) "Este estudio es el primero que relaciona las actividades realizadas en la enseñanza gamificada con el rendimiento académico para analizar qué ejercicios son los que mejor disponen para el examen." (p. 69) 2) "Esta investigación se realiza con un carácter exploratorio" (p. 68) Detailed Analysis: The authors describe the study as exploratory and as the first to relate specific gamified activities to exam performance, so no replication existed at publication. The paper cites other gamification studies in EFL (e.g., Purgina et al., 2019; Mogrovejo et al., 2019; Lam et al., 2018), but these are different interventions in different designs and contexts, not replications of this specific two-month gamified programme with these materials and outcomes. Internet search update: Crossref and Semantic Scholar records for DOI 10.4067/S0718-09342024000100052 show a citation count of one. The single citing work is "Emerging Technological Tools and Post-Pandemic Academic Performance in College Students" (Diaz Hinostroza, Cerna Gonzales, Aranda Dextre, Borja Torres, Blas Montenegro & Manco Chavez, 2024, DOI 10.62754/joe.v3i8.4937). This citing paper studies college students and broader post-pandemic technology tools, not a replication of this specific two-month secondary-school gamification intervention by an independent team using the same design. No independent, peer-reviewed replication of this particular study by a different research team could be identified. Criterion R is not met because no independent replication of this specific study has been published; the authors present it as a first exploratory study, and citation-database searches confirm only a single unrelated citing work with no replication.
  • A

    All-subject Exams

    • Only performance in English was measured with ad hoc exams; no other core subjects were assessed and criterion E is not met.
    • "La variable dependiente es el rendimiento académico en la asignatura de Inglés" (p. 59)
    • Relevant Quotes: 1) "La variable dependiente es el rendimiento académico en la asignatura de Inglés" (p. 59) 2) "En cuanto a las competencias lingüísticas, se trabajaron aquellas establecidas para el nivel B2 según el Marco Común Europeo de Referencia para las Lenguas en la primera lengua extranjera." (p. 62) Detailed Analysis: Outcomes were measured only in the subject of English as a foreign language; no other core subjects (mathematics, Spanish language, science, social studies, etc.) of the Spanish fourth-year ESO curriculum were assessed, and no justification for a specialised exception is provided (this is general compulsory secondary education, not vocational training). Furthermore, per the ranking instructions, criterion A requires criterion E to be met, and criterion E is not met because the exams were created ad hoc. Criterion A is not met because only English was assessed, with ad hoc exams, and criterion E is already not met.
  • G

    Graduation Tracking

    • Tracking ended with the March 2020 trimester exam, no graduation follow-up was found in the paper or in the authors' later publication record, and criterion Y is not met.
    • "la primera semana de marzo de 2020 se tuvo que realizar los exámenes de la 2ª evaluación por la subsiguiente suspensión de clases y el curso completo." (p. 62)
    • Relevant Quotes: 1) "la primera semana de marzo de 2020 se tuvo que realizar los exámenes de la 2ª evaluación por la subsiguiente suspensión de clases y el curso completo." (p. 62) 2) "En futuras investigaciones sería recomendable ampliar el tamaño muestral, extender la duración de la aplicación de gamificación de todo el curso escolar y recoger varias pruebas de rendimiento durante el curso" (p. 68) Detailed Analysis: Measurement ended with the second-evaluation exam in early March 2020, after which classes were suspended due to COVID-19. The students were in the fourth (final) year of compulsory secondary education, but no tracking to the end of the school year, to ESO graduation, or beyond is reported, and the authors' suggestions for future research confirm that longer tracking was not performed. Internet search update: An author-level search of Semantic Scholar's publication records for Beatriz Chaves-Yuste (a frequent, actively publishing co-author) lists numerous later papers (2021-2026) on EFL/ESP gamification, digital resources, literature integration, dictionary writing systems and related topics, but none of these titles or abstracts describe continued tracking of the original 60-student ESO cohort from this study through graduation. No follow-up publication tracking this specific cohort was identified. Additionally, per the ranking instructions, criterion G cannot be met because criterion Y is not met. Criterion G is not met because measurement stopped at the trimester exam in March 2020 with no tracking of students until graduation, no follow-up publication on this cohort was found via a search of the authors' later publication record, and criterion Y is already not met.
  • P

    Pre-Registered

    • The paper contains no reference to any pre-registered protocol or registry entry, and none was found online either.
    • Relevant Quotes: 1) "En este mes se solicitó permiso a la Dirección del Centro Educativo y se detallaron las condiciones éticas siguiendo la Declaración de Helsinki." (p. 62) Detailed Analysis: The paper mentions only school permissions, informed consent from families and adherence to the Declaration of Helsinki's ethical conditions. There is no mention of any registration of the study protocol on a public registry (e.g., OSF, ClinicalTrials.gov, AEA registry), no registration ID, and no date establishing that hypotheses and analysis plans were registered before data collection began in December 2019. Internet search update: A search of the paper's indexed metadata (Crossref, Semantic Scholar) found no pre-registration identifiers, registry links, or DOIs pointing to a registered protocol associated with this study, corroborating the absence of any pre-registration statement in the text itself. Criterion P is not met because no pre-registration of the study protocol is mentioned anywhere in the paper, and no registry record for this study was found online.