The Impact of Text Messaging as an Instructional Tool to Enhance Learner Autonomy and Perception

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Randomisation was done at the individual student level, not at the class or school level, and the intervention is not one-to-one tutoring, so the exception does not apply.
    • "After the OPT, a number of seventy-four students ... were selected as the study participants who randomly assigned into two groups, i.e., an experimental group and a control one, each including 37 research informants." (p. 189)
    • Relevant Quotes: 1) "Participants are randomly assigned in experimental and control groups, including 37 participants each." (p. 184, Abstract) 2) "After the OPT, a number of seventy-four students (48 females and 26 males) within the age range of 21 to 26 years old who scored between 28 and 36 points (pre-intermediate level) were selected as the study participants who randomly assigned into two groups, i.e., an experimental group and a control one, each including 37 research informants." (p. 189) 3) "The same teacher took the responsibility of teaching both groups." (p. 191) Detailed Analysis: The ERCT C criterion requires randomisation at the class level or stronger (school level), or a valid exception for one-to-one tutoring/personal teaching interventions. The quotes show that individual students drawn from a common pool of General English learners at one university were randomly assigned to the experimental and control groups; the unit of randomisation is the individual student, not intact classes. The intervention (SMS text messages sent to students plus classroom teaching by the same teacher for both groups) is group classroom instruction supplemented by broadcast text messages, not personal one-to-one tutoring, so the tutoring exception does not apply. Student-level randomisation with both groups taught by the same teacher also raises the exact contamination risk the criterion is designed to prevent. Criterion C is not met because randomisation was carried out at the individual student level rather than at the class or school level, and no tutoring exception applies.
  • E

    Exam-based Assessment

    • Outcomes were measured with a self-report autonomy questionnaire and a researcher-made interview, not with any standardised exam-based assessment.
    • "As far as participants' autonomy is concerned, the 21-item questionnaire designed by Zhang and Li (2004) is used to measure this variable." (p. 190)
    • Relevant Quotes: 1) "As far as participants' autonomy is concerned, the 21-item questionnaire designed by Zhang and Li (2004) is used to measure this variable." (p. 190) 2) "The autonomy questionnaire is administered twice, as the pretest and posttest, to determine the differences between learners' autonomy towards implementing MALL as an instructional tool before and after the treatment." (p. 190) 3) "To begin with, an Oxford Placement Test (OPT) was conducted before the treatment to measure the group's homogeneity and if learners had a similar proficiency English level." (p. 190) 4) "The third instrument is a semi-structured interview prepared and administered to the experimental group by the researchers, including seven open-ended questions..." (p. 191) Detailed Analysis: Criterion E requires that the study outcomes be measured with standardised, widely recognised exams rather than researcher-selected questionnaires or custom instruments. In this study the only standardised test used, the Oxford Placement Test, served solely as a screening tool to ensure homogeneity of proficiency before the treatment; it was not used as an outcome measure. The primary outcome, learner autonomy, was measured with the 21-item self-report questionnaire of Zhang and Li (2004), which is a research questionnaire measuring attitudes and reported behaviours, not a standardised academic examination. The secondary outcome (perception) was measured with a researcher-made semi-structured interview. No standardised exam-based assessment of learning outcomes was administered after the intervention. Criterion E is not met because outcomes were measured with a self-report autonomy questionnaire and a researcher-made interview rather than a standardised exam.
  • T

    Term Duration

    • The interval from intervention start to outcome measurement was only six weeks, which is far shorter than the required full academic term.
    • "...the study is carried out over an 18-session treatment during six successive weeks (the participants in the experimental group would receive SMSs three times a week)." (p. 191)
    • Relevant Quotes: 1) "During the reading course in the fall semester of the academic year 2019-2020, the study is carried out over an 18-session treatment during six successive weeks (the participants in the experimental group would receive SMSs three times a week)." (p. 191) 2) "After finishing the treatment process, the learner autonomy questionnaire is administered for the second time as a posttest to check SMS text messaging effectiveness as the treatment procedure." (p. 191) 3) "After the treatment procedure, the experimental group participants who were exposed to SMS text messaging over a period of six weeks took part in a face-to-face interview." (p. 191) Detailed Analysis: Criterion T requires that outcomes be measured at least one full academic term (approximately 3-4 months) after the intervention begins. Here the treatment lasted six successive weeks (18 sessions), and the posttest questionnaire was administered immediately "after finishing the treatment process," i.e., about six weeks after the intervention start. Six weeks is roughly half of a typical academic term and well short of the 3-4 month minimum. No delayed follow-up measurement extending the tracking window to a full term is reported. Criterion T is not met because outcomes were measured about six weeks after the intervention began, which is shorter than one full academic term.
  • D

    Documented Control Group

    • The control group's size (37), demographics, baseline OPT proficiency, pretest autonomy scores, and traditional face-to-face condition are all clearly documented.
    • "Participants from the experimental group use text messaging (the treatment) to receive instructions, whereas those from the control group receive traditional classroom instructions in a face-to-face channel." (p. 184)
    • Relevant Quotes: 1) "After the OPT, a number of seventy-four students (48 females and 26 males) within the age range of 21 to 26 years old who scored between 28 and 36 points (pre-intermediate level) were selected as the study participants who randomly assigned into two groups, i.e., an experimental group and a control one, each including 37 research informants." (p. 189) 2) "They were all Persian native speakers and all of them had a mobile phone to use for the study." (p. 190) 3) "Participants from the experimental group use text messaging (the treatment) to receive instructions, whereas those from the control group receive traditional classroom instructions in a face-to-face channel." (p. 184, Abstract) 4) "The same teacher took the responsibility of teaching both groups." (p. 191) 5) "Table 2 reveals that the mean and the standard deviation of the homogenized participants are 31.66 and 2.22, respectively... participants' scores show that they have a homogenous general English proficiency level in the experimental and control groups." (p. 192) 6) "Table 3 shows the descriptive statistics for the autonomy scores of both groups. Both groups' means related to the autonomy scores are 71.05 and 73.94." (p. 192) Detailed Analysis: Criterion D requires the control group to be documented in terms of size, characteristics, baseline performance, and the condition it received. The paper documents the control group's size (37 students), the shared demographic profile (undergraduates aged 21-26, Persian native speakers, mixed majors, pre-intermediate OPT scores of 28-36 establishing baseline proficiency homogeneity), and what the control group received (traditional face-to-face classroom instruction from the same teacher, with no SMS treatment). Baseline autonomy was measured for both groups via the pretest questionnaire and used as a covariate in the ANCOVA, and control-group autonomy statistics are reported in Table 3. Although the paper does not break down gender per group, the overall documentation of who the control group is, its baseline performance, and its condition is comparable to the level accepted for this criterion. Criterion D is met because the control group's size, baseline proficiency, demographics, and business-as-usual face-to-face condition are clearly documented.

• Level 2 Criteria

  • S

    School-level RCT

    • Randomisation occurred at the individual student level within a single university branch, so there was no school-level assignment.
    • "...seventy-four students ... were selected as the study participants who randomly assigned into two groups, i.e., an experimental group and a control one..." (p. 189)
    • Relevant Quotes: 1) "A number of eighty-eight undergraduate university students with different majors ... studying General English at the Islamic Azad University-South Tehran Branch, Iran voluntarily have participated in this study." (p. 189) 2) "...seventy-four students ... were selected as the study participants who randomly assigned into two groups, i.e., an experimental group and a control one, each including 37 research informants." (p. 189) Detailed Analysis: Criterion S requires randomisation at the level of whole schools or equivalent institutional units. This study was conducted at a single institution (Islamic Azad University-South Tehran Branch), and randomisation occurred at the individual student level within that one site. No multiple schools, campuses, or sites were involved, and no institution-level assignment took place. Criterion S is not met because the study randomised individual students within a single university branch rather than randomising schools or institutional units.
  • I

    Independent Conduct

    • The same researchers designed, delivered, and evaluated the intervention themselves, with no external or independent evaluation team involved.
    • "The researcher used text messaging to improve learners' autonomy and determine their perceptions toward MALL in the language learning context." (p. 191)
    • Relevant Quotes: 1) "The researcher administered the OPT initially to ensure the participants' homogeneity prior to the treatment's commencement." (p. 191) 2) "The researcher used text messaging to improve learners' autonomy and determine their perceptions toward MALL in the language learning context." (p. 191) 3) "The third instrument is a semi-structured interview prepared and administered to the experimental group by the researchers..." (p. 191) 4) "As a final step, the teacher interviewed the learners in the experimental group to investigate interviewees' perceptions of MALL." (p. 191) Detailed Analysis: Criterion I requires the study to be conducted independently of those who designed the intervention, or at least to document third-party oversight of data collection and analysis. In this paper, the researchers themselves designed the SMS-based treatment, administered the placement test, delivered the treatment, prepared and administered the questionnaires and interviews, and analysed the data. There is no mention of an external evaluation team, independent test administrators, blinded assessors, or any third-party oversight anywhere in the paper. Criterion I is not met because the same researchers designed, implemented, and evaluated the intervention with no documented independent conduct or oversight.
  • Y

    Year Duration

    • The six-week study duration is far shorter than 75% of an academic year, and the prerequisite term-duration criterion is also unmet.
    • "...the study is carried out over an 18-session treatment during six successive weeks..." (p. 191)
    • Relevant Quotes: 1) "During the reading course in the fall semester of the academic year 2019-2020, the study is carried out over an 18-session treatment during six successive weeks..." (p. 191) 2) "After finishing the treatment process, the learner autonomy questionnaire is administered for the second time as a posttest..." (p. 191) Detailed Analysis: Criterion Y requires outcome tracking covering at least 75% of a full academic year (roughly 9-10 months) from the intervention start. This study ran for only six weeks within a single fall semester, with the posttest given immediately after the treatment ended. Six weeks is a small fraction of an academic year. Additionally, per the ranking rules, because criterion T (Term Duration) is not met, criterion Y cannot be met. Criterion Y is not met because tracking lasted only six weeks, far below 75% of an academic year, and criterion T is also unmet.
  • B

    Balanced Control Group

    • The thrice-weekly SMS messages are the explicit treatment variable integral to the tested intervention, and both groups shared the same course and teacher, so the design is acceptably balanced.
    • "The same teacher took the responsibility of teaching both groups." (p. 191)
    • Relevant Quotes: 1) "Participants from the experimental group use text messaging (the treatment) to receive instructions, whereas those from the control group receive traditional classroom instructions in a face-to-face channel." (p. 184, Abstract) 2) "During the reading course in the fall semester of the academic year 2019-2020, the study is carried out over an 18-session treatment during six successive weeks (the participants in the experimental group would receive SMSs three times a week)." (p. 191) 3) "The same teacher took the responsibility of teaching both groups." (p. 191) 4) "The researcher used text messaging to improve learners' autonomy and determine their perceptions toward MALL in the language learning context." (p. 191) Detailed Analysis: Following the criterion B decision tree: first, does the intervention add extra time or budget? Both groups attended the same reading course taught by the same teacher, which keeps core instructional time and staffing balanced. The experimental group additionally received SMS text messages three times per week as the delivery channel for instructional content, while the control group received the equivalent instruction through the traditional face-to-face channel. The added resource (SMS messages on students' own phones) is minimal in cost and time, and, more importantly, it is the explicit treatment variable: the stated aim of the study is to test "the efficiency of text messaging as an ... instructional tool," so the text-messaging channel is integral to the intervention being tested against business-as-usual traditional instruction. Under the decision tree, when the extra resource is itself the treatment being tested (RESOURCES_ARE_TREATMENT), the control group may remain business-as-usual and the criterion is met. It should be clearly noted that the experimental group did receive an additional input (thrice-weekly instructional SMS messages) that the control group did not, but this input is the defined intervention itself rather than a separable, confounding add-on, and core classroom teaching was identical for both groups (same teacher, same course). Criterion B is met because the additional SMS instruction is the explicit treatment variable integral to the tested intervention, while core teaching time and teacher were the same for both groups.

• Level 3 Criteria

  • R

    Reproduced

    • No independent replication of this specific study exists; a citation search confirms the cited similar studies and citing papers are earlier or unrelated works, not replications of this trial.
    • Relevant Quotes: 1) "The results are consistent with a study carried out by Nasr and Abbas (2018), which examined the role of MALL in improving learner autonomy in the EFL reading context among students of Najran University in Saudi Arabia." (p. 195) 2) "Moreover, findings on autonomy agree with those reported by Farangi, Kamyab, Izanlu, and Ghodrat (2017), which examined the impact of SMS on Iranian upper-intermediate EFL students' grammar learning." (p. 196) Detailed Analysis: Criterion R requires that this specific study be independently replicated by a different research team in a different context, published in a peer-reviewed journal. The related studies cited in the discussion (Nasr & Abbas 2018; Hazaea & Alzubi 2018; Farangi et al. 2017; Leis et al. 2015) all predate this 2020 paper and investigated broadly similar MALL/SMS themes; they are prior thematically related work, not replications of this study's specific design, instrument, and population. A citation search (Semantic Scholar, DOI 10.26803/ijlter.19.11.11) identified eight papers citing this study: Nguyen and Nguyen (2024) on self-directed learning readiness and MALL; Ramaoka, Kekana, and Montle (2024) on textism exposure and spelling; Budiyono (2023) on a "Brilliant Solution" teaching method; Zhen and Hashim (2022), a systematic review of MALL and speaking readiness; Che Mustaffa and Sailin (2022), a systematic review of MALL research in Malaysia; Aprianti and Winarto (2021) on e-portfolios and writing autonomy; Puspitasari et al. (2024) on pre-service teachers' MALL perceptions; and Behforouz and Frumuselu (2021), "The Effect of Text Messaging on EFL Learners' Lexical Depth and Breadth," a same-author follow-up on vocabulary outcomes. None of these is an independent attempt by a different research team to reproduce this study's specific design, population, and outcome measures; the citing works are unrelated MALL studies or literature reviews, and the one same-cohort follow-up paper is by the same authors, so it cannot count as independent replication. No independent replication of this specific text-messaging autonomy trial at Islamic Azad University-South Tehran Branch was identified. Criterion R is not met because no independent replication of this specific study by a different team has been identified; cited similar studies are prior related work, not replications, and a citation search found no independent replication either.
  • A

    All-subject Exams

    • Criterion E is unmet and only an EFL autonomy questionnaire was measured, so no all-subject standardised exam assessment exists.
    • "The main focus of the current study is on the vocabulary learning, SMS, and learner's autonomy..." (p. 198)
    • Relevant Quotes: 1) "As far as participants' autonomy is concerned, the 21-item questionnaire designed by Zhang and Li (2004) is used to measure this variable." (p. 190) 2) "The main focus of the current study is on the vocabulary learning, SMS, and learner's autonomy, and therefore, further studies can be done on the role of SMS in other learning skills." (p. 198) Detailed Analysis: Criterion A requires standardised exam-based assessment across all main subjects, and criterion E is an explicit prerequisite. Criterion E is not met here, since the sole outcome measures were a self-report autonomy questionnaire and a researcher-made interview; therefore criterion A automatically fails. Moreover, the study measured only learner autonomy and perception within EFL - no academic achievement in English or any other subject was assessed, and no subject-area exams of any kind were administered as outcomes. Criterion A is not met because the prerequisite criterion E fails and no subject exams, let alone all-subject exams, were used as outcome measures.
  • G

    Graduation Tracking

    • Measurement ended immediately after the six-week treatment, with no follow-up tracking of participants until graduation; the one same-author follow-up paper found addresses vocabulary outcomes, not graduation tracking.
    • "After finishing the treatment process, the learner autonomy questionnaire is administered for the second time as a posttest..." (p. 191)
    • Relevant Quotes: 1) "After finishing the treatment process, the learner autonomy questionnaire is administered for the second time as a posttest to check SMS text messaging effectiveness as the treatment procedure." (p. 191) 2) "As a final step, the teacher interviewed the learners in the experimental group to investigate interviewees' perceptions of MALL." (p. 191) Detailed Analysis: Criterion G requires tracking participants until graduation from their educational stage, and per the ranking rules it cannot be met when criterion Y is unmet, which is the case here. Data collection ended with the posttest and interviews immediately after the six-week treatment. There is no mention of any longer-term follow-up, no plan to track the undergraduate participants to degree completion, and no follow-up publications on this cohort are referenced in the paper itself. A citation and author search found one later paper by the same authors, Behforouz and Frumuselu (2021), "The Effect of Text Messaging on EFL Learners' Lexical Depth and Breadth," also involving 37 EFL learners. Based on the available abstract, this follow-up study examines vocabulary breadth and depth outcomes from SMS-based instruction; it does not describe tracking the original autonomy-study cohort through to graduation, so it does not satisfy the graduation-tracking requirement. No other follow-up publications tracking this cohort to graduation were found. Criterion G is not met because measurement stopped immediately after the six-week treatment with no tracking of participants to graduation, prerequisite criterion Y is unmet, and the one identified follow-up paper by the same authors addresses vocabulary outcomes rather than graduation tracking.
  • P

    Pre-Registered

    • The paper contains no mention of a pre-registered protocol, registry platform, or registration date, and an independent search found no registration record.
    • Relevant Quotes: 1) "This section is meant to present the method used to design the study and how data collection has taken place. In hope to meet the objectives of the present study, an experimental design with the total procedure of sampling, instrumentation, data collection, and data analysis are explained in the following sub-sections." (p. 189) 2) No statement referencing any trial registry, pre-registration platform, registration ID, or protocol publication appears anywhere in the paper. Detailed Analysis: Criterion P requires the full study protocol to be publicly pre-registered before data collection began, with a verifiable registry reference and timing. The paper contains no mention of pre-registration, no registry name or ID (such as ClinicalTrials.gov, OSF, or a national registry), and no published protocol. An independent search for a pre-registration record under the authors' names and the study title found no matching registration entry. This is consistent with the paper's own text, and with this journal (International Journal of Learning, Teaching and Educational Research) not requiring pre-registration for submission. Criterion P is not met because the paper contains no reference to any pre-registered protocol or registry entry, and no such registration could be located independently.