The Efficacy of DreamBuilder-Mediated TBLT in Enhancing English Proficiency: An Experimental Study on Women Entrepreneurs

ERCT Criteria Breakdown

• Level 1 Criteria

  • C

    Class-level RCT

    • Participants were not randomised at all; they were assigned to the TBLT or control condition based on their pre-existing enrollment cohort in an explicitly quasi-experimental design.
    • "The researcher chose a quasi-experimental design because random assignment to groups was not possible due to the program-based structure of the participant pool." (p. 110)
    • Relevant Quotes: 1) "This study employed a mixed-methods research methodology, combining a quasi-experimental quantitative method with qualitative data collection, to investigate the effects of Task-Based Language Teaching (TBLT) on the English ability of Indonesian women entrepreneurs." (p. 109-110) 2) "The researcher chose a quasi-experimental design because random assignment to groups was not possible due to the program-based structure of the participant pool. Instead, participants were assigned to groups based on their enrollment cohort." (p. 110) 3) "A purposeful sample of 40 participants was chosen based on active program enrollment and their consent to participate. These participants were set into two groups: the experimental group (n = 20), which received TBLT-based instruction through the DreamBuilder platform, and the control group (n = 20), which got traditional, form-focused instruction." (p. 110) Detailed Analysis: The ERCT Standard requires that an RCT use random assignment, at minimum at the class level, to allocate participants to intervention and control conditions. This paper explicitly and repeatedly states that it used a quasi-experimental design and that "random assignment to groups was not possible." Participants were instead sorted into groups according to their pre-existing "enrollment cohort," which is a form of convenience/cohort assignment, not randomisation. There is no description anywhere in the Methods section of a random number generator, lottery, or any other randomisation procedure being used at the student, class, or school level. The intervention is also not a one-to-one tutoring exception case (it is delivered to pre-formed groups of 20), so the tutoring exception for student-level assignment does not apply. Because there was no randomisation of any kind at any unit, criterion C is not met, and this finding alone means the paper does not qualify as an RCT under the ERCT Standard.
  • E

    Exam-based Assessment

    • The English proficiency test was designed by the research team specifically for this study rather than being a widely recognised, validated standardised exam.
    • "the proficiency test was designed by the research team rather than using a validated standard instrument" (p. 119)
    • Relevant Quotes: 1) "To get started, participants were given an English proficiency test as a pre-test and post-test to track changes in their language skills. This test assessed productive and receptive skills needed for business communication, including reading comprehension, writing, and oral tasks." (p. 110) 2) "Third, the proficiency test was designed by the research team rather than using a validated standard instrument; future research should employ validated assessments such as those based on the Bachman & Palmer (2010) assessment framework to enhance comparability." (p. 119, Limitations) 3) "The proficiency instrument was additionally researcher-designed rather than a validated standard measure." (p. 116) Detailed Analysis: The ERCT Standard requires that outcome measures be standard, widely recognised assessments rather than instruments custom-built for the study. The authors themselves explicitly acknowledge, both in the Discussion and in the Limitations section, that the proficiency test used as the primary outcome measure was "designed by the research team" rather than being a validated, standardised instrument. They further recommend that future studies use a recognised framework (Bachman & Palmer, 2010) instead, which confirms that no such standardised instrument was used here. Because the primary outcome measure was a researcher-designed test and not a recognised standardised exam, criterion E is not met.
  • T

    Term Duration

    • The intervention and outcome measurement together spanned only an eight-week period, far short of a full academic term, with no delayed follow-up testing.
    • "Pre-test and post-test scores were collected from both groups before and after the eight-week intervention in order to determine to what extent TBLT enhanced the participants' English proficiency." (p. 111)
    • Relevant Quotes: 1) "Pre-test and post-test scores were collected from both groups before and after the eight-week intervention in order to determine to what extent TBLT enhanced the participants' English proficiency." (p. 111) 2) "In the implementation stage, the experimental group worked on these tasks during the designated teaching period, while the control group received teacher-led, grammar-focused lessons on similar content. In the data collection stage, pre- and post-tests were given to both groups..." (p. 110) 3) "Fifth, this study did not measure skill retention on a delayed post-test; whether these gains can be sustained over time in this context remains unknown." (p. 119) Detailed Analysis: The ERCT Standard requires that outcomes be measured at least one academic term (roughly 3-4 months) after the intervention begins. Here, the entire intervention and the post-test measurement occurred within an eight-week window, which is shorter than a typical academic term. The authors themselves flag as a limitation that no delayed post-test was administered, meaning there is no evidence of any outcome tracking beyond the immediate end of the eight-week program. Because the interval from intervention start to measurement was only eight weeks with no further tracking, criterion T is not met.
  • D

    Documented Control Group

    • The control group's size, composition, baseline proficiency scores, and the specific instruction it received are clearly described and quantified.
    • "Both groups were matched based on pre-test English proficiency scores to ensure they were similar at the start (experimental group M = 57.8, SD = 2.6; control group M = 58.0, SD = 2.6)." (p. 110)
    • Relevant Quotes: 1) "the control group (n = 20), which got traditional, form-focused instruction." (p. 110) 2) "Both groups were matched based on pre-test English proficiency scores to ensure they were similar at the start (experimental group M = 57.8, SD = 2.6; control group M = 58.0, SD = 2.6)." (p. 110) 3) "In the implementation stage, the experimental group worked on these tasks during the designated teaching period, while the control group received teacher-led, grammar-focused lessons on similar content." (p. 110) 4) "Traditional (Control) 20 58.0 (2.6) 67.8 (2.9) +9.8 9.8" (Table 1, p. 111) Detailed Analysis: The ERCT Standard requires that the control group be documented in enough detail (size, baseline characteristics, and what it received) to allow proper comparison with the intervention group. Here, the paper states the control group's exact size (n = 20), its baseline mean and SD on the proficiency pre-test, its post-test mean and SD, and a description of what instruction it received ("teacher-led, grammar-focused lessons on similar content" during "the designated teaching period"). Both groups are also described as being drawn from the same AWE program cohort and matched on pre-test scores, allowing a reader to assess comparability at baseline. Because the control group's characteristics, sample size, baseline scores, and treatment are clearly documented, criterion D is met.

• Level 2 Criteria

  • S

    School-level RCT

    • No school-level (or any-level) randomisation occurred; participants were assigned to groups by pre-existing enrollment cohort in a quasi-experimental design.
    • "The researcher chose a quasi-experimental design because random assignment to groups was not possible due to the program-based structure of the participant pool." (p. 110)
    • Relevant Quotes: 1) "The researcher chose a quasi-experimental design because random assignment to groups was not possible due to the program-based structure of the participant pool. Instead, participants were assigned to groups based on their enrollment cohort." (p. 110) 2) "The target population included Indonesian women entrepreneurs enrolled in the Academy for Women Entrepreneurs (AWE) program in the Makassar region. A purposeful sample of 40 participants was chosen based on active program enrollment and their consent to participate." (p. 110) Detailed Analysis: Criterion S requires randomisation at the school or site/institution level, which is a stronger requirement than class-level randomisation. Since this study used no randomisation whatsoever, and instead relied on pre-existing cohort membership within a single AWE program site in Makassar, it clearly does not meet the weaker class-level requirement, let alone the school-level requirement. Because there was no site- or school-level randomisation of any kind, criterion S is not met.
  • I

    Independent Conduct

    • There is no statement in the paper indicating that the study was conducted by an independent, third-party evaluation team separate from the researchers who designed and ran the intervention.
    • "In the implementation stage, the experimental group worked on these tasks during the designated teaching period, while the control group received teacher-led, grammar-focused lessons on similar content." (p. 110)
    • Relevant Quotes: 1) "In the preparation stage, TBLT task sequences were designed and connected to real-world entrepreneurial communication activities on the DreamBuilder platform." (p. 110) 2) "In the implementation stage, the experimental group worked on these tasks during the designated teaching period, while the control group received teacher-led, grammar-focused lessons on similar content. In the data collection stage, pre- and post-tests were given to both groups, and questionnaires and interviews were conducted with participants in the experimental group..." (p. 110) 3) "The authors express their sincere gratitude to the English Education Department of the Faculty of Teacher Training and Education at Universitas Sembilanbelas November (USN) Kolaka and Universitas Tomakaka for their institutional support throughout the conduct of this research." (p. 120, Acknowledgments) Detailed Analysis: Criterion I requires clear evidence that the trial was conducted independently from the people who designed the intervention, e.g. by an external evaluation team unaware of the intervention design. Here, the paper describes the same research team as designing the TBLT task sequences, implementing the intervention, delivering the control condition, collecting pre/post-test data, and conducting the interviews and thematic analysis. The Acknowledgments section thanks the researchers' own institution for "institutional support" but does not mention any independent evaluator, external data collection agency, or blinded test administrators. No statement anywhere in the paper claims independence between the intervention designers and those who conducted the evaluation. Because there is no evidence of independent conduct and the same team appears to have designed, delivered, and evaluated the intervention, criterion I is not met.
  • Y

    Year Duration

    • The study lasted only eight weeks in total, which is far short of the 75%-of-a-year requirement, and this criterion cannot be met because the weaker T criterion is also not met.
    • "Pre-test and post-test scores were collected from both groups before and after the eight-week intervention..." (p. 111)
    • Relevant Quotes: 1) "Pre-test and post-test scores were collected from both groups before and after the eight-week intervention in order to determine to what extent TBLT enhanced the participants' English proficiency." (p. 111) 2) "Fifth, this study did not measure skill retention on a delayed post-test; whether these gains can be sustained over time in this context remains unknown." (p. 119) Detailed Analysis: Criterion Y requires tracking of outcomes for at least 75% of a full academic year (roughly 9-10 months). The total duration of this study, from intervention start to final outcome measurement, was eight weeks (about two months), which is nowhere close to the required threshold. Per the criteria-specific instruction, since criterion T (Term Duration) is not met, criterion Y cannot be met either. Because the study duration of eight weeks is far shorter than the year-long requirement, and T is also not met, criterion Y is not met.
  • B

    Balanced Control Group

    • Both groups received comparable instructional time during the same designated teaching period and covered similar content, with the TBLT platform itself being the instructional method under test rather than an unmatched extra resource.
    • "In the implementation stage, the experimental group worked on these tasks during the designated teaching period, while the control group received teacher-led, grammar-focused lessons on similar content." (p. 110)
    • Relevant Quotes: 1) "In the implementation stage, the experimental group worked on these tasks during the designated teaching period, while the control group received teacher-led, grammar-focused lessons on similar content." (p. 110) 2) "In the preparation stage, TBLT task sequences were designed and connected to real-world entrepreneurial communication activities on the DreamBuilder platform. These included writing professional emails, preparing product pitches, and simulating virtual business negotiations." (p. 110) 3) "These participants were set into two groups: the experimental group (n = 20), which received TBLT-based instruction through the DreamBuilder platform, and the control group (n = 20), which got traditional, form-focused instruction." (p. 110) Detailed Analysis: Applying the criterion B decision procedure: the intervention being tested is the TBLT method itself, delivered through the DreamBuilder platform (including its task sequences, videos, articles, and forums), compared against traditional, teacher-led, grammar-focused instruction. Both conditions were delivered "during the designated teaching period" over the same eight-week window, on "similar content," so there is no evidence of extra class time or budget layered on top of a shared curriculum. The DreamBuilder platform access and its task-based resources are not a separable add-on resource; they are the treatment variable that RQ1 explicitly sets out to test ("How much does TBLT through DreamBuilder improve participants' English language skills?"). Under the decision tree, since the additional resources (platform access, task materials) are integral to the instructional method being tested rather than a supplementary extra given only to the intervention group on top of an otherwise shared design, the resource-balance requirement is satisfied by design (met() via the RESOURCES_ARE_TREATMENT branch), consistent with the spec's guidance that the treatment variable itself need not be matched in the control when it is explicitly the object of study. Because both groups received instruction during the same teaching period on similar content, with the platform and its resources being the pedagogical method itself under test rather than an unmatched extra resource, criterion B is met.

• Level 3 Criteria

  • R

    Reproduced

    • The paper explicitly presents this as a novel, first-of-its-kind study, no independent replication by a different research team was found via internet search, and the authors themselves call for future replication.
    • "Extending TBLT evidence to adult women entrepreneur learners through a purpose-built digital entrepreneurship platform constitutes the study's primary novelty." (Abstract, p. 108)
    • Relevant Quotes: 1) "Extending TBLT evidence to adult women entrepreneur learners through a purpose-built digital entrepreneurship platform constitutes the study's primary novelty." (Abstract, p. 108) 2) "However, most research on TBLT has focused on school-age students or general EFL groups in traditional classrooms. Its use in specialized digital entrepreneurship platforms targeting adult women learners in developing countries has not been studied." (p. 109) 3) "Future research should address the gaps left by this study: longitudinal studies to examine whether skill gains can be sustained over time, replication with larger and more geographically diverse samples, and studies that isolate specific TBLT task components..." (p. 119-120) Internet search findings: a search for independent replications of this specific study (published June 2026) found no peer-reviewed paper by a different research team replicating this DreamBuilder-TBLT trial with Indonesian women entrepreneurs. The only related item located was "Enhancing Women's Entrepreneurial Competencies through Intensive English Training Integrated with the DreamBuilder LMS," a community-service report published in the Indonesian Journal of Community Services (November 2025) by an overlapping author team (Khartha, Alfian, Bohang, Marhamah, plus additional co-authors). Because this item predates the present study, is not an independently authored replication, and is a community-service report rather than a peer-reviewed RCT replication, it does not satisfy criterion R. Detailed Analysis: Criterion R requires evidence that this specific study (or its central experimental claim in the same design and population) has been independently replicated by a different research team, ideally in a peer-reviewed outlet. The paper itself states that this is the first study of its kind ("primary novelty") examining TBLT with this population (adult women entrepreneurs) and this delivery platform (DreamBuilder), and explicitly calls for future "replication with larger and more geographically diverse samples" as an unmet need. This confirms that no replication of this study has yet occurred, and the internet search for subsequent independent replications (by other author teams, in peer-reviewed journals) returned no such evidence. Because the study is presented as novel and unreplicated, with replication explicitly listed as a direction for future research, and no independent peer-reviewed replication was found via internet search, criterion R is not met.
  • A

    All-subject Exams

    • Only English language proficiency was assessed, no other core subjects were measured, and this criterion cannot be met because the weaker E criterion is also not met.
    • "This test assessed productive and receptive skills needed for business communication, including reading comprehension, writing, and oral tasks." (p. 110)
    • Relevant Quotes: 1) "To get started, participants were given an English proficiency test as a pre-test and post-test to track changes in their language skills. This test assessed productive and receptive skills needed for business communication, including reading comprehension, writing, and oral tasks." (p. 110) 2) "the proficiency test was designed by the research team rather than using a validated standard instrument" (p. 119) Detailed Analysis: Criterion A requires that all main subjects be assessed using standardised exam-based assessments, and per the criteria-specific instruction, if criterion E is not met, criterion A cannot be met either. Here, only English language proficiency (reading, writing, and oral sub-skills within English) was measured; no other core subjects (e.g., mathematics, science) were assessed, which would be expected in a broader curriculum context, though this is a language-focused intervention. More decisively, because the assessment instrument used was researcher-designed rather than standardised (criterion E not met), criterion A automatically fails as well. Because criterion E is not met and only a single subject (English proficiency) was assessed with a non-standardised instrument, criterion A is not met.
  • G

    Graduation Tracking

    • The study did not track participants beyond the eight-week intervention, no follow-up publications tracking this cohort were found, and this criterion cannot be met because the weaker Y criterion is also not met.
    • "this study did not measure skill retention on a delayed post-test; whether these gains can be sustained over time in this context remains unknown." (p. 119)
    • Relevant Quotes: 1) "Fifth, this study did not measure skill retention on a delayed post-test; whether these gains can be sustained over time in this context remains unknown." (p. 119) 2) "Future research should address the gaps left by this study: longitudinal studies to examine whether skill gains can be sustained over time, replication with larger and more geographically diverse samples, and studies that isolate specific TBLT task components to identify the precise mechanisms driving language development..." (p. 119-120) Internet search findings: no subsequent, follow-up publication by this author team was located that tracks this same cohort of 40 AWE program participants over a longer period. The related community-service report found via search ("Enhancing Women's Entrepreneurial Competencies through Intensive English Training Integrated with the DreamBuilder LMS," Indonesian Journal of Community Services, November 2025) predates the present study and does not report longitudinal tracking of this RCT's participants. Detailed Analysis: Criterion G requires tracking participants until graduation from the relevant educational stage. This study involves adult, out-of-school learners in a short professional development program rather than a K-12 or degree program with a graduation milestone, and outcome data collection stopped at the immediate post-test after the eight-week intervention. The authors explicitly acknowledge the lack of any delayed or longitudinal follow-up as a limitation and call for future longitudinal research. Per the criteria-specific instruction, since criterion Y (Year Duration) is not met, criterion G cannot be met either. No follow-up publications tracking this cohort were identified via internet search. Because there was no long-term or graduation-level follow-up, no tracking publications were found, and Y is also not met, criterion G is not met.
  • P

    Pre-Registered

    • The paper contains no statement, link, or date indicating that the study protocol was pre-registered on any registry platform before data collection began, and no registration record was found via internet search.
    • Relevant Quotes: No quotes referencing a study registry, pre-registration platform, protocol registration number, or registration date were found anywhere in the Methods, Discussion, Limitations, or Acknowledgments sections of the paper. Internet search findings: a search for a pre-registration record for this study (registries such as OSF, AsPredicted, or similar platforms) found no matching entry for this title, authors, or intervention. Detailed Analysis: Criterion P requires explicit evidence that the full study protocol (hypotheses, methods, planned analyses) was registered on a public platform before data collection began. A thorough review of the Methods section, the Limitations section (which lists five specific limitations but does not mention pre-registration), and the rest of the paper found no mention of any pre-registration registry, protocol number, or registration date, and no external registry record was located via internet search. Because there is no evidence anywhere in the paper or via internet search of a pre-registered protocol, criterion P is not met.