Two randomized trials from 2026 produce opposite results for different educational AI systems. In Taipei, 1,047 high school students made progress equivalent to six to eight months of schooling with an AI-powered tutor, provided the learning path was structured. In the subsample of identical sections, access to the conversational AI was associated with a drop of 0.37 standard deviations in final grades. The studies suggest that configuration and pedagogical integration may matter, without causally isolating this explanation between the two trials.
The essentials
- A randomized trial in Taipei shows a gain of 0.15 standard deviations with an AI tutor integrated into a structured learning path (Academy of Management Proceedings, July 2026).
- An American trial on 2,379 students shows a drop of 0.37 standard deviations when the same type of tool is deployed without usage rules (IZA, October 2026).
- The effect measured in Taipei stems from assignment to an adaptive sequence, in a setting where both groups had the same safeguard against cognitive shortcutting.
- Configuration and pedagogical integration appear as important dimensions of the tool’s effect.
- Students with the least home support risk suffering most from an unregulated deployment, and benefiting most from a well-designed one.
Taipei measured what works, and what works is precise
Students were divided between a personalized problem sequence and a fixed easy-to-difficult sequence, with the same AI tutor in both groups. The measured gain, 0.15 standard deviations, corresponds to what education researchers estimate as six to eight months of additional learning. This result is substantial for an intervention spanning one school year.
What makes this result interesting is its condition. The Taipei learning path did not let students navigate freely through the tutor. It defined precise sequences: the student had to answer alone first, potentially make mistakes, then receive targeted help. The AI intervened after effort, not in its place. The trial shows that the adaptive sequencing algorithm improves results compared to a fixed sequence with the same tutor.
Large-scale standardized assessments, including the OECD’s PISA surveys, have for years documented a similar finding for computer-assisted instruction in general: the benefit depends less on the tool than on the pedagogical framework in which it is embedded. The Taipei trial establishes this for conversational tutors with new precision.
The American trial shows the flip side of the same mechanism
The study published by the Institut für Zukunft der Arbeit (IZA) in October 2026 examined university students using an AI tutor integrated into their courses. The result: a drop of 0.37 standard deviations in final grades, an effect twice as large in absolute value as the Taipei gain.
The two trials observed distinct results. This stems from the absence of protocol. Students could ask the conversational AI questions about their course content. Many asked for answers, not hints. The tutor answered.
The study suggests that access to the conversational AI may have displaced certain existing learning activities.
This silent displacement of work is what learning specialists call circumventing desirable cognitive load. When a student struggles with a problem and figures out how to solve it, they build a lasting mental representation. When the machine solves it for them, the correct answer passes through without leaving a trace in their reasoning. Short-term grades may seem stable or even improve; the real cost only appears in final evaluations, when the student faces new problems alone.
Classroom governance decides more than the algorithm
Lê Nguyên Hoang, a machine learning researcher and cofounder of Tournesol, a platform working toward pluralistic governance of algorithmic recommendations, argues that an AI system’s behavior in real situations depends less on its code than on the rules governing its use. Applied to the classroom, this thesis becomes concrete: usage modalities matter, without causally isolating when or how to question the tutor.
The two trials are compatible with the importance of usage design, without jointly confirming the same causal explanation. The results differ between two distinct devices, one testing adaptive sequencing and the other access to a course conversational AI. The experimental variables are adaptive sequencing in Taipei and access to a conversational AI in Maryland.
A competing interpretation deserves examination. Economists working on education markets, in the tradition of Philippe Aghion on innovation and growth, argue that the diffusion of a powerful tool eventually forces institutions to adapt: competitive pressure between schools, or between national education systems, gradually selects usages that work. This Schumpeterian dynamic has its logic. But it assumes time, losers during the transition, and information mechanisms that allow actors to distinguish good usages from bad ones. The two 2026 trials show that these mechanisms are not automatic: large-scale deployment without protocol can produce measurable harm before selection occurs.
This finding aligns with what algorithmic surveillance at work illustrates in other contexts: delegating a decision to an algorithm without defining the scope of that delegation shifts the risk toward those least able to manage it.
The least supported students pay the highest price
Widening gaps are not a metaphor. In Maryland, estimated losses were higher among first-generation students; Taipei reports mostly effects by initial level and school selectivity.
Conversely, estimated gains in Taipei were higher among beginning students and in less selective high schools. A well-supervised AI tutor can compensate for some of this lack. It is available at all hours, patient, adaptive. This is a real advantage for families who cannot afford tutoring. But a negative result was observed for access to the conversational AI in the American trial.
The geography of access to AI careers traces a similar inequality at the scale of labor markets: gains concentrate where integration conditions are already favorable. At school, the same pattern appears at the classroom scale: AI amplifies what exists, in both directions.
The choices ahead
Over a fifteen-year horizon, the question is no longer whether conversational tutors will enter classrooms, but in what institutional state they will find them. If education systems treat this period as a window for collective learning, current choices about usage protocols could produce a stabilized pedagogical doctrine, comparable to what reading research gradually imposed as the norm for teaching over preceding decades. Schools that formalize usage rules now will accumulate experience and professional culture that others will take years to catch up with. This experience asymmetry risks reproducing, between schools, the same gap that the absence of protocol already produces between students.
If, conversely, deployments generalize under pressure from budgets and commercial availability of tools, without pedagogical conditions being defined, the risk is not a visible and correctable failure, but slow and diffuse degradation of independent reasoning capacity, invisible in short-term statistics and politically difficult to attribute to a specific decision. The moment this degradation became measurable would coincide with the arrival on the labor market of a generation trained without explicit cognitive demands—a horizon distant enough that responsibility for current choices becomes untraceable. This is precisely why signals to monitor now—the gap in results between students using a tutor with or without protocol, the share of schools having formalized usage rules—have the value of an early warning that final evaluations cannot provide in time.
The collective needs conditioning a favorable trajectory remain structurally underfunded. Training teachers to design sequences where effort precedes help requires reform of initial and continuing education that cannot be improvised at deployment time. Having assessment protocols capable of measuring reasoning independently of assistance received requires pedagogical engineering investment that few institutions have yet budgeted. Conditioning public funding for tutoring tools on prior production of a validated pedagogical protocol would be a path forward within reach of current regulators, without waiting for research to accumulate new trials. Creating an observatory of educational AI uses tasked with comparing results according to integration modalities would make this evidence continuous, accessible, and comparable across systems, where it remains today fragmented in isolated trials that few practitioners know about.
Two trajectories are open, and current data allow us to distinguish them, without predicting the outcome.
In the first, schools deploying AI tutors progressively formalize usage rules: the student must produce an attempt at a solution before questioning the tutor, final evaluations are designed to measure reasoning independently of help received, teachers are trained to design sequences where AI intervenes after effort, not in its place. In this scenario, the Taipei gain of 0.15 standard deviations could generalize, even in contexts where family support is weaker. Students benefiting least from a rich educational environment would gain relatively the most.
In the second, tutors are deployed as immediate-access tools, without protocol, because it is faster to implement and negative effects take time to appear in statistics. The effect observed in the United States would present a risk of large-scale replication. Students capable of self-regulating their use, those who intuitively understand that questioning the machine before trying weakens their own learning, fare well. The others lose ground invisibly, until exams.
What would allow distinguishing these two trajectories before they become irreversible: tracking the gap in results on standardized assessments between students using a tutor with defined protocol and those using it without constraint, measured each school year. And observing the share of schools having formalized usage rules in their pedagogical project, a simple indicator making the diffusion of good practices visible before results record it.
Several collective needs remain in both scenarios. Training teachers to design sequences that explicitly define what the student produces alone remains an unfulfilled condition in most education systems. Having assessment protocols capable of measuring student reasoning independently of help received is a technical requirement that few schools have begun to take seriously.
Two paths forward stand out from available data. Conditioning public funding for AI tutoring tools on production of a validated pedagogical protocol would force deployments to define usage moments before funding them. Creating an observatory of educational AI uses tasked with comparing results according to integration modalities, drawing on existing trials and those to come, would allow accumulating evidence that schools could consult before choosing their system. The question of who builds this data, and for whom, is already emerging elsewhere in AI deployment: it arises here with particular urgency, because children cannot wait for the market to select good uses.
How teachers use this evidence
Some of the burden rests on teachers, but not all. Designing a pedagogical sequence that defines what the student must produce alone before accessing the tutor requires specific training that most education systems have not yet integrated. In Taiwanese schools that produced the 0.15 standard deviation gain, this design work was done beforehand, with explicit teacher support.
Unregulated deployment requires no training. It simply means opening access. This is precisely why it is more common.
The 2026 data can inform public purchasing and deployment policies. It does not say which technology to choose. It shows that access to a conversational AI without prescribed pedagogical integration was accompanied by negative results, without causally establishing that the absence of protocol is the cause. Decision-makers deploying AI tutors at the district or ministry scale now have two clear points of comparison. Using this data or ignoring it is, at this stage, a deliberate choice.
Sources
- Academy of Management Proceedings, randomized trial in Taipei (1,047 high school students, July 2026), https://journals.aom.org/doi/10.5465/AMPROC.2026.17622abstract
- IZA, Institute of Labor Economics, The Effects of Course-Integrated AI Tutoring, October 2026 (2,379 students, randomized trial, United States), IZA Discussion Papers
- Lê Nguyên Hoang, Tournesol, pluralistic and participatory governance of AI models, https://tournesol.app/
