A large GenAI learning average still needs a local pilot
A 58-study meta-analysis reports a large pooled effect, but very high heterogeneity and a prediction interval crossing zero make the average a planning prior, not a programme guarantee.
What happened
Frontiers in Education published a random-effects meta-analysis of 68 effect sizes from 58 studies on generative AI and learning, cognitive and linguistic outcomes.
Why it matters
The pooled estimate is encouraging, yet variation across studies is large enough that a local intervention can plausibly underperform the average or fail to transfer.
The average is large; the spread is larger
A Frontiers in Education meta-analysis synthesized 68 effect sizes from 58 studies of generative AI in learning, cognitive and linguistic outcomes. The random-effects estimate was Hedges g=0.864, with a 95% confidence interval from 0.720 to 1.009.
That headline is not a forecast for a particular academy or workforce programme. Heterogeneity was about 87.9%, and the reported prediction interval ran from -0.213 to 1.942. In practical terms, the evidence is compatible with substantially different effects in a new setting. The review also notes mixed designs and outcomes, possible participant overlap and concentration in East Asian settings. Those limits do not erase the average; they change how it should be used.
Structure may matter more than access
An independent randomized study assigned 95 learners to structured generative-AI training grounded in self-regulated learning or to a control condition. The structured group improved revision performance with d=0.81. Among initially incorrect answers, 69.3% became correct after revision versus 41.3% in the control group.
The experiment offers a mechanism: learners were trained to plan, interrogate and revise rather than simply request answers. It is still a short-term result in one scientific subject and one tool context. It does not establish retention, workplace transfer or the effect of the same training on experienced employees.
Convert the average into a pilot prior
Use the pooled effect to justify testing, not to pre-commit a benefit. Randomize or phase the rollout where feasible. Define the target task and baseline before training. Record tool use, the quality of questions and revisions, independent task performance and a delayed assessment after access is removed.
Keep assisted output and acquired capability separate. A polished answer can show useful augmentation while masking weak diagnosis or retention. Track who benefits, who does not and whether the intervention adds review workload. Report the local estimate and uncertainty even when the result is smaller than the literature average.
This sharpens the temporary-upskilling assessment: the new synthesis supports experimentation, while its spread makes transfer tests more important, not less.
Decision rule
Do not promise g=0.864 in a local business case. Treat it as a prior for a bounded pilot, and scale only when independent task performance and delayed retention improve in the population and workflow that will actually use the programme.