AI-assisted performance is not the same as acquired skill

The tool raised task performance

A randomized study in Nature Human Behaviour assigned 487 management consultants to complete unfamiliar data-science work with or without ChatGPT. The tasks covered cleaning and combining data, building a predictive model and interpreting results. Consultants with access to the tool performed better on each task than consultants without it, but neither group reached the expert benchmark used by the researchers.

That result is valuable and bounded. It shows that a general-purpose assistant can expand what experienced knowledge workers accomplish in a neighbouring technical domain during the experiment. It does not show that the participants became data scientists, that quality would persist in production, or that the tool eliminated the need for expert review.

The most important result came after the tasks. Participants took a technical-knowledge test without ChatGPT. The researchers found no evidence that access during the work produced greater post-experiment technical knowledge. Assisted performance and acquired knowledge moved differently.

The experiment used specified tasks and one population of consultants. A later assessment cannot capture every form of learning, and a null result does not prove that learning is impossible. It does show why programmes should stop using polished assisted output as the sole proxy for skill development.

Transfer can depend on the work and the worker

A separate three-month field experiment in patent drafting followed 133 patent lawyers at 11 firms. Researchers reported improved work with AI assistance and an overall 0.32-standard-deviation gain in unaided judgment after the intervention. The gain was concentrated among senior lawyers; junior lawyers showed no average improvement.

The settings are not directly comparable. Patent drafting extends established professional judgment over repeated work, while the consultant experiment introduced unfamiliar technical tasks. The second study therefore does not overturn the first. Together they suggest that transfer may depend on prior knowledge, repetition, feedback and whether the worker must actively diagnose the output.

Design two assessments

First measure performance with the approved tool. Use representative work, real constraints and a scoring rubric that covers correctness, explanation, uncertainty, privacy and escalation. Record the model, prompt context and reviewer effort so the result can be reproduced.

Then test transfer without the tool. Ask the learner to explain why a result is plausible, locate a planted error, reconstruct a method, compare two approaches or identify when an expert is required. The objective is not to ban assistance. It is to find which parts of capability remain available when the tool fails, changes or produces a confident error.

Use the gap between the two assessments to choose an intervention. If assisted output is strong but unaided diagnosis is weak, add prediction-before-generation, explanation and error-analysis exercises. If both are weak, narrow the task or provide foundational instruction. If both are strong, expand authority gradually and continue sampling difficult cases.

This sharpens the earlier warning in the UK workflow-depth analysis: access and frequency are not evidence that a workflow or skill has become robust.

Decision rule

Do not label a worker upskilled because an AI-supported task passed. Record tool-present performance and tool-absent transfer separately, and grant broader responsibility only when the required form of judgment survives a representative unaided test.