SAP’s AI roleplays turn recertification into a validity test
Simulated customer conversations move beyond recall. Credential owners still need evidence that scoring is comparable, fair and tied to the capability being claimed.
What happened
SAP added AI-powered customer roleplays as an option for maintaining more than 100 certifications, alongside a non-AI quiz route.
Why it matters
A more realistic task can improve representation while also adding language, interface and scoring variance that changes what a pass means.
Applied evidence is promising, but the score still needs an argument
SAP said on 20 October that its Stay Certified programme will add short AI-powered roleplays across more than 100 certifications. Learners will work through simulated customer interactions tied to defined knowledge goals and current product content, then receive scenario-specific feedback. A stand-alone quiz remains available for people who prefer or require a non-AI route. SAP also introduced a 90-day renewal window and will pilot credit for hands-on TechEd sessions.
This changes the evidence surface from recall toward application, but the announcement does not publish a scoring rubric, reliability estimate, subgroup fairness analysis or evidence that the result predicts performance with customers. A realistic conversation can still measure irrelevant factors such as language fluency, familiarity with chat interfaces or a narrow response style.
The Guidelines for Technology-Based Assessment frame validity as evidence supporting an intended score interpretation and use. They distinguish evidence from content, response processes, internal structure, relations to other variables and consequences. They also warn that technology can increase construct representation while introducing construct-irrelevant variance and comparability problems.
For a certification owner, the release therefore creates a validation task. Define the exact capability inferred from passing; map each scenario turn and scoring feature to it; compare human ratings and automated scores on a blinded sample; test agreement across scenario variants; and examine outcomes by language and accommodation group where lawful. Compare the roleplay and quiz pathways on the decision they support, not merely average pass rates. Give candidates a review and appeal route for consequential decisions.
These are proposed controls, not evidence that SAP's roleplays are invalid. The earlier Workday applied-skill article made the case for work evidence over labels. Here the sharper question is whether a simulation yields comparable, fair evidence for maintaining a credential. Preserve the non-AI route while the validity argument is assembled, and do not describe personalised feedback as proof of job performance.