AI in Psychology Education: How to Evaluate a Training Pilot

Introducing AI into a psychology or counselling programme does not have to begin with a major technology investment. A small, carefully structured educational pilot can answer the more important question: does simulated-client practice create useful learning opportunities for these particular trainees?

The success criterion is not whether students find the software impressive. It is whether they can practise defined skills in ways that can be evaluated.

Begin with learning outcomes

Before choosing a platform, educators should identify the intended skill: recognising emotion, using open-ended questions, pacing interventions, or explaining clinical communication choices.

If the goal is only “try AI,” evaluation often becomes a survey of how much learners liked the interface.

A sample small pilot

A seminar might have three elements: orientation to a shared skills rubric, brief simulations with fully fictional clients, and discussion of selected moments with an instructor.

Learners could encounter a common core scenario so their responses are easier to compare. Difficulty could increase gradually if doing so helps meet the educational goals.

What should be measured?

Participation: how many learners completed the exercises and returned for further practice?

Perceived usefulness: did learners find the exercise realistic or stressful? That matters but is not the same as demonstrated competence.

Performance against clear criteria: can trained reviewers evaluate anonymised role-play samples before and after the activity? Where feasible, reviewers should not know which learners used the AI tool.

Comparison: how does the approach compare with an available alternative, such as peer role-play or instructional video?

Ethics and safety: are data handling, learner support, and human oversight appropriate?

What can research teach programme designers?

A 2026 randomized study with 87 participants compared AI virtual-patient exercises with watching role-play videos. Despite improvements on some self-report measures in the simulation group, the AI condition did not show overall superiority to the video condition. That is a strong reason to demand meaningful comparison, not just positive testimonials.

A research paper on AI in CBT training and supervision also highlights implementation challenges involving data quality, ethics, and the need for more rigorous evidence.

Who remains accountable?

The educational institution defines learning objectives, selects activities, manages ethical requirements, and decides how feedback is interpreted. AI does not confer clinical qualifications and should not replace supervision.

A thoughtful pilot can also allow an institution to disable AI-generated coaching while retaining simulation, should educators prefer exclusively human feedback.

From enthusiasm to measurable learning

The useful question is not “How do we add AI to the curriculum?” but “Which skill should improve, and how will we tell?”

TheraMentor can provide fictional simulation exercises that educators incorporate into structured training. Any claimed educational outcomes still need to be evaluated in the actual setting where the tool is used.