AI Feedback on Therapy Skills: What Can It Actually Assess?

“Was my response good?” is a common question after a training role-play. A simple yes or no rarely helps. Meaningful feedback needs to point to an observable response, explain why it might matter, and offer a direction for further practice.

AI-generated feedback promises to make that process more immediate. But different aspects of psychotherapy performance cannot be assessed with equal confidence.

What may be observable in a transcript?

A system reviewing a fictional training conversation may identify whether the trainee:

• uses open or closed questions,
• attempts a reflection or paraphrase,
• moves quickly to advice,
• asks many questions in succession,
• invites collaboration on a goal.

These are potentially observable features of language. Their identification can still be wrong: a sentence that looks like advice in isolation may make sense in the surrounding exchange.

What is harder to judge?

Whether a response was clinically appropriate, whether the client felt safe, and whether a strong therapeutic alliance developed cannot be established from automated text analysis alone. Tone, pauses, culture, history, and the relationship all affect meaning.

A numerical “empathy score,” for example, should not be treated as an objective verdict on the therapist. At best, it is a limited measure that needs careful validation for its specific setting.

What does research suggest?

A 2025 Journal of Medical Internet Research study examined AI feedback in communication training for mental health care professionals. Experts rated 79.06% of 1,108 assessed feedback points as completely correct; false positives and false negatives were also documented. This is evidence of potential usefulness and a reminder that errors are real.

A separate 2026 study compared successive graduate-class cohorts in a clinical intervention course. The cohort receiving weekly AI feedback improved more rapidly in measured skills, though the authors called for stronger randomized research. Findings from one educational implementation cannot simply be assumed to apply to every AI tool.

A practical way to review AI feedback

Suppose a system flags your phrase “You need to set boundaries” as premature advice. First, return to the exact moment. Was advice requested? What preceded the statement? How did the simulated client respond?

Then try an alternative: “What options feel available to you right now?” Compare the two responses and explain your reasoning. The objective is not to obey an algorithm: it is to develop a more considered therapeutic choice.

What good feedback looks like

Useful feedback is specific, traceable to a passage, and open to challenge. It may support reflection between supervised learning sessions, but cannot replace expert supervision or certify competence by itself.

TheraMentor treats feedback as an educational prompt rather than a professional qualification. After your next simulated session, revisit two moments and decide which suggestion genuinely improves your reasoning.