The parallels to large language models—AI tools that simulate human language—are striking. Nearly half of U.S. adults who had used AI chatbots reported using them for therapeutic support. With millions of total users on platforms like ChatGPT, the number of people using AI for mental health reasons is potentially staggering. Many people likely turn to these tools because of lack of access, stigma, or distrust of traditional services.
AI’s all-too-human flaws
Much like jails, AI chatbots aren’t designed for these complex, emotional tasks, and there are serious dangers of relying on them for mental health support. But many of AI’s problems start with humans.
In our latest brief, Julian Adler—our Managing Director of Research, Innovation, and National Impact—argues that those problems should be met with investment in higher-quality care across the board. Instead of reflexively relying on AI or discounting it altogether, leaders from across sectors must work to improve therapeutic practices from both humans and machines.
Many of AI’s shortcomings in a mental health support context are problems for people, too. While artificial intelligence can mimic empathetic language, it’s less effective at calibrating empathy and balancing it with accountability and safety. In many cases, AI bots fail to push back against misguided thinking or risky behaviors. But these issues aren’t unique to AI. Human practitioners may also struggle with showing empathy in the right ways, without overwhelming the client or avoiding tough questions.
The tendency to overlook wider contexts is another challenge for both AI chatbots and human clinicians. In a setting like the justice system, what looks like a behavioral health issue might be a perfectly normal reaction to frightening circumstances like incarceration, housing insecurity, or family separation. Confusing the impact of these systemic harms for individual mental health struggles is a danger for AI and human practitioners alike.
Beyond the algorithm
Striking the right balance—between empathy and accountability, or between individual and systemic perspectives—isn’t an exact science. While models are critical guides, effective mental health support often means recognizing when a client would benefit from a different approach.
As our brief argues, some of the most important skills in mental health practice require going off script.
This poses a unique challenge for AI tools, which are trained on text. If much of what happens in clinical practice isn’t written down, more data for AI won’t necessarily solve the problem. What’s needed is deeper care and attention to the unwritten processes—not just the models—that make for effective, high-quality treatment. And that means taking seriously the experiences of both practitioners and people receiving help, especially in sensitive settings.
AI tools are increasingly being used in place of traditional support systems. That’s especially true for people who struggle to access typical care, including those impacted by the justice system. Amid well-founded concerns about the risks of AI, these new challenges present an opportunity to improve the quality and accessibility of care for those who need it most.
Learn more about our work as a founding partner of the AI and Justice Consortium, where we bring together leaders in criminal justice, tech, and communities to make sure we get AI right in the justice space.
