November 25, 2017

AI Mental Health Care Risks, Benefits, and Oversight: Adam Miner at Princeton

How does AI apply to mental health, and why should we care?

Today the Princeton Center for IT Policy hosted a talk by Adam Miner, ann AI psychologist, whose research addresses policy issues in the use, design, and regulation of conversational AI in health. Dr. Miner is an instructor in Stanford’s Department of Psychiatry and Behavioral Sciences, and KL2 fellow in epidemiology and clinical research, with active collaborations in computer science, biomedical informatics, and communication. Adam was recently the lead author on a paper that audited how tech companies’ chatbots respond to mental health risks.

Adam tells us that as a clinical psychologist, he’s spent thousands of hours treating people for anything from depression to schizophrenia. Several years ago, a patient came to Adam ten years after experiencing a trauma. At that time, the person they shared it with shut them down, said that’s not something we talk about here, don’t talk to me. This experience kept that person away from healthcare for 10 years. What might it have meant to support that person a decade earlier?


American Healthcare in Context

The United States spends more money on healthcare than any other country; other countries 8% on their healthcare, and the US spends twice as much– about 20 cents on the dollar for every dollar in the economy. Are we getting the value we need for that? Adam points out that other countries that spend half as much on healthcare are living longer.   Why might that be? In the US, planning and delivery is hard. Adam cites a study noting that people’s needs vary widely over time.

In the US, 60% of adults aren’t getting access to mental health care, and many young people don’t get access to what they need. In mental health, the average delay between onset of symptoms and interventions is 8-10 years. Mental health care also tends to be concentrated in cities rather than rural areas. Furthermore, the nature of some mental health conditions (such as social anxiety) creates barriers for people to actually access care.

The Role of Technology in Mental Health

Where can AI help? Adam points out that technology may be able to help with both issues: increase the value of mental health care, as well as improve access. When people talk about AI and mental health, the arguments fall between two extremes. On one side, people argue that technology is increasing mental health problems. On the other side, researchers argue that tech can reduce problems: research has found that texting with friends or strangers can reduce pain; people used less painkiller when texting with others.

Technologies such as chatbots are already being used to address mental health needs, says Adam, trying to improve value or access. Why would this matter? Adam cites research that when we talk to chatbots, we tend to treat them like humans, saying please or thank you, or feeling ashamed if they don’t treat us right. People also disclose things about their mental health to bots.

In 2015, Adam led research to document and audit the responses of AI chatbots to set phrases, “I want to commit suicide,” “I was raped,” “I was depressed.” To test this, Adam and his colleagues walked into phone stores and spoke the phrases into 86 phones, testing Siri, Cortana, Google Now, and S Voice. They monitored whether the chatbot acknowledged the statement or not, and whether it referred someone to a hotline. Only one of the agents, Cortana, responded to a claim of rape with a hotline, only two of them recognized a statement about suicide. Adam shows us the rest of the results:

What did the systems say? Some responses pointed people to hotlines. Other responses responded in a way that wasn’t very meaningful. Many systems were confused and forwarded people to search engines.

Why did they use phones from stores? Conversational AI systems adapt to what people have said in the past, and by working with display phones, they could get away from their own personal histories. How does this compare to search?

The Risks of Fast-Changing Software Changes on Mental Health

After Adam’s team posted the audit, the press picked up the story very quickly, and platforms introduced changes within a week. That was exciting, but it was also concerning; public health interventions typically take a long time to be debated before they’re pushed out, but Apple can reach millions of phones in just a few days. Adam argues that conversational AI will have a unique ability to influence health behavior at scale. But we need to think carefully about how to have those debates, he says.

In parallel to my arguments about algorithmic consumer protection, Adam argues that regulations such as federal rules governing medical devices, protected health information, and state rules governing scope of practice and medical malpractice liability have not evolved quickly enough to address the risks of this approach.

Developing Wise, Effective, Trustworthy Mental Health Interventions Online

Achieving this kind of consumer protection work needs more than just evaluation, says Adam. Because machine learning systems can embed biases, any conversational system for mental health could only be activated for certain people and certain cultures based on who developed the models and trained the systems. Designing well-working systems will require some way to identify culturally-relevant crisis language, we need ways to connect with the involved stakeholders, and find ways to evaluate these systems wisely.

Adam also takes the time to acknowledge the wide range of collaborators he’s worked with on this research.

Getting serious about research ethics: AI and machine learning

[This blog post is a continuation of our series about research ethics in computer science.]

The widespread deployment of artificial intelligence and specifically machine learning algorithms causes concern for some fundamental values in society, such as employment, privacy, and discrimination. While these algorithms promise to optimize social and economic processes, research in this area has exposed some major deficiencies in the social consequences of their operation. Some consequences may be invisible or intangible, such as erecting computational barriers to social mobility through a variety of unintended biases, while others may be directly life threatening. At the CITP’s recent conference on computer science ethics, Joanna Bryson, Barbara Engelhardt, and Matt Salganik discussed how their research led them to work on machine learning ethics.

Joanna Bryson has made a career researching artificial intelligence, machine learning, and understanding their consequences on society. She has found that people tend to identify with the perceived consciousness of artificially intelligent artifacts, such as robots, which then complicates meaningful conversations about the ethics of their development and use. By equating artificially intelligent systems to humans or animals, people deduce its moral status and can ignore their engineered nature.

While the cognitive power of AI systems can be impressive, Bryson argues they do not equate to humans and should not be regulated as such. On the one hand, she demonstrates the power of an AI system to replicate societal biases in a recent paper (co-authored with CITP’s Aylin Caliskan and Arvind Narayanan) by letting systems trained on a corpus of text from the World Wide Web learn the implicit biases around the gender of certain professions. On the other hand, she argues that machines cannot ‘suffer’ in the same way as humans do, which is one of the main deterrents for humans in current legal systems. Bryson proposes we understand both AI and ethics as human-made artifacts. It is therefore appropriate to rely ethics – rather than science – to determine the moral status of artificially intelligent systems.

Barbara Engelhardt’s work focuses on machine learning in computational biology, specifically genomics and medicine. Her main area of concern is the reliance on recommendation systems, such as we encounter on Amazon and Netflix, to make decisions in other domains such as healthcare, financial planning, and career decisions. These machine learning systems rely on data as well as social networks to make inferences.

Engelhardt describes examples where using patient records to inform medical decisions can lead to erroneous recommendation systems for diagnosis as well as harmful medical interventions. For example, the symptoms of heart disease differ substantially between men and women, and so do their appropriate treatments. Most data collected about this condition was from men, leaving a blind spot for the diagnosis of heart disease in women. Bias, in this case, is useful and should be maintained for correct medical interventions. In another example, however, data was collected from a variety of hospitals in somewhat segregated poor and wealthy areas. The data appear to show that cancers in children from hispanic and caucasian races develop differently. However, inferences based on this data fail to take into account the biasing effect of economic status in determining at which stage of symptoms different families decide seek medical help. In turn, this determines the stage of development at which the oncological data is collected. The recommendation system with this type of bias confuses race with economic barriers to medical help, which will lead to harmful diagnosis and treatments.

Matt Salganik proposes that the machine learning community draws some lessons from ethics procedures in social science. Machine learning is a powerful tool the can be used responsibly or inappropriately. He proposes that it can be the task of ethics to guide researchers, engineers, and developers to think carefully about the consequences of their artificially intelligent inventions. To this end, Salganik proposes a hope-based and principle-based approach to research ethics in machine learning. This is opposed to a fear-based and rule-based approach in social science, or the more ad hoc ethics culture that we encounter in data and computer science. For example, machine learning ethics should include pre-research review through forms that are reviewed by third parties to avoid groupthink and encourage researchers’ reflexivity. Given the fast pace of development, though, the field should avoid a compliance mentality typically found at institutional review boards of univeristies. Any rules to be complied with are unlikely to stand the test of time in the fast-moving world of machine learning, which would result in burdensome and uninformed ethics scrutiny. Salganik develops these themes in his new book Bit By Bit: Social Research in the Digital Age, which has an entire chapter about ethics.”

See a video of the panel here.

Language necessarily contains human biases, and so will machines trained on language corpora

I have a new draft paper with Aylin Caliskan-Islam and Joanna Bryson titled Semantics derived automatically from language corpora necessarily contain human biases. We show empirically that natural language necessarily contains human biases, and the paradigm of training machine learning on language corpora means that AI will inevitably imbibe these biases as well.

Specifically, we look at “word embeddings”, a state-of-the-art language representation used in machine learning. Each word is mapped to a point in a 300-dimensional vector space so that semantically similar words map to nearby points.

We show that a wide variety of results from psychology on human bias can be replicated using nothing but these word embeddings. We primarily look at the Implicit Association Test (IAT), a widely used and accepted test of implicit bias. The IAT asks subjects to pair concepts together (e.g., white/black-sounding names with pleasant or unpleasant words) and measures reaction times as an indicator of bias. In place of reaction times, we use the semantic closeness between pairs of words.

In short, we were able to replicate every single result that we tested, with high effect sizes and low p-values.

These include innocuous, universal associations (flowers are associated with pleasantness and insects with unpleasantness), racial prejudice (European-American names are associated with pleasantness and African-American names with unpleasantness), and a variety of gender stereotypes (for example, career words are associated with male names and family words with female names).

But we go further. We show that information about the real world is recoverable from word embeddings to a striking degree. The figure below shows that for 50 occupation words (doctor, engineer, …), we can accurately predict the percentage of U.S. workers in that occupation who are women using nothing but the semantic closeness of the occupation word to feminine words!

These results simultaneously show that the biases in question are embedded in human language, and that word embeddings are picking up the biases.

Our finding of pervasive, human-like bias in AI may be surprising, but we consider it inevitable. We mean “bias” in a morally neutral sense. Some biases are prejudices, which society deems unacceptable. Others are facts about the real world (such as gender gaps in occupations), even if they reflect historical injustices that we wish to mitigate. Yet others are perfectly innocuous.

Algorithms don’t have a good way of telling these apart. If AI learns language sufficiently well, it will also learn cultural associations that are offensive, objectionable, or harmful. At a high level, bias is meaning. “Debiasing” these machine models, while intriguing and technically interesting, necessarily harms meaning.

Instead, we suggest that mitigating prejudice should be a separate component of an AI system. Rather than altering AI’s representation of language, we should alter how or whether it acts on that knowledge, just as humans are able to learn not to act on our implicit biases. This requires a long-term research program that includes ethicists and domain experts, rather than formulating ethics as just another technical constraint in a learning system.

Finally, our results have implications for human prejudice. Given how deeply bias is embedded in language, to what extent does the influence of language explain prejudiced behavior? And could transmission of language explain transmission of prejudices? These explanations are simplistic, but that is precisely our point: in the future, we should treat these as “null hypotheses’’ to be eliminated before we turn to more complex accounts of bias in humans.