October 20, 2017

What does it mean to ask for an “explainable” algorithm?

One of the standard critiques of using algorithms for decision-making about people, and especially for consequential decisions about access to housing, credit, education, and so on, is that the algorithms don’t provide an “explanation” for their results or the results aren’t “interpretable.”  This is a serious issue, but discussions of it are often frustrating. The reason, I think, is that different people mean different things when they ask for an explanation of an algorithm’s results.

Before unpacking the different flavors of explainability, let’s stop for a moment to consider that the alternative to algorithmic decisionmaking is human decisionmaking. And let’s consider the drawbacks of relying on the human brain, a mechanism that is notoriously complex, difficult to understand, and prone to bias. Surely an algorithm is more knowable than a brain. After all, with an algorithm it is possible to examine exactly which inputs factored in to a decision, and every detailed step of how these inputs were used to get to a final result. Brains are inherently less transparent, and no less biased. So why might we still complain that the algorithm does not provide an explanation?

We should also dispense with cases where the algorithm is just inaccurate–where a well-informed analyst can understand the algorithm but will see it as producing answers that are wrong. That is a problem, but it is not a problem of explainability.

So what are people asking for when they say they want an explanation? I can think of at least four types of explainability problems.

The first type of explainability problem is a claim of confidentiality. Somebody knows relevant information about how a decision was made, but they choose to withhold it because they claim it is a trade secret, or that disclosing it would undermine security somehow, or that they simply prefer not to reveal it. This is not a problem with the algorithm, it’s an institutional/legal problem.

The second type of explainability problem is complexity. Here everything about the algorithm is known, but somebody feels that the algorithm is so complex that they cannot understand it. It will always be possible to answer what-if questions, such as how the algorithm’s result would have been different had the person been one year older, or had an extra $1000 of annual income, or had one fewer prior misdemeanor conviction, or whatever. So complexity can only be a barrier to big-picture understanding, not to understanding which factors might have changed a particular person’s outcome.

The third type of explainability problem is unreasonableness. Here the workings of the algorithm are clear, and are justified by statistical evidence, but the result doesn’t seem to make sense. For example, imagine that an algorithm for making credit decisions considers the color of a person’s socks, and this is supported by unimpeachable scientific studies showing that sock color correlates with defaulting on credit, even when controlling for other factors. So the decision to factor in sock color may be justified on a rational basis, but many would find it unreasonable, even if it is not discriminatory in any way. Perhaps this is not a complaint about the algorithm but a complaint about the world–the algorithm is using a fact about the world, but nobody understands why the world is that way. What is difficult to explain in this case is not the algorithm, but the world that it is predicting.

The fourth type of explainability problem is injustice. Here the workings of the algorithm are understood but we think they are unfair, unjust, or morally wrong. In this case, when we say we have not received an explanation, what we really mean is that we have not received an adequate justification for the algorithm’s design.  The problem is not that nobody has explained how the algorithm works or how it arrived at the result it did. Instead, the problem is that it seems impossible to explain how the algorithm is consistent with law or ethics.

It seems useful, when discussing the explanation problem for algorithms, to distinguish these four cases–and any others that people might come up with–so that we can zero in on what the problem is. In the long run, all of these types of complaints are addressable–so that perhaps explainability is not a unique problem for algorithms but rather a set of commonsense principles that any system, algorithmic or not, must attend to.

Job Opening: Associate Director at Princeton CITP

Princeton’s Center for Information Technology Policy (CITP), which, among other things, hosts Freedom to Tinker, is looking for a new Associate Director. Please come and work with us!

CITP is an interdisciplinary nexus of expertise in technology, engineering, public policy, and the social sciences. In keeping with the strong University tradition of service, the Center’s researchteaching, and events address digital technologies as they interact with society.

The Associate Director is the primary public face of CITP and plays a vital role in the management and direction of the Center, as the principal administrator for our core research, education and outreach both on campus and beyond. The position involves a combination of academic and administrative tasks.

This individual develops, plans and executes the Center’s lecture series, workshops and policy briefings; recruits visiting researchers and policy experts, coordinating the selection and appointment process; contributes to the Center’s research initiatives; promotes and supports the Center’s undergraduate certificate offerings and other student programs; develops the Center budget in collaboration with the Director and is responsible for the careful and appropriate management of Center funds; edits the Center’s research blog; performs day-to-day Center operations; cultivates high profile research collaborations and joint public events with other institutions; manages public communications through the website and print materials; coordinates grant writing and development initiatives as appropriate; and supervises two administrative staff members.

This is a 3 year term position with the possibility of renewal.

All applicants must apply on the Princeton’s job website, requisition number: 2017-7403:

https://main-princeton.icims.com/jobs/7403/associate-director%2c-center-for-information-technology-policy/job?mobile=false&width=1200&height=500&bga=true&needsRedirect=false&jan1offset=-300&jun1offset=-240

Multiple Intelligences, and Superintelligence

Superintelligent machines have long been a trope in science fiction. Recent advances in AI have made them a topic for nonfiction debate, and even planning. And that makes sense. Although the Singularity is not imminent–you can go ahead and buy that economy-size container of yogurt–it seems to me almost certain that machine intelligence will surpass ours eventually, and quite possibly within our lifetimes.

Arguments to the contrary don’t seem convincing. Kevin Kelly’s recent essay in Backchannel is a good example. His subtitle, “The AI Cargo Cult: The Myth of a Superhuman AI” implies that AI of superhuman intelligence will not occur. His argument centers on five “myths”:

  1. Artificial intelligence is already getting smarter than us, at an exponential rate.
  2. We’ll make AIs into a general purpose intelligence, like our own.
  3. We can make human intelligence in silicon.
  4. Intelligence can be expanded without limit.
  5. Once we have exploding superintelligence it can solve most of our problems.

He rebuts these “myths” with five “heresies” :

  1. Intelligence is not a single dimension, so “smarter than humans” is a meaningless concept.
  2. Humans do not have general purpose minds, and neither will AIs.
  3. Emulation of human thinking in other media will be constrained by cost.
  4. Dimensions of intelligence are not infinite.
  5. Intelligences are only one factor in progress.

This is all fine, but notice that even if all five “myths” are false, and all five “heresies” are true, superintelligence could still exist.  For example, superintelligence need not be “like our own” or “human” or “without limit”–it only needs to outperform us.

The most interesting item on Kelly’s lists is heresy #1, that intelligence is not a single dimension, so “smarter than humans” is a meaningless concept. This is really two claims, so let’s consider them one at a time.

First, is intelligence a single dimension, or are there different aspects or skills involved in intelligence?  This is an old debate in human psychology, on which I don’t have an informed opinion. But whatever the nature and mechanisms of human intelligence might be, we shouldn’t assume that machine intelligence will be the same.

So far, AI practice has mostly treated intelligence as multi-dimensional, building distinct solutions to different cognitive challenges. Perhaps this is fundamental, and machine intelligence will always be a bundle of different capabilities. Or perhaps there will be a future unification of some sort, to create a single capability that can outperform people on all or nearly all cognitive tasks. At this point it seems like an open question whether machine intelligence is inherently multi-dimensional.

The second part of Kelly’s claim is that, assuming intelligence is multi-dimensional, “smarter than humans” is a meaningless concept. This, to put it bluntly, is not correct.

To see why, consider that playing center field in baseball requires multi-dimensional skills: running, throwing, distinguishing balls from strikes, hitting accurately, hitting with power, and so on. Yet every single major league center fielder is vastly better than I am at playing center field, because they dominate me by far in every one of the component skills.

Like playing center field, intelligence may be multi-dimensional, and yet one entity can be more intelligent than another by being superior in every dimension.

What this suggests about the future of machine intelligence is that we may live for quite a while in a state where machines are better than us at some aspects of intelligence and we are better than them at others. Indeed, that is the case now, and has been for years.

If machine intelligence remains multi-dimensional, then machines will surpass our intelligence not at a single point in time, but gradually, and in more and more dimensions of intelligence.