October 23, 2017

SESTA May Encourage the Adoption of Broken Automated Filtering Technologies

The Senate is currently considering the Stop Enabling Sex Traffickers Act (SESTA, S. 1693), with a scheduled hearing tomorrow. In brief, the proposed legislation threatens to roll back aspects of Section 230 of the Communications Decency Act (CDA), which relieve content providers, or so-called “intermediaries” (e.g., Google, Facebook, Twitter) of liability for the content that is hosted on their platforms. Section 230 protects these platforms from prosecution in federal civil or state courts for the activities of their customers.

One of the corollaries of SESTA is that, with increased liability, content providers might feel compelled to rely more on automated classification filters and algorithms to detect and eliminate unwanted content on the Internet. Having spent more than ten years on developing these types of classifiers to detect “unwanted traffic” ranging from spam to phishing attacks to botnets, I am deeply familiar with the potential—and limitations—of automated filtering algorithms for identifying such content. Existing algorithms can be effective for detecting and predicting certain types of “unwanted traffic”—most notably, attack traffic—but the current approaches to detecting unwanted speech fall far short of being able to reliably detect illegal speech.

Content filters are inaccurate. Notably, the oft-referenced technologies for detecting illegal speech or imagery (e.g., PhotoDNA, EchoPrint), rely on matching content that is posted online against a corpus of content that is known to contain illegal content (e.g., text, audio, imagery). Unfortunately, because these technologies rely on analyzing the content of the posted material. the potential for false positives (i.e., mistakenly identifying innocuous content as illegal) and false negatives (i.e., failing to detect illegal content entirely) are both possible. The network security community has been through this scenario before, in the context of spam filtering: years ago, spam filters would analyze the text of messages to determine whether a particular message was legitimate or spam; it wasn’t long before spammers developed tens of thousands of ways to spell “Rolex” and “Viagra” to evade these filters. They also came up with other creative ways to evade them, by stuffing messages with Shakespeare, and delivering their messages through a variety of formats, ranging from compressed audio to images to spreadsheets.

In short, content-based filters have largely failed to keep up with the agility of spammers.  Evaluation of EchoPrint, for example, suggests that false positive rates are far too high to be used in an automated filtering context. Depending on the length of the file and the type of encoding, error rates are around 1–2 %, where an error could either be a false negative or a false positive. On the other hand, when we were working on spam filters, our discussions with online email service providers suggested that any spam filtering algorithm whose false positive rate exceeded 0.01% would be far too high to avoid raising free speech questions and concerns. In other words, some of the existing automated fingerprinting services that providers might rely on as a result of SESTA might have false positive rates that are many orders of magnitude greater than might otherwise be considered acceptable. We have written extensively about the limitations of these automated filters in the context of copyright.

Content filters cannot identify context. Similarly, today, users who post content online have many tools at their disposal to evade the relatively brittle content-based filters. Detecting unwanted or illegal content on intermediary platforms is even more challenging. Instead of simply classifying unwanted email traffic such as spam (which are typically readily apparent, as they have links to advertisers, phishing sites, and so forth), the challenge on intermediary platforms entails detecting copyright, hate speech, terrorist speech, sex trafficking, and so forth. Yet, simply detecting the presence of something that matches content in a database cannot evaluate considerations fair use, parody, or coercion. Relying on automated content filters will not only produce mistakes in classifying content, but also these filters have no hope of classifying context.

A possible step forward: Classifiers based on network traffic and sending pattens. About ten years ago, we realized the failure of content filters and began exploring how network traffic patterns might produce a stronger signal for illicit activity. We observed that while it was fairly easy for a spammer to change the content of a message it was potentially much more costly for a spammer to change sending patterns, such as email volumes and where messages were originating from and going to. We devised classifiers for email traffic that relied on “network-level features” that now form the basis of many online spam filters. I think there are several grand challenges that lie ahead in determining whether similar approaches could be used to identify unwanted or illegal posts on intermediary content platforms. For example, it might be the case that certain types of illegal speech are characterized by high volumes of re-tweets, short reply times, many instances of repeated messages, or some other feature that is characteristic of the traffic or the accounts that post those messages.

Unfortunately, the reality is that we are far from having automated filtering technology that can reliably detect a wide range of illegal content. Determining how and whether various types of illegal content could be identified remains an open research problem. To suggest that “Any start-up has access to low cost and virtually unlimited computing power and to advanced analytics, artificial intelligence and filtering software.”—a comment that was made in a recent letter to Congress on the question of SESTA—vastly overstates the current state of the art. The bottom line is that whether we can design automated filters to detect illegal content on today’s online platforms is an open research question. A potentially unwanted side effect of SESTA is that intermediaries might feel compelled to deploy these imperfect technologies on their platforms as a result of this law, for fear of liability—thus potentially resulting in over-blocking of legal, legitimate content while failing to effectively deter or prevent the illegal speech that can easily evade today’s content-based filters.

Summary: Automated filters are not “there yet”. Automated filters are often incapable of simply matching content against known offending content, typically because content-based filters are so easily evaded. An interesting question concerns whether other “signals”, such as network traffic and posting patterns, or other characteristics of user accounts (e.g., age of account, number and characteristics of followers) might help us identify illegal content of various types. But, much research is needed before we can comfortably say that these algorithms are even remotely effective at curbing illegal speech. And, even as we work to improve the effectiveness of these automated fingerprinting and filtering technologies, they will likely at best remain an aid that intermediaries might opt to use; I cannot foresee false positive rates ever reaching zero; by no means should we require intermediaries to use these algorithms and technologies in hopes that doing so will eliminate all illegal speech. Doing so would undoubtedly also curb legal and legitimate speech, even as we work to improve them.

Getting serious about research ethics: AI and machine learning

[This blog post is a continuation of our series about research ethics in computer science.]

The widespread deployment of artificial intelligence and specifically machine learning algorithms causes concern for some fundamental values in society, such as employment, privacy, and discrimination. While these algorithms promise to optimize social and economic processes, research in this area has exposed some major deficiencies in the social consequences of their operation. Some consequences may be invisible or intangible, such as erecting computational barriers to social mobility through a variety of unintended biases, while others may be directly life threatening. At the CITP’s recent conference on computer science ethics, Joanna Bryson, Barbara Engelhardt, and Matt Salganik discussed how their research led them to work on machine learning ethics.

Joanna Bryson has made a career researching artificial intelligence, machine learning, and understanding their consequences on society. She has found that people tend to identify with the perceived consciousness of artificially intelligent artifacts, such as robots, which then complicates meaningful conversations about the ethics of their development and use. By equating artificially intelligent systems to humans or animals, people deduce its moral status and can ignore their engineered nature.

While the cognitive power of AI systems can be impressive, Bryson argues they do not equate to humans and should not be regulated as such. On the one hand, she demonstrates the power of an AI system to replicate societal biases in a recent paper (co-authored with CITP’s Aylin Caliskan and Arvind Narayanan) by letting systems trained on a corpus of text from the World Wide Web learn the implicit biases around the gender of certain professions. On the other hand, she argues that machines cannot ‘suffer’ in the same way as humans do, which is one of the main deterrents for humans in current legal systems. Bryson proposes we understand both AI and ethics as human-made artifacts. It is therefore appropriate to rely ethics – rather than science – to determine the moral status of artificially intelligent systems.

Barbara Engelhardt’s work focuses on machine learning in computational biology, specifically genomics and medicine. Her main area of concern is the reliance on recommendation systems, such as we encounter on Amazon and Netflix, to make decisions in other domains such as healthcare, financial planning, and career decisions. These machine learning systems rely on data as well as social networks to make inferences.

Engelhardt describes examples where using patient records to inform medical decisions can lead to erroneous recommendation systems for diagnosis as well as harmful medical interventions. For example, the symptoms of heart disease differ substantially between men and women, and so do their appropriate treatments. Most data collected about this condition was from men, leaving a blind spot for the diagnosis of heart disease in women. Bias, in this case, is useful and should be maintained for correct medical interventions. In another example, however, data was collected from a variety of hospitals in somewhat segregated poor and wealthy areas. The data appear to show that cancers in children from hispanic and caucasian races develop differently. However, inferences based on this data fail to take into account the biasing effect of economic status in determining at which stage of symptoms different families decide seek medical help. In turn, this determines the stage of development at which the oncological data is collected. The recommendation system with this type of bias confuses race with economic barriers to medical help, which will lead to harmful diagnosis and treatments.

Matt Salganik proposes that the machine learning community draws some lessons from ethics procedures in social science. Machine learning is a powerful tool the can be used responsibly or inappropriately. He proposes that it can be the task of ethics to guide researchers, engineers, and developers to think carefully about the consequences of their artificially intelligent inventions. To this end, Salganik proposes a hope-based and principle-based approach to research ethics in machine learning. This is opposed to a fear-based and rule-based approach in social science, or the more ad hoc ethics culture that we encounter in data and computer science. For example, machine learning ethics should include pre-research review through forms that are reviewed by third parties to avoid groupthink and encourage researchers’ reflexivity. Given the fast pace of development, though, the field should avoid a compliance mentality typically found at institutional review boards of univeristies. Any rules to be complied with are unlikely to stand the test of time in the fast-moving world of machine learning, which would result in burdensome and uninformed ethics scrutiny. Salganik develops these themes in his new book Bit By Bit: Social Research in the Digital Age, which has an entire chapter about ethics.”

See a video of the panel here.

Blockchains and voting

I’ve been asked about a number of ideas lately involving voting systems and blockchains. This blog piece talks about all the security properties that a voting system needs to have, where blockchains help, and where they don’t.

Let’s start off a decade ago, when Daniel Sandler and I first wrote a paper saying blockchains would be useful for voting systems. We observed that voting machines running on modern computers have overwhelming amounts of CPU and storage, so let’s use it in a serious way. Let’s place a copy of every vote on every machine and let’s use timeline entanglement (Maniatis and Baker 2002), so every machine’s history is protected by hashes stored on other machines. We even built a prototype voting system called VoteBox that used all of this, and many of the same ideas now appear in a design called STAR-Vote, which we hope could someday be used by real voters in real elections.

What is a blockchain good for? Fundamentally, it’s about having a tamper-evident history of events. In the context of a voting system, this means that a blockchain is a great place to store ballots to protect their integrity. STAR-Vote and many other “end-to-end” voting systems have a concept of a “public bulletin board” where encrypted votes go, and a blockchain is the obvious way to implement the public bulletin board. Every STAR-Vote voter leaves the polling place with a “receipt” which is really just the hash of their encrypted ballot, which in turn has the hash of the previous ballot. In other words, STAR-Vote voters all leave the polling place with a pointer into the blockchain which can be independently verified.

So great, blockchain for the win, right? Not so fast. Turns out, voting systems need many additional security properties before they can be meaningfully secure. Here’s a simplified list with some typical vocabulary used for these security properties.

  • Cast as intended. A voter is looking at a computer of some sort and indicates “Alice for President!”, and our computer handily indicates this with a checkbox or some highlighting, but evil malware inside the computer can silently record the vote as “Bob for President!” instead. Any voting system needs a mechanism to defeat malware that might try to compromise the integrity of the vote. One common approach is to have printed paper ballots (and/or hand-marked paper ballots) which can be statistically compared to the electronic ballots. Another approach is to have a process whereby the machine can be “challenged” to prove that it correctly encrypted the ballot (Benaloh 2006, Benaloh 2007).
  • Vote privacy. It’s important that there is no way to identify a particular voter with how they voted. To understand the importance of vote privacy, consider a hypothetical alternate where all votes were published, in the newspaper, with the voter’s name next to each vote. At that point, you could trivially bribe or coerce people to vote in a particular way. The modern secret ballot, also called the Australian ballot, ensures that votes are secret, with various measures taken to make it hard or impossible for voters to violate this secrecy. When you wish to maintain a privacy property in the face of voting computers, that means you have to prevent the computer from retaining state (i.e., keeping a private list of the plaintext votes in the order cast) and you have to ensure that the ciphertext votes, published to the blockchain, aren’t quietly leaking information about their plaintext through various subliminal channels.
  • Counted as cast. If we have voters taking home a receipt of some sort that identifies their ciphertext vote in the blockchain, then they also want to have some sort of cryptographic proof that the final vote tally includes their specific vote. This turns out to be a straightforward application of homomorphic cryptographic primitives and/or mixnets.

If you look at these three properties, you’ll notice that the blockchain doesn’t do much to help with the first two, although they are very useful for the third.

Achieving a “cast as intended” property requires a variety of mechanisms ranging from paper ballots and spot challenges of machines. The blockchain protects the integrity of the recorded vote, but has nothing to say about its fidelity to the intent of the voter.

Achieving a “vote privacy” property requires locking down the software on the voting platform, and for that matter locking down the entire computer. And how can that lock-down property be verified? We need strong attestations that can be independently verified. We also need to ensure that the user cannot be spoofed into running a fake voting application. We can almost imagine how we can achieve this in the context of electronic voting machines which are used exclusively for voting purposes. We can centrally deploy a cryptographic key infrastructure and place physical controls over the motion of the machines. But for mobile phones and personal computers? We simply don’t have the infrastructure in place today, and we probably won’t have it for years to come.

To make matters worse, a commonly expressed desire is to vote from home. It’s convenient! It increases turnout! (Maybe.) Well, it also makes it exceptionally easy for your spouse or your boss or your neighbor to watch over your shoulder and “help” you vote the way they want you to vote.

Blockchains do turn out to be incredibly helpful for verifying a “counted as cast” property, because they force everybody to agree on the exact set of ballots being tabulated. If an election official needs to disqualify a ballot for whatever reason, that fact needs to be public and everybody needs to know that a specific ballot, right there in the blockchain, needs to be discounted, otherwise the cryptographic math won’t add up.

Wrapping up, it’s easy to see how blockchains are an exceptionally useful primitive that can help build voting systems, with particular value in verifying that the final tally is consistent with the cast ballot records. However, a good voting system needs to satisfy many additional properties which a blockchain cannot provide. While there’s an intellectual seduction to pretend that casting votes is no different than moving coins around on a blockchain, the reality of the problem is a good bit more complicated.