August 18, 2018

What Are Machine Learning Models Hiding?

Machine learning is eating the world. The abundance of training data has helped ML achieve amazing results for object recognition, natural language processing, predictive analytics, and all manner of other tasks. Much of this training data is very sensitive, including personal photos, search queries, location traces, and health-care records.

In a recent series of papers, we uncovered multiple privacy and integrity problems in today’s ML pipelines, especially (1) online services such as Amazon ML and Google Prediction API that create ML models on demand for non-expert users, and (2) federated learning, aka collaborative learning, that lets multiple users create a joint ML model while keeping their data private (imagine millions of smartphones jointly training a predictive keyboard on users’ typed messages).

Our Oakland 2017 paper, which has just received the PET Award for Outstanding Research in Privacy Enhancing Technologies, concretely shows how to perform membership inference, i.e., determine if a certain data record was used to train an ML model.  Membership inference has a long history in privacy research, especially in genetic privacy and generally whenever statistics about individuals are released.  It also has beneficial applications, such as detecting inappropriate uses of personal data.

We focus on classifiers, a popular type of ML models. Apps and online services use classifier models to recognize which objects appear in images, categorize consumers based on their purchase patterns, and other similar tasks.  We show that if a classifier is open to public access – via an online API or indirectly via an app or service that uses it internally – an adversary can query it and tell from its output if a certain record was used during training.  For example, if a classifier based on a patient study is used for predictive health care, membership inference can leak whether or not a certain patient participated in the study. If a (different) classifier categorizes mobile users based on their movement patterns, membership inference can leak which locations were visited by a certain user.

There are several technical reasons why ML models are vulnerable to membership inference, including “overfitting” and “memorization” of the training data, but they are a symptom of a bigger problem. Modern ML models, especially deep neural networks, are massive computation and storage systems with millions of high-precision floating-point parameters. They are typically evaluated solely by their test accuracy, i.e., how well they classify the data that they did not train on.  Yet they can achieve high test accuracy without using all of their capacity.  In addition to asking if a model has learned its task well, we should ask what else has the model learned? What does this “unintended learning” mean for the privacy and integrity of ML models?

Deep networks can learn features that are unrelated – even statistically uncorrelated! – to their assigned task.  For example, here are the features learned by a binary gender classifier trained on the “Labeled Faces in the Wild” dataset.

While the upper layer of this neural network has learned to separate inputs by gender (circles and triangles), the lower layers have also learned to recognize race (red and blue), a property uncorrelated with the task.

Our more recent work on property inference attacks shows that even simple binary classifiers trained for generic tasks – for example, determining if a review is positive or negative or if a face is male or female – internally discover fine-grained features that are much more sensitive. This is especially important in collaborative and federated learning, where the internal parameters of each participant’s model are revealed during training, along with periodic updates to these parameters based on the training data.

We show that a malicious participant in collaborative training can tell if a certain person appears in another participant’s photos, who has written the reviews used by other participants for training, which types of doctors are being reviewed, and other sensitive information. Notably, this leakage of “extra” information about the training data has no visible effect on the model’s test accuracy.

A clever adversary who has access to the ML training software can exploit the unused capacity of ML models for nefarious purposes. In our CCS 2017 paper, we show that a simple modification to the data pre-processing, without changing the training procedure at all, can cause the model to memorize its training data and leak it in response to queries. Consider a binary gender classifier trained in this way.  By submitting special inputs to this classifier and observing whether they are classified as male or female, the adversary can reconstruct the actual images on which the classifier was trained (the top row is the ground truth):

Federated learning, where models are crowd-sourced from hundreds or even millions of users, is an even juicier target. In a recent paper, we show that a single malicious participant in federated learning can completely replace the joint model with another one that has the same accuracy but also incorporates backdoor functionality. For example, it can intentionally misclassify images with certain features or suggest adversary-chosen words to complete certain sentences.

When training ML models, it is not enough to ask if the model has learned its task well.  Creators of ML models must ask what else their models have learned. Are they memorizing and leaking their training data? Are they discovering privacy-violating features that have nothing to do with their learning tasks? Are they hiding backdoor functionality? We need least-privilege ML models that learn only what they need for their task – and nothing more.

This post is based on joint research with Eugene Bagdasaryan, Luca Melis, Reza Shokri, Congzheng Song, Emiliano de Cristofaro, Deborah Estrin, Yiqing Hua, Thomas Ristenpart, Marco Stronati, and Andreas Veit.

Thanks to Arvind Narayanan for feedback on a draft of this post.

Against privacy defeatism: why browsers can still stop fingerprinting

In this post I’ll discuss how a landmark piece of privacy research was widely misinterpreted, how this misinterpretation deterred the development of privacy technologies rather than spurring it, how a recent paper set the record straight, and what we can learn from all this.

The research in question is about browser fingerprinting. Because of differences in operating systems, browser versions, fonts, plugins, and at least a dozen other factors, different users’ web browsers tend to look different. This can be exploited by websites and third-party trackers to create so-called fingerprints. These fingerprints are much more effective than cookies for tracking users across websites: they leave no trace on the device and cannot easily be reset by the user.

The question is simply this: how effective is browser fingerprinting? That is, how unique is the typical user’s device fingerprint? The answer has big implications for online privacy. But studying this question scientifically is hard: while there are many tracking companies that have enormous databases of fingerprints, they don’t share them with researchers.

The first large-scale experiment on fingerprinting, called Panopticlick, was done by the Electronic Frontier Foundation starting in 2009. Hundreds of thousands of volunteers visited panopticlick.eff.org and agreed to have their browser fingerprinted for research. What the EFF found was remarkable at the time: 83% of participants had a fingerprint that was unique in the sample. Among those with Flash or Java enabled, fingerprints were even more likely to be unique: 94%. A project by researchers at INRIA in France with an even larger sample found broadly similar results. Meanwhile, researchers, including us, found that an ever larger number of browser features — Canvas, Battery, Audio, and WebRTC — were being abused by tracking companies for fingerprinting.

The conclusion was clear: fingerprinting is devastatingly effective. It would be futile for web browsers to try to limit fingerprintability by exposing less information to scripts: there were too many leaks to plug; too many fingerprinting vectors. The implications were profound. Browser vendors concluded that they wouldn’t be able to stop third-party tracking, and so privacy protection was left up to extensions. [1] These extensions didn’t aim to limit fingerprintability either. Instead, most of them worked in a convoluted way: by manually compiling block lists of thousands of third-party tracking scripts, constantly playing catch up as new players entered the tracking game.

But here’s the twist: a team at INRIA (including some of the same researchers responsible for the earlier study) managed to partner with a major French website and test the website’s visitors for fingerprintability. The findings were published a few months ago, and this time the results were quite different: only a third of users had unique fingerprints (compared to 83% and 94% earlier), despite the researchers’ use of a comprehensive set of 17 fingerprinting attributes. For mobile users the number was even lower: less than a fifth. There were two reasons for the differences: a larger sample in the new study, and because self-selection of participants appears to have introduced a bias in the earlier studies. There’s more: since the web is evolving away from plugins such as Flash and Java, we should expect fingerprintability to drop even further. A close look at the paper’s findings suggests that even simple interventions by browsers to limit the highest-entropy attributes would greatly improve the ability of users to hide in the crowd.

Apple recently announced that Safari would try and limit fingerprinting, and it’s likely that the recent paper had an influence in this decision. Notably, a minority of web privacy experts never subscribed to the view that fingerprinting protection is futile, and W3C, the main web standards body, has long provided guidance for developers of new standards on how to minimize fingerprintability. It’s still not too late. But if we’d known in 2009 what we know today, browsers would have had a big head start in developing and deploying fingerprinting defenses.

Why did the misinterpretation happen in the first place? One easy lesson is that statistics is hard, and non-representative samples can thoroughly skew research conclusions. But there’s another pill that’s harder to swallow: the recent study was able to test users in the wild only because the researchers didn’t ask or notify the users. [2] With Internet experiments, there is a tension between traditional informed consent and validity of findings, and we need new ethical norms to resolve this.

Another lesson is that privacy defenses don’t need to be perfect. Many researchers and engineers think about privacy in all-or-nothing terms: a single mistake can be devastating, and if a defense won’t be perfect, we shouldn’t deploy it at all. That might make sense for some applications such as the Tor browser, but for everyday users of mainstream browsers, the threat model is death by a thousand cuts, and privacy defenses succeed by interfering with the operation of the surveillance economy.

Finally, the fingerprinting-defense-is-futile argument is an example of privacy defeatism. Faced with an onslaught of bad news about privacy, we tend to acquire a form of learned helplessness, and reach the simplistic conclusion that privacy is dying and there’s nothing we can do about it. But this position is not supported by historical evidence: instead, we find that there is a constant re-negotiation of the privacy equilibrium, and while there are always privacy-infringing developments, there are offset from time to time by legal, technological, and social defenses.

Browser fingerprinting remains on the frontlines of the privacy battle today. The GDPR is making things harder for fingerprinters. It’s time for browser vendors to also get serious in cracking down on this sneaky practice. 

Thanks to Günes Acar and Steve Englehardt for comments on a draft.

[1] One notable exception is the Tor browser, but it comes at a serious cost to performance and breakage of features on websites. Another is Brave, which has a self-selected userbase presumably willing to accept some breakage in exchange for privacy.

[2] The researchers limited their experiment to users who had previously consented to the site’s generic cookie notice; they did not specifically inform users about their study.

 

 

Fast Web-based Attacks to Discover and Control IoT Devices

By Gunes Acar, Danny Y. Huang, Frank Li, Arvind Narayanan, and Nick Feamster

Two web-based attacks against IoT devices made the rounds this week. Researchers Craig Young and Brannon Dorsey showed that a well known attack technique called “DNS rebinding” can be used to control your smart thermostat, detect your home address or extract unique identifiers from your IoT devices.

For this type of attack to work, a user needs to visit a web page that contains malicious script and remain on the page while the attack proceeds. The attack simply fails if the user navigates away before the attack completes. According to the demo videos, each of these attacks takes longer than a minute to finish, assuming the attacker already knew the IP address of the targeted IoT device.

According to a study by Chartbeat, however, 55% of typical web users spent fewer than 15 seconds actively on a page. Does it mean that most web users are immune to these attacks?

In a paper to be presented at ACM SIGCOMM 2018 Workshop on IoT Security and Privacy, we developed a much faster version of this attack that takes only around ten seconds to discover and attack local IoT devices. Furthermore, our version assumes that the attacker has no prior knowledge of the targeted IoT device’s IP address. Check out our demo video below.

[Read more…]