May 18, 2022

How the National AI Research Resource can steward the datasets it hosts

Last week I participated on a panel about the National AI Research Resource (NAIRR), a proposed computing and data resource for academic AI researchers. The NAIRR’s goal is to subsidize the spiraling costs of many types of AI research that have put them out of reach of most academic groups.

My comments on the panel were based on a recent study by researchers Kenny Peng, Arunesh Mathur, and me (NeurIPS ‘21) on the potential harms of AI. We looked at almost 1,000 research papers to analyze how they used datasets, which are the engine of AI.

Let me briefly mention just two of the many things we found, and then I’ll present some ideas for NAIRR based on our findings. First, we found that “derived datasets” are extremely common. For example, there’s a popular facial recognition dataset called Labeled Faces in the Wild, and there are at least 20 new datasets that incorporate the original data and extend it in some way. One of them adds race and gender annotations. This means that a dataset may enable new harms over time. For example, once you have race annotations, you can use it to build a model that tracks the movement of ethnic minorities through surveillance cameras, which some governments seem to be doing.

We also found that dataset creators are aware of their potential for misuse, so they often have licenses restricting their use for research and not for commercial purposes. Unfortunately, we found evidence that many companies simply get around this by downloading a model pre-trained on that dataset (in a research context) and using that model in commercial products.

Stepping back, the main takeaway from our paper is that dataset creators can sometimes — but not always — anticipate the ways in which a dataset might be used or misused in harmful ways. So we advocate for what we call dataset stewarding, which is a governance process that lasts throughout the lifecycle of a dataset. Note that some prominent datasets see active use for decades.

I think NAIRR is ideally positioned to be the steward of the datasets that it hosts, and perform a vital governance role over datasets and, in turn, over AI research. Here are a few specific things NAIRR could do, starting with the most lightweight ones.

1. NAIRR should support a communication channel between a dataset creator and the researchers who use that dataset. For example, if ethical problems — or even scientific problems — are uncovered in a dataset, it should be possible to notify users about it. As trivial as this sounds, it is not always the case today. Prominent datasets have been retracted over ethical concerns without a way to notify the people who had downloaded it.

2. NAIRR should standardize dataset citation practices, for example, by providing Digital Object Identifiers (DOIs) for datasets. We found that citation practices are chaotic, and there is currently no good way to find all the papers that use a dataset to check for misuse.

3. NAIRR could publish standardized dataset licenses. Dataset creators aren’t legal experts, and most of the licenses don’t accomplish what dataset creators want them to accomplish, enabling misuse.

4. NAIRR could require some analog of broader impact statements as part of an application for data or compute resources. Writing a broader impact statement could encourage ethical reflection by the authors. (A recent study found evidence that the NeurIPS broader impact requirement did result in authors reflecting on the societal consequences of their technical work.) Such reflection is valuable even if the statements are not actually used for decision making about who is approved. 

5. NAIRR could require some sort of ethical review of proposals. This goes beyond broader impact statements by making successful review a condition of acceptance. One promising model is the Ethics and Society Review instituted at Stanford. Most ethical issues that arise in AI research fall outside the scope of Institutional Review Boards (IRBs), so even a lightweight ethical review process could help prevent obvious-in-hindsight ethical lapses.

6. If researchers want to use a dataset to build and release a derivative dataset or pretrained model, then there should be an additional layer of scrutiny, because these involve essentially republishing the dataset. In our research, we found that this is the start of an ethical slippery slope, because data and models can be recombined in various ways and the intent of the original dataset can be lost.

7. There should be a way for people to report to NAIRR that some ethics violation is going on. The current model, for lack of anything better, is vigilante justice: journalists, advocates, or researchers sometimes identify ethical issues in datasets, and if the resulting outcry is loud enough, dataset creators feel compelled to retract or modify them. 

8. NAIRR could effectively partner with other entities that have emerged as ethical regulators. For example, conference program committees have started to incorporate ethics review. If NAIRR made it easy for peer reviewers to check the policies for any given data or compute resource, that would let them verify that a submitted paper is compliant with those policies.

There is no single predominant model for ethical review of AI research analogous to the IRB model for biomedical research. It is unlikely that one will emerge in the foreseeable future. Instead, a patchwork is taking shape. The NAIRR is set up to be a central player in AI research in the United States and, as such, bears responsibility for ensuring that the research that it supports is aligned with societal values.

——–

I’m grateful to the NAIRR task force for inviting me and to my fellow panelists and moderators for a stimulating discussion.  I’m also grateful to Sayash Kapoor and Mihir Kshirsagar, with whom I previously submitted a comment on this topic to the relevant federal agencies, and to Solon Barocas for helpful discussions.

A final note: the aims of the NAIRR have themselves been contested and are not self-evidently good. However, my comments (and the panel overall) assumed that the NAIRR will be implemented largely as currently conceived, and focused on harm mitigation.

CITP Case Study on Regulating Facial Recognition Technology in Canada

Canada, like many jurisdictions in the United States, is grappling with the growing usage of facial recognition technology in the private and public sectors. This technology is being deployed at a rapid pace in airports, retail stores, social media platforms, and by law enforcement – with little oversight from the government. 

To help address this challenge, I organized a tech policy case study on the regulation of facial recognition technology with Canadian members of parliament – The Honorable Greg Fergus and Matthew Green. Both sit on the House of Commons’ Standing Committee on Access to Information, Privacy, and Ethics (ETHI) Committee and I served as a legislative aide to them through the Parliamentary Internship Programme before joining CITP. Our goal for the session was to put policymakers in conversation with subject matter experts. 

The core problem is that there is lack of accountability in the use of facial recognition technology that excarbates historical forms of discrimination and puts marginalized communities at risk for a wide range of harms. For instance, a recent story describes the fate of three black men who were wrongfully arrested because of being misidentified by facial recognition software. As the Canadian Civil Liberties Association argues, the police’s use of facial recognition technology, notably provided by the New York-based company, Clearview AI, “points to a larger crisis in police accountability when acquiring and using emerging surveillance tools.

A number of academics and researchers – such as DAIR Instititute’s Timnit Gebru and the Algorithmic Justice League’s Joy  Buolamwini, who documented the missclassification of darker-skinned women in a recent paper – are bringing attention to the discriminatory algorithms associated with facial recognition that have put racialized people, women, and members of the LGBTIQ community, at greater risk of false identification.  

Meanwhile, Canadian officials are beginning to tackle the real world consequences of the use of facial recognition. A year ago, the Office of the Privacy Commissioner found that Clearview AI, had scraped billions of images of people from from the internet in what “represented mass surveillance and was a clear violation of the privacy rights of Canadians.” 

Following that investigation, Clearview AI stopped providing services to the Canadian market, including the Royal Canadian Mounted Police. In light of these findings and the absence of dedicated legislation, the ETHI Committee began studying the uses of facial recognition technology in May 2021, and has recently resumed this work by focusing on the use by various levels of government in Canada, law enforcement agencies, and private corporations. 

The CITP case study session on March 24, began with a presentation by Angelina Wang, a graduate affiliate of CITP, who provided a technical overview where she explained the different functions and harms associated with this technology. Following Wang’s presentation, I provided a regulatory overview of how U.S. lawmakers have addressed facial recognition by noting the different legislative strategies deployed for law enforcement, private, and public sector uses. We then had a substantive, free-flowing discussion with CITP researchers and the policymakers about the challenges and opportunities for different regulatory strategies. 

Following CITP’s case study session, Wang and Dr. Elizabeth Anne Watkins, a CITP Fellow, were invited to testify before the ETHI committee in an April 4 hearing. Wang discussed the different tasks facial recognition technology can and cannot perform, how the models are created, why they are susceptible to adversarial attacks, and the ethical implications behind the creation of this technology. Dr. Watkins’ testimony provided an overview of the privacy, security, and safety concerns related to the private industry’s use of facial verification on workers as informed by her research.  The committee is expected to report its findings by the end of May 2022. 

We continue to do research on how Canada might regulate facial recognition technology and will publish those analyses in the coming months.

Recommendations for introducing greater safeguards and transparency into CS conference funding

In Part 1 of this piece, I provided evidence of the extent to which some of the world’s top computer science conferences are financially reliant upon some of the world’s most powerful technology companies. In this second part, I lay out a set of recommendations for ways to help ensure that these entanglements of industry and academia don’t grant companies undue influence over the conditions of knowledge creation and exchange.

To be clear, I am not suggesting that conferences stop accepting money from tech companies, nor am I saying there is no place for Big Tech investment in academic research. I am simply advocating for conference organizers to adopt greater safeguards to increase transparency and mitigate the potential agenda-setting effects associated with companies’ funding of and presence in academic spaces.

While I am not claiming that sponsors have any say over which papers are or aren’t published, in the next few paragraphs I will show how agenda-setting can happen in a much more subtle yet pervasive way.

Resurrecting conferences as “trading zones”

Setting the agenda in a given field means determining and prioritizing topics of focus and investment. Research priorities are not neutral or naturally occurring—they are the result of social and political construction. And because a great deal of CS funding comes from tech companies, these priorities are likely to be shaped by what is considered valuable or profitable to those companies.

An example of the tech industry’s agenda-setting power includes the way in which AI/ML research has been conceptualized in narrower terms to prioritize technical work. For instance, despite its valuable contributions to the understanding of priorities inherent in ML research, the Birhane et. al. paper I cited in Part 1 was rejected from publication at the 2021 NeurIPS Conference with a dismissive meta-review, which is just one example of how the ML community has marginalized critical work and elevated technical work. Other examples of corporate agenda-setting in CS include the aforementioned way in which tech companies’ definitions of privacy and security vary from those of consumer advocates, and the way in which the field of human-computer interaction (HCI) often focuses on influencing user behavior rather than stepping back to reflect on necessary systemic changes at the platform level.

In deciding which conferences to fund, and shaping which ideas and work get elevated within those conferences, tech companies contribute to the creation of a prestige hierarchy. This, in turn, influences which kinds of people who self-select into submitting their work to and attending those conferences. Further, the sponsorship perks afford companies a prominent presence at CS conferences through expos and other events. Combined, these factors mold CS conferences into sites of commercially oriented activity.

It is important to make space at top conferences for work that doesn’t necessarily advance commercial innovation. Beyond simply serving as a channel for publishing and broadcasting academic papers, conferences have the potential to serve as sites of critique, activism and advocacy. These seemingly secondary functions of academic gatherings are, in actuality, critical functions that need to be preserved.

In “Engaging, Designing and Making Digital Systems,” Janet Vertesi et al. describe spaces of collaboration between scholarship and design as “trading zones”, where engagements can be corporate, critical, inventive, or focused on inquiry. While corporate work engages from within companies, critical engagement requires the existence of a trading zone in which domain scientists, computer scientists and engineers can meet and engage in dialogue. Vertesi et al. write, “Critical engagements typically embrace intersections between IT research and corporations yet eschew immediate pay-offs for companies or designers.”

Even if sponsoring companies don’t have a direct hand in deciding which work gets published, their presence at academic conferences gives them both insight into ideas and work being shared among attendees, and opportunities to push specific messaging around their brand through advertising and recruitment events. Therefore, instituting sponsorship policies and increasing transparency would help to both curb their potential influence, as well as make clear to conference participants the terms of companies’ financial contributions.

Introducing greater safeguards around conference sponsorship would not be unprecedented; for example, there have been similar efforts in the medical community to curb the influence of pharmaceutical and medical device manufacturing companies on clinical conferences.

Asking accountability conferences to practice what they preach

In particular, tech conferences whose mission is explicitly related to ethics and accountability deserve a higher level of scrutiny for their donor relationships. However, my survey of some of the most prominent conferences in this space found that many of them do not provide a list of donors, nor do they disclose any sponsorship policies on their websites.

That said, some conferences have been reevaluating their fundraising practices after recognizing that certain sponsors’ actions were not aligning with their values. For example, in March 2021, the ACM Conference for Fairness, Accountability, and Transparency (FAccT) suspended its sponsorship relationship with Google in protest of the company’s firing of two of its top Ethical AI researchers, who had been examining biases built into the company’s AI systems.

FAccT committee member Suresh Venkatasubramanian tweeted that the decision to drop Google as a supporter was “in the best interests of the community” while the committee revised its sponsorship policy. Conference sponsorship co-chair Michael Ekstrand told VentureBeat that having Google as a sponsor could impede FAccT’s Strategic Plan. (It should be noted that FAccT still accepted funding from DeepMind, a subsidiary of Google’s parent company Alphabet, for its 2021 conference.)

The conference recently published a new sponsorship policy, acknowledging that “outside contributions raise serious concerns about the independence of the conference and the legitimacy that the conference may confer on sponsors and supporters.” Other conferences, like the ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) and the Association for Computational Linguistics (ACL) Conference have also posted sponsorship and/or conflict of interest policies on their websites.

While it might be expected that ethics and fairness-oriented conferences would have a more robust protocol around which funds they accept, it is in the best interest of all CS conferences to think critically about and mitigate the constraints associated with accepting corporate sponsorship. 

Recommendation of Best Practices

In many instances, accepting corporate sponsorship is a necessary evil that enables valuable work to be done and allows greater access to resources and opportunities like conferences. In the long term, there should be a concerted effort to resurrect computer science conferences as a neutral territory for academic exploration based on what scholars, not corporations, deem to be worthy of pursuit. However, a more immediate solution could be to establish and enforce a series of best practices to ensure greater academic integrity of conferences that do rely on corporate sponsorship. 

Many scholars, like those who signed the Funding Matters petition in 2018, have argued in favor of establishing rigorous criteria and guidelines for corporate sponsorship of research conferences. I have developed a set of recommendations for conferences to serve as a jumping-off point for ensuring greater transparency and accountability in their decision-making process: 

  • Evaluate sponsors through the lens of your organization’s mission and values. Determine which lines you’re not willing to cross.
    • Are there companies whose objectives or outputs run counter to your values? Are there actions you refuse to legitimize or companies whose reputation might significantly compromise the integrity of the conferences they fund? Review your existing sponsors to ensure that none of them are crossing that line, and use it as a threshold for determining whether to accept funding from others in the future.
    • For example, in the sponsorship policy for the ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO), organizers reserve the right to “decline or return any funding should the sponsorship committee decide that the funding source is misaligned with the mission of the initiative and conference.”
  • Be transparent about who is sponsoring your conference, how much they are contributing, and what benefits they receive as a condition of their contributions.
    • While many conferences do list the logos of their sponsors on their websites, it is not often clear how much money those organizations gave and how exactly it was used. To ensure greater transparency, publish a list of sponsors on your website and in other promotional materials and make the details of your call for sponsorship publicly available and easily accessible. 
    • Make sure to make this information public ahead of the conference, so that invited speakers and other attendees can make an informed decision about whether or not they want to participate. (1)
  • Develop rigorous policies to prevent sponsors from influencing the content or speakers of conference events. 
    • Establish a solid gift acceptance policy and thorough gift agreement outlining the kinds of funding you will and will not accept to ensure that your donors’ support is not restricted and does not come with strings attached.
    • For example, the FAccT conference recently published a new statement outlining their practices around sponsorship and financial support, which denies sponsors say over any part of the conference organization or content. In addition, sponsors can only contribute to a general fund, rather than being able to specify how their contributions are used.
  • Encourage open discussion during the conference about the implications of accepting corporate funding and potential alternatives.
    • For example, the ACM Conference on Computer Science and Law has committed to devoting time to a “discussion of practical strategies for and ethical implications of different funding models for both research and conference sponsorship in the nascent ACM Computer Science and Law community.”
  • Make sure the industry in general, or any one company in particular, is not over-represented among sponsors or conference organizers
    • Consider whether certain sponsors might be working to whitewash or silence certain areas of research. What are the interests or intentions of the organization offering you sponsorship funds? What do they hope to gain from this relationship? (2)
    • For example, the EEAMO sponsorship committee commits to “seek[ing] funding from a diverse set of sources which may include academic institutions, charitable organizations, foundations, industry, and government sources.”
  • Consider seeking alternative, industry-independent sources of funding whose interests are less likely to conflict with the subject/mission of your conference.
    • That being said, it is important to bear in mind that, as Phan et al. pointed out in their recent paper, “philanthropic foundation funding from outside Big Tech interests present different and complex considerations for researchers as producers and suppliers of ethics work.” This is why having a diversity of sources is preferable.

In working to reclaim conferences as a space of academic exploration untainted by corporate interests, the field of computer science can help to ensure that their research is better positioned to serve the best interests of the public.

(1) Several speakers backed out of their scheduled appearances at the UCLA Institute for Technology, Law & Policy’s November 2021 Power and Accountability in Tech conference after learning the center had accepted sponsorship money from the Koch Foundation, which has funded attacks on antiracist scholarship.

(2) For example, in 2016, the Computer, Privacy, & Data Protection Conference (CPDP) chose to stop accepting sponsorship funding from Palantir after participants like Aral Balkan pulled out of a panel and described CPDP’s acceptance of the company’s contributions as “privacy-washing”.

Many thanks, once again, to Prof. Arvind Narayanan for his guidance and support.