Freedom to Tinker

Voters in Georgia polling places, 2020, used Ballot-Marking Devices (BMDs), touchscreen computers that print out paper ballots; then voters fed those ballots into Precinct-Count Optical Scan (PCOS) voting machines for tabulation. There were many allegations about hacking of Georgia’s Presidential election. Based on the statewide audit, we can know that the PCOS machines were not cheating (in any way that changed the outcome). But can we know that the touchscreen BMDs were not cheating? And what about next time? There’s a nightmare scenario waiting to happen if Georgia (or other states) continue to use touchscreen BMDs on a large scale.

Part 1: What happened in November 2020

There were many allegations about hacking of Georgia’s voting-machine computers in the November 2020 election—accusations about who owned the company that made the voting machines, accusations about who might have hacked into the computers. An important principle of election integrity is “software independence,” which I’ll paraphrase as saying that we should be able to verify the outcome of the election without having to know who wrote the software in the voting machines.

Indeed, the State of Georgia did a manual audit of all the paper ballots in the November 2020 Presidential election. The audit agreed with the outcome claimed by the optical-scan voting machines. This means,

• The software in Georgia’s PCOS scanners is now irrelevant to the outcome of the 2020 Presidential election in Georgia, which has been confirmed by the audit.
• Georgia’s PCOS scanners were not cheating in the 2020 Presidential election (certainly not by enough to change the outcome), which we know because the hand-count audits closely agreed with the PCOS counts.
• The audit gave election officials the opportunity to notice that several batches of ballots hadn’t even been counted the first time; properly counting those ballots changed the vote totals but not the outcome. I’ll discuss that in a future post.

Suppose the polling-place optical scanners had been hacked (enough to change the outcome). Then this would have been detected in the audit, and (in principle) Georgia would have been able to recover by doing a full recount.† That’s what we mean when we say optical-scan voting machines have “strong software independence”—you can obtain a trustworthy result even if you’re not sure about the software in the machine on election day.

If Georgia had still been using the paperless touchscreen DRE voting machines that they used from 2003 to 2019, then there would have been no paper ballots to recount, and no way to disprove the allegations that the election was hacked. That would have been a nightmare scenario. I’ll bet that Secretary of State Raffensperger now appreciates why the Federal Court forced him to stop using those DRE machines (Curling v. Raffensperger, Case 1:17-cv-02989-AT Document 579).

But optical scanners are not the only voting machines in Georgia’s polling places. Every in-person Georgia voter uses two machines: first, voters select candidates on a touch-screen ballot-marking device (BMD) that prints out a ballot paper; then, they feed that ballot paper into a precinct-count optical scanner (PCOS). The software independence of BMDs is much more problematic.

The audit confirmed that the PCOS was not cheating. How do we know that the BMD was not cheating, printing different votes onto the ballot paper than what the voter selected on the touch screen? This is a much more difficult question, and it can’t be answered by any audit or recount of the ballot papers.

You might think, “the voter would notice if the ballot paper differs from what they indicated on the touch screen.” But two different scientific studies have shown that most voters don’t notice. Only about 7% of voters speak up if a touchscreen BMD fraudulently prints a wrong vote. And that’s just one estimate from one study—it might actually be overoptimistic.***

Biden got about 50.125% of the votes in Georgia, and Trump got 49.875%. Suppose, hypothetically, that 50.125% of the voters chose Trump, but (hypothetically) hacked BMDs were changing votes on 0.25% of the ballots, in favor of Biden. Then the result we’d see would be Biden 50.125%, and the recount would confirm that—because that’s what’s printed on the paper.

In this scenario, if 7% (1 out of 15) of voters carefully review their paper ballot, and 0.25% (1 out of 400) of paper ballots had votes for Biden when the voter had really chosen Trump, then we might expect 1 out of 6000 (15×400) voters to complain to the pollworkers. And the pollworkers would supposedly tell those voters, “no problem, don’t put that ballot into the PCOS, we’ll void that for you and you can mark a fresh ballot.” But all those other voters who didn’t carefully check the printout would still be voting for a candidate they didn’t intend to, and the hack would be successful.

You might think (in this hypothetical scenario), “at least some voters caught the BMDs cheating”. But even if a voter catches the machine cheating, so what? Election officials can’t void an entire election, or “correct” the vote totals, based on the say-so of 0.017% (that is, 1/6000) of the voters.

Did the touchscreen BMDs cheat in the Georgia 2020 Presidential Election? We can guess that they did not cheat this time, and here’s a weak basis for that guess: If the BMDs had been shifting enough votes from Trump to Biden to make a difference, then at least 0.017% of voters would have noticed. There were 5 million votes cast, so that’s about 83 voters statewide**. If those voters complained, then presumably the local news media would have reported contemporaneous reports of such “BMD vote flipping.” But we didn’t hear any such reports.**** So probably the BMDs weren’t flipping any votes.

That’s a pretty weak basis to assert that the BMDs weren’t cheating. But it could be a lot worse . . .

Part 2: The nightmare scenario just waiting to happen next time.

But what about the next election? Suppose in Georgia’s 2022 Senate election between Raphael Warnock and his Republican challenger (whoever that will be), one of those candidates wins with 50.125% of the vote. And suppose 100 voters statewide claim that the BMDs flipped their vote. What should Secretary of State Raffensperger do? He cannot change the election results based on the say-so of 100 voters—those voters might be mistaken (or lying) about what they indicated on the touch screen. He cannot fix it by a recount, because (if the BMDs were really cheating) the paper ballots are fraudulent. He will be in a bind, and there will be no way out. And no way out for the people of Georgia, either.

You might argue, “More than 7% of voters would notice that their paper ballot was incorrectly marked.” Even if that were true (there’s no evidence for it), it just means 200 or 300 voters statewide (1 or 2 per county) would have noticed, instead of just 83. The problem is the same.

The solution is simple.  Voters should mark their optical-scan bubble ballots with a pen.  That way, you know the recount is counting the ballots that the voter actually marked. Touchscreen BMDs (which also have audio interfaces for blind voters) should be reserved for those voters with disabilities who cannot mark a paper ballot by hand.

Georgia should continue using their PCOS (optical scan) voting machines, which will readily count hand-marked optical-scan “bubble” ballots. No major investment in new equipment is needed. This change can easily be implemented before the next election.

And other states and counties that are considering BMDs-for-all-voters—some counties in Pennsylvania and New Jersey have bought those, New York is considering them—should consider the nightmare scenario, and stick with hand-marked paper ballots.

Everything I’ve described here is consistent with the peer-reviewed scientific paper,  Ballot-Marking Devices Cannot Assure the Will of the Voters, by Andrew W. Appel, Richard A. DeMillo, and Philip B. Stark, in Election Law Journal, vol. 19 no. 3, pp. 432-450, September 2020. [non-paywall version here]

† Georgia’s law doesn’t actually say what’s required if the audit detects a problem. The law doesn’t specify that audit results are binding on official results. This year that didn’t matter, because the audit agreed with the official outcome.

*Georgia’s audit was done by examining the ballots with human eyes. Later, at the request of the Trump campaign, Georgia also did a recount using their central-count optical scanners. If those optical scanners had been hacked to cheat consistently with (hypothetically) cheating precinct-count optical scanners, then the machine recount wouldn’t catch the fraud. For that reason, a hand-count is more effective protection than a machine recount. In any case, all three counts (the polling-place count using PCOS, the audit, and the machine recount) showed a Biden victory, although their actual numbers of votes differed.

**Actually, this year a large proportion of Georgians voted by mail, on hand-marked paper ballots, so they didn’t use BMDs at all. Those votes are safe from BMD hacks. But it doesn’t change the “83 voters statewide” result of my analysis.

***That statistic (“7% of voters will notice if the BMD prints the wrong candidate on their ballot”) comes from a single study in Michigan. Here’s why it might be overoptimistic, as applied to this voting machine and these voters. First, look at the BMD ballot and how hard it is to read.***** In November, one observer watched a constant stream of voters during about 20 minutes in Cobb County: they voted without a glance at their paper ballots, but then they told the poll workers that they had checked them. It is just too much trouble to try to read and check them.  In the January 2021 Senate runoffs, another observer saw that only 6 of 46 voters even glanced at the paper—which is not the same as checking it carefully.

****We would like to think “there was no local news reporting of BMD-flipped votes” means that “BMDs didn’t flip votes”. But so much of Georgia is quite rural with very little local reporting, and certainly without the experience to know how to even report something like that. And (in other elections) it often happens that there are verified stories of discrepancies months after the election that never made it to any newspaper.

*****I mean, really! not easy to decode the paper printout. In the Senate race, this is what the ballot says:

For United States Senate (Loeffler) -
Special (Vote for One) (NP)
   Vote for Annette Davis Jackson
     (Rep)

Is that a vote for Kelly Loeffler, whose name appears on the first line? Apparently not, I’d guess it’s a vote for Annette Davis Jackson. And what does (NP) mean? And what does (I) mean attached to votes for many other candidates? Certainly (I) does not mean Independent. This ballot is a masterpiece of bad design, and it’s no wonder that real-life voters are discouraged from looking at it very carefully.

Alicia Wanless, Kristen DeCaires Gall, and Jacob N. Shapiro
Freedom to Tinker: https://freedom-to-tinker.com/

Expanding the knowledge base around influence operations has proven challenging, despite known threats to elections,COVID-related misinformation circulating worldwide, and recent tragic events at the U.S. Capitol fueled in part by political misinformation and conspiracy theories. Credible, replicable evidence from highly sensitive data can be difficult to obtain. The bridge between industry and academia remains riddled with red tape. Intentional and systemic obstructions continue to hinder research on a range of important questions about how influence operations spread, their effects, and the efficacy of countermeasures.

A key part of the challenge lies in the basic motivations for both industry and academic sectors. Tech companies have little incentive to share sensitive data or allocate resources to an effort that does not end in a commercial product, and may even jeopardize their existing one. As a result, cross-platform advances to manage the spread of influence operations have been limited, with the notable exception of successful counter-terrorism data sharing. Researchers who seek to build relationships with specific companies encounter well-documented obstacles in accessing and sharing information, and subtler ones in the time-consuming process of learning how to navigate internal politics. Companies face difficulties recruiting in-house experts from academia as well, as many scholars worry about publication limitations and lack of autonomy when moving to industry.  

The combination of these factors leaves a gap in research on non-commercial issues, at least in relation to the volume of consumer data tech companies ingest. And, unfortunately, studying influence in a purely academic setting presents all the challenges of normal research—inconsistent funding streams, access to quality data, and retaining motivated research staff—as well as the security and confidentiality issues that accompany any mass transfer of data. 

We are left with a lack of high-quality, long-term research on influence operations. 

Fortunately, a way forward exists. The U.S. government long-ago recognized that neither market nor academic incentives can motivate all the research large organizations need. Following World War II, it created a range of independent research institutions. Among them, the Federally Funded Research and Development Centers (FFRDCs) were created explicitly to “provide federal agencies with R&D capabilities that cannot be effectively met by the federal government or the private sector alone”. FFRDCs – IDA, MITRE, and RAND for example – are non-profit organizations funded by Congress for longer periods of time (typically five years) to pursue specific limited research agendas. They are prohibited from competing for other contracts, which enable for-profit firms to share sensitive data with them, even outside of the protections of the national security classification system, and can invest in staffing choices and projects that span short government budget cycles. These organizations bridge the divide between university research centers and for-profit contractors, allowing them to fill critical analytical gaps for important research questions. 

The FFRDC model is far from perfect. Like many government contractors, some have historically had cost inefficiencyand security issues. But by solving a range of execution challenges, they enable important, but not always market-driven research on topics ranging from space exploration, to renewable energy, to cancer treatment. 

Adopting a similar model of a multi-stakeholder research and development center (MRDC) funded by industry and civil society could lay a foundation for collaboration on issues pertaining to misinformation and influence operations by accomplishing five essential tasks: 

• Facilitate funding for long-term projects.
  • Provide infrastructure for developing shared research agendas and a mechanism for executing studies.
  • Create conditions that help build trusted, long-term relationships between sectors.
  • Offer career opportunities for talented researchers wishing to do basic research with practical application.
  • Guard against inappropriate disclosures while enabling high-credibility studies with sensitive information that cannot be made public.

The MDRC model fills a very practical need for flexibility and speed on the front end of addressing immediate problems, such as understanding what, if any, role foreign nations played in the discussions which led up to January 6. Such an organization would provide a bridge for academics and practitioners to come together quickly and collaborate for a sustained period, months or years, on real-world operational issues. A research project at a university can take six months to a year to set up funding and fully staff a project. Furthermore, most universities, and even organizations like the Stanford Internet Observatory fully dedicated to these issues, cannot do “work for hire”. Meaning, if there’s no unique intellectual product or no true research question at hand, their ability to work on a given problem is limited or non-existent. An established contract organization that clearly owns a topic, fully staffed with experts in house, minimizes these hindrances.

Because an MDRC focused on influence operations does not fit neatly into existing organizational structures, its initial setup should be an iterative process. It should start with two or more tech companies joining with a cluster of academic organizations on a discrete set of deliverables, all with firm security agreements in place. Once the initial set of projects proves the model’s value, and plans for budgets and researcher time are solidified, the organization could be expanded. The negative impact of internet platforms’ impact on society did not grow over night, and we certainly do not expect the solution to either. And, tempting as it is to think the U.S. government could simply fund such an institution, it likely needs to remain independent of government funding in order to avoid collusion concerns from the international community. 

Steps toward bridging the gap between academia and the social media firms have already taken place. Facebook’s recent provision of academic access to Crowdtangle, meant in part to provide increased transparency on influence operations and disinformation, is a good step, as is its data-sharing partnership with several universities to look at election-related content. Such efforts will enable some work currently stymied by data sharing, but they do not address the deeper incentive-related issues. 

Establishing a long-term MDRC around the study of influence operations and misinformation is more crucial than ever. It is a logical way forward to address these questions at the scale they deserve.  

In 2020, CITP launched the Public Interest Technology Summer Fellowship (PIT-SF) program aimed at rising juniors and seniors interested in getting first-hand experience working on technology policy at the federal, state and local level. The program is supported by the PIT-UN network and accepts students from member universities. We pay students a stipend and cover their reasonable travel costs. We are delighted to announce that applications are open for second year of the program. This post describes the firsthand reflections of three students from the program’s inaugural cohort. 

Who are we and where were our fellowships?

Manish Nagireddy: I’m a sophomore at Carnegie Mellon studying statistics and machine learning. I worked in the AI/ML division of the Data Science and Analytics group at the Consumer Financial Protection Bureau (CFPB).

Julia Meltzer: I’m a junior at Stanford, doing a major in Symbolic Systems (part linguistics, part computer science, part philosophy, and part psychology) and minoring in Ethics and Technology. I worked on the Policy Team for the NYC Mayor’s Office of the Chief Technology Officer (MoCTO).

Meena Balan: I’m a junior at Georgetown studying International Politics with a concentration in International Law, Ethics, and Institutions, and minors in Russian and Computer Science. Last summer I had the opportunity to work with the Office of Policy Planning (OPP) at the Federal Trade Commission (FTC). 

What made you apply for the PIT-SF fellowship? 

Meena: As a student of both the practical and the qualitative aspects of technology, I am strongly drawn to the PIT field because of the opportunity to combine my interests in law, policy, ethics, and technological governance and engage with the social and economic impacts of technology at a national scale. In addition to gaining unique real-world experience and working on cutting-edge issues in the field, I found the PIT-SF fellowship particularly compelling because of its emphasis on mentorship, both from peers and experts in the field, which I believed would help me to grapple more meaningfully with issues I had previously only encountered in a classroom environment. 

Julia: I have long been attracted to and inspired by the Public Interest Technology (PIT) sphere which allows technologists, policymakers, activists, and experts in all fields to ensure that the technological era is just and that the incredible power tech offers is used for social good. As a student with interests in policy, programming, and social impact, I was thrilled to find the rare opportunity to make a difference, in an entry-level position, working on the problems I find most essential. The fellowship also offered the benefit of wisdom from the program’s leaders and guest speakers.

Manish: PIT to me, at face value, means creating and using technology in responsible manners. Specifically, this term represents the mindset of always keeping social values and humanitarian ethics when designing sophisticated technological systems. I applied to this fellowship because it offered a unique opportunity to combine my love of technology for social good as well as gain insight into how government agencies deal with tech-related issues.

How did the PIT-SF fellowship influence you?

Julia: From CITP and the orientation for the fellowship, I learned about the wide range of policy issues central to regulating technology. The personal narratives that guest speakers and the program leaders shared provided assurance that there is no wrong way to join the PIT coalition and inspired me to follow the path that I feel drawn to instead of whatever may seem like the correct one.

At MoCTO, I experienced the full range of what it means to work on local (city-wide) PIT efforts. From watching the design team navigate website accessibility to tracking global COVID-19 technical solutions to advocating for new legislation, my summer as a fellow has compelled me to enter a career in civil service at the same intersection into which MoCTO provided me a foray. I’ve had the privilege to continue working for MoCTO where I’ve begun to gain a deep and full understanding of the ways in which technology policy is written and passed into law. Thanks to the role models I found through MoCTO, I am now applying to law schools not only to become a lawyer, but to increase my comprehension of PIT. I learned by watching my supervisor and the rest of our team that a systematic and complete mastery of the technical logistics, the historical use, the social implications, and the legal context are all essential knowledge bases for those working in the PIT sphere.

Meena: As a fellow working with the FTC, I worked on analyzing acquisitions by prominent technology companies. The process of acquisition analysis is one that combines both technical and qualitative skills, allowing me to uniquely leverage my multidisciplinary background to engage with the business structures, technological features, and post-acquisition implications of hundreds of companies. In addition to gaining a better understanding of investment and growth patterns in the tech sector, I developed a deeper understanding of the economic theories and laws underlying antitrust analysis through direct mentorship with experts in the field. At the culmination of my fellowship, my peers and I presented our findings to the OPP and received valuable feedback from senior leadership, which fueled my interest in the field of tech policy and guided me to follow cutting-edge trends in the applications of emerging technologies more closely. 

Through the course of the fellowship, CITP also offered incredible exposure to PIT niches outside of antitrust, empowering me to develop a greater understanding of both public and private sector perspectives and the broader issue landscape. During the bootcamp, fellows were invited to participate in meaningful discussions with industry leaders and senior experts across federal and local government, intelligence, law, and the technology sectors. This provided us with unique opportunities to understand the issues of privacy, equity and access, and algorithmic fairness not only through a regulatory lens, but also in terms of the technical, business, and ethical challenges that play a significant role in shaping PIT initiatives. Given the broad complexity of the PIT field and the evolving nature of professional exposure at the undergraduate level, the PIT-SF fellowship offered impressive and unparalleled real world experience that has contributed significantly to my pursuit of a career at the intersection of technology, law, and policy.

Manish: During my fellowship at the CFPB, I worked on fair-lending models and this introduced me to the field that I wish to join full time: fairness in machine learning. Borne out of a need to create models that maintain equality with respect to various desirable features/metrics, fair-ml is an interdisciplinary topic that deals with both the algorithmic foundations as well as the real-world implications of fairness-aware machine learning systems.

My fellowship directly introduced me to this field and, by the end of my stint at the CFPB, I compiled all of the knowledge I had amassed through a literature deep-dive in the form of a formal summary paper (linked here). Moreover, this fellowship gave me the necessary background for my current role of leading a research team based in Carnegie Mellon’s Human-Computer Interaction Institute (HCII) where the focus is on how industry practitioners formulate and solve fairness-related tasks.

One of the best parts about this fellowship is that public interest technology itself is broad enough of a field to allow for extremely diverse experiences with one common thread: relevance. Every fellowship dealt with, in some capacity, a timely and cutting edge topic. Personally, the field of fair-ml has only been rigorously studied within the past decade, which allowed me to easily find the most important papers and people to read and reach out to, respectively. The ability to find both incredibly pertinent and also rather interesting work is an immediate consequence of my PIT-SF fellowship.

Conclusion: We plan to invite approximately 16 students to this year program, which will operate in a hybrid format. Like last year, we begin with a virtual three-day policy bootcamp led by Mihir Kshirsagar and Tithi Chattopadhyay. The bootcamp will educate students about law and policy, and will feature leading experts as guest speakers in the fields of computer science and policy. After the bootcamp, fellows will travel to (or join virtually) the host government agencies in different cities that our program has matched them with to spend approximately eight weeks working with the agency. We will also have weekly virtual clinic-style seminars to support the fellows during their internships. At the conclusion of the summer, we aim to bring the 2021 and 2020 PIT-SF fellows for an in-person debriefing session in Princeton (subject to the latest health guidelines). CITP is committed to building a culturally diverse community, and we are interested in receiving applications from members of groups that have been historically underrepresented in this field. The deadline to apply is February 10, 2021 and the application is available here.