Preliminary unofficial election results posted at 4am after the November 3rd 2020 election, by election administrators in Antrim County Michigan, were incorrect by thousands of votes–in the Presidential race and in local races. Within days, Antrim County election administrators corrected the error, as confirmed by a full hand recount of the ballots, but everyone wondered: what went wrong? Were the voting machines hacked?

The Michigan Secretary of State and the Michigan Attorney General commissioned an expert to conduct a forensic examination. Fortunately for Michigan, one of the world’s leading experts on voting machines and election cybersecurity is a professor at the University of Michigan: J. Alex Halderman. Professor Halderman submitted his report to the State on March 26, 2021 and the State has released the report.

Analysis of the Antrim County, Michigan
November 2020 Election Incident
J. Alex Halderman
March 26, 2021

And here’s what Professor Halderman found: “In October, Antrim changed three ballot designs to correct local contests after the initial designs had already been loaded onto the memory cards that configure the ballot scanners. … [A]ll memory cards should have been updated following the changes. Antrim used the new designs in its election management system and updated the memory cards for one affected township, but it did not update the memory cards for any other scanners.”

Here’s what that means: Optical-scan voting machines don’t (generally) read the text of the candidates’ names, they look for an oval filled in at a specific position on the page. The Ballot Definition File tells the voting machine what name corresponds to what position. And also informs the election-management system (EMS) that runs on the county’s election management computers how to interpret the memory cards that transfer results from the voting machines to the central computers.

Shown here at left is the original ballot layout, and at right is the new ballot layout. I have added the blue rectangles to explain Professor Halderman’s report.

Now, if the voting machine is loaded with a memory card with the ballot definition at left, but fed ballots in the format at right, what will happen?

A voter’s mark next to the name “Melanie Eckhart” will be interpreted as a vote for “Mark Edward Groenink”. That is, in the first blue rectangle, you can see that the oval at that same ballot position is interpreted differently, in the two different ballot layouts.

A voter’s mark next to “Yes” in Proposal 20-1 will be interpreted as “No” (as you can see by looking at the second blue rectangle).

We’d expect that problem with any bubble-ballot voting system (though there are ways of preventing it, see below). But the Dominion’s results-file format makes the problem far worse.

In Dominion’s file format for storing the results, every subsequent oval on the paper is given a sequential ID number, cumulative across all ballot styles used in the county. Now look at the figure above, just below the first blue rectangle. You’ll see that in the original “Local School District” race (at left) there are two write-in bubbles, but in the revised “Local School District” race (at right), there are three write-in bubbles. That means the ID number of every subsequent bubble, on this ballot and in all the ballot styles that come after it in this county, the ID numbers will be off by one. Figure 2 of the report illustrates:

Within three days, Antrim County officials had basically figured out what went wrong, and corrected most of the errors before publishing and certifying official election results on November 6th. By November 21, Antrim County had corrected almost all of the errors in its official restatement of its election results.

How do we know that the original results were wrong and the new results are right? That is, how do we know that the “corrected” results are true, and not fraudulent? We have two ways of knowing:

• Hand-marked paper ballots speak for themselves. The contest for President of the U.S. was recounted by hand in Antrim County. Those results–from what bipartisan workers and witnesses could see with their own eyes–matched the results from scanning the paper ballots using the ballot-definition file that matches the layout of the paper ballot.
• A careful forensic examination by a qualified expert can explain what happened, and that is why Professor Halderman’s report is so valuable–it explains things step by step.

But not every contest was recounted by hand. The expert analysis finds a few contests where the reported vote totals are still incorrect; and in one of those contests (a marijuana ballot question) the outcome of the election was affected.

In the court case of Bailey v. Antrim, plaintiffs had submitted a report (December 13, 2020) from one Russell J. Ramsland making many claims about the Dominion voting machines and their use in Antrim County: adjudication, error rates, log entries, software updates, Venezuela. Section 5 of Professor Halderman’s report addresses all of these claims and finds them unsupported by the evidence.

What can we learn from all of this?

• Although the unofficial reports posted at 4am on November 4th showed Joseph R. Biden getting more votes in Antrim County than Donald J. Trump, the results posted November 6th show correctly that, in Antrim County, Mr. Trump got more votes.
• Regarding the presidential contest, election administrators figured this out for themselves without needing any experts.
• In other contests, where no recount was done, most of the errors got corrected, but not all.
• There is no evidence that Dominion voting systems used in Antrim County were hacked.

And what can we learn about election administration in general?

• Hand marked paper ballots are extremely useful as a source of “ground truth”.
• If the ballot definition doesn’t match the paper ballot, results reported by the optical-scan voting machine can be nonsense. This has happened before–see my report describing a 2011 incident in New Jersey.
• “Unforced error:” Dominion’s election-management system (EMS) software doesn’t check candidate names. The EMS computer has a file mapping ballot-position numbers to candidate names; and the memory card uploaded from the voting machine has its own file mapping ballot-position numbers to candidate names. If only the EMS software had checked that these files agreed, then the problem would have been detected on election night, during the upload process.
• Even without that built-in checking, to catch mistakes like this before the election, officials should do the kind of end-to-end pre-election logic-and-accuracy testing described in Professor Halderman’s report.
• Risk-Limiting Audits (RLAs) could have detected and corrected this error, if they had been used systematically in the State of Michigan. RLAs are good protection not only against hacking, but also against mistakes and bugs.

Over the years I have written many articles about direct-recording electronic (DRE) voting machines, precinct-count optical-scan (PCOS) voting machines, ballot-marking devices (BMDs), and other 21st-century voting technology. But I haven’t written much about 20th-century lever machines; these machines were banned by the U.S. Congress in the Help America Vote Act and have not been used since 2012.

Recently, upon a midnight dreary, while I pondered, weak and weary, over many a quaint and curious volume of forgotten technology, I came across the excellent 1993 book, The Way Things Really Work, by Henry Beard and Rod Barrett. This book has a clear explanation of the inner workings of mechanical lever voting machines, as follows.

I think it should now be clear why Congress banned this technology.

The book also has explanations of “How candy machines eat your quarters,” “How airlines lose your luggage,” “How elevators know to close their doors when you come running,” and so on.

By Andrew Appel and Susan Greenhalgh

The accuracy of a voting machine is dependent on the software that runs it. If that software is corrupted or hacked, it can misreport the votes.  There is a common assumption that we can check the legitimacy of the software that is installed by checking a “hash code” and comparing it to the hash code of the authorized software.  In practice the scheme is supposed to work like this:  Software provided by the voting-machine vendor examines all the installed software in the voting machine, to make sure it’s the right stuff.

There are some flaws in this concept:  it’s hard to find “all the installed software in the voting machine,” because modern computers have many layers underneath what you examine.  But mainly, if a hacker can corrupt the vote-tallying software, perhaps they can corrupt the hash-generating function as well, so that whenever you ask the checker “does the voting machine have the right software installed,” it will say, “Yes, boss.”  Or, if the hasher is designed not to say “yes” or “no,” but to report the hash of what’s installed, it can simply report the hash of what’s supposed to be there, not what’s actually there. For that reason, election security experts never put much reliance in this hash-code idea; instead they insist that you can’t fully trust what software is installed, so you must achieve election integrity by doing recounts or risk-limiting audits of the paper ballots.

But you might have thought that the hash-code could at least help protect against accidental, nonmalicious errors in configuration.  You would be wrong.  It turns out that ES&S has bugs in their hash-code checker:  if the “reference hashcode” is completely missing, then it’ll say “yes, boss, everything is fine” instead of reporting an error.  It’s simultaneously shocking and unsurprising that ES&S’s hashcode checker could contain such a blunder and that it would go unnoticed by the U.S. Election Assistance Commission’s federal certification process. It’s unsurprising because testing naturally tends to focus on “does the system work right when used as intended?”  Using the system in unintended ways (which is what hackers would do) is not something anyone will notice.

Until somebody does notice.  In this case, it was the State of Texas’s voting-machine examiner, Brian Mechler.  In his report dated September 2020 he found this bug in the hash-checking script supplied with the ES&S EVS 6.1.1.0 election system (for the ExpressVote touch-screen BMD, the DS200 in-precinct optical scanner, the DS450 and DS850 high-speed optical scanners, and other related voting machines).  (Read Section 7.2 of Mr. Mechler’s report for details).

We can’t know whether that bug was intentional or not.  Either way, it’s certainly convenient for ES&S, because it’s one less hassle when installing firmware upgrades.  (Of course, it’s one less hassle for potential hackers, too.)

Another gem in Mr. Mechler’s report is in Section 7.1, in which he reveals that acceptance testing of voting systems is done by the vendor, not by the customer.  Acceptance testing is the process by which a customer checks a delivered product to make sure it satisfies requirements.  To have the vendor do acceptance testing pretty much defeats the purpose.  

When the Texas Secretary of State learned that their vendor was doing the acceptance testing themselves, the SoS’s Election Division took an action “to work with ES&S and their Texas customers to better define their roles and responsibilities with respect to acceptance testing,” according to the report. They may encounter a problem, though: the ES&S sales contract specifies that ES&S must perform the acceptance testing, or they will void your warranty (see clause 7b) . 

There’s another item in Mr. Mechler’s report, Section 7.3.  The U.S. Election Assistance Commission requires that “The vendor shall have a process to verify that the correct software is loaded, that there is no unauthorized software, and that voting system software on voting equipment has not been modified, using the reference information from the [National Software Reference Library] or from a State designated repository. The process used to verify software should be possible to perform without using software installed on the voting system.”  This requirement is usually interpreted to mean, “check the hash code of the installed software against the reference hash code held by the EAC or the State.”

But ES&S’s hash-checker doesn’t do that at all.  Instead, ES&S instructs its techs to create some “golden” hashes from the first installation, then subsequently check the hash code against these.  So whatever software was first installed gets to be “golden”, regardless of whether it’s been approved by the EAC or by the State of Texas. This design decision was probably a convenient shortcut by engineers at ES&S, but it directly violates the EAC’s rules for how hash-checking is supposed to work.

So, what have we learned?

We already knew that hash codes can’t protect against hackers who install vote-stealing software, because the hackers can also install software that lies about the hash code.  But now we’ve learned that hash codes are even more useless than we might have thought.  This voting-machine manufacturer

• has a hash-code checker that erroneously reports a match, even when you forget to tell it what to match against;
• checks the hash against what was first installed, not against the authorized reference that they’re supposed to;
• and the vendor insists on running this check itself — not letting the customer do it — otherwise the warranty is voided.

As a bonus we learned that the EAC certifies voting systems without checking if the validation software functions properly. 

Are we surprised?  You know: fool me once, shame on you; fool me twice, shame on me.  Every time that we imagine that a voting-machine manufacturer might have sound cybersecurity practices, it turns out that they’ve taken shortcuts and they’ve made mistakes.  In this, voting-machine manufacturers are no different from any other makers of software.  There’s lots of insecure software out there made by software engineers who cut corners and don’t pay attention to security, and why should we think that voting machines are any different?

So if we want to trust our elections, we should vote on hand-marked paper ballots, counted by optical scanners, and recountable by hand.  Those optical scanners are pretty accurate when they haven’t been hacked — even the ES&S DS200 — and it’s impractical to count all the ballots without them.  But we should always check up on the machines by doing random audits of the paper ballots.  And those audits should be “strong” enough — that is, use good statistical methods and check enough of the ballots — to catch the mistakes that the machines might make, if the machines make mistakes (or are hacked).  The technical term for those “strong enough” audits is Risk-Limiting Audit.

Andrew W. Appel is Professor of Computer Science at Princeton University.

Susan Greenhalgh is Senior Advisor on Election Security at Free Speech For People.