Yesterday, Kim Zetter at Wired News reported an amazing e-voting story about lost ballots and the public advocates who found them.
Here’s a summary: Humboldt County, California has an innovative program to put on the Internet scanned images of all the optical-scan ballots cast in the county. In the online archive, citizens found 197 ballots that were not included in the official results of the November election. Investigation revealed that the ballots disappeared from the official count due to a programming error in central tabulation software supplied by Premier (formerly known as Diebold), the county’s e-voting vendor.
The details of the programming error are jaw-dropping. Here is Zetter’s deadpan description:
Premier explained that due to a programming problem, the first “deck” or batch of ballots that are counted by the GEMS software sometimes gets randomly deleted if any subsequent deck is intentionally deleted. The GEMS system names the first deck of ballots “deck 0”, with subsequent batches called “deck 1,” “deck 2,” etc. For some reason “deck 0” is sometimes erased from the system if any other deck is erased. Since it’s common for officials to intentionally erase a deck in the normal counting process if they’ve made an error and want to rescan a deck, the chance that a GEMS system containing this flaw will delete a batch of ballots is pretty high.
The system never provides any indication to election officials when it’s deleting a batch of ballots in this manner. The problem occurs with version 1.18.19 of the GEMS software, though it’s possible that other versions have the problem as well. [County election director Carolyn] Crnich said an official in the California secretary of state’s office told her the problem was still prevalent in version 1.18.22 of Premier’s software and wasn’t fixed until version 1.18.24.
Neither Premier nor the secretary of state’s office, which certifies voting systems for use in the state, has returned calls for comment about this.
After examining Humboldt’s database, Premier determined that the “deck 0” in Humboldt was deleted at some point in between processing decks 131 and 135, but so far Crnich has been unable to determine what caused the deletion. She said she did at one point abort deck 132, instead of deleting it, when she made a mistake with it, but that occurred before election day, and the “deck 0” batch of ballots was still in the system on November 23rd, after she’d aborted deck 132. She couldn’t recall deleting any other deck after election night or after the 23rd that might have caused “deck 0” to disappear in the manner Premier described.
The deletion of “deck 0” wasn’t the only problem with the GEMS system. As I mentioned previously, the audit log not only didn’t show that “deck 0” had been deleted, it never showed that the deck existed in the first place.
The system creates a “deck 0” for each ballot type that is scanned. This means, the system should have three “deck 0” entries in the log — one for vote-by-mail ballots, one for provisional ballots, and one for regular ballots cast at the precinct. Crnich found that the log did show a “deck 0” for provisional ballots and precinct-cast ballots but none for vote-by-mail ballots, even though the machine had printed a receipt at the time that an election worker had scanned the ballots into the machine. In fact, the regular audit log provides no record of any files that were deleted, including deck 132, which she intentionally deleted. She said she had to go back to a backup of the log, created before the election, to find any indication that “deck 0” had ever been created.
I don’t know which is more alarming: that the vendor failed to treat as an emergency a programming error that silently deletes ballots, or that the tabulator’s “audit log” looks more like an after the fact reconstruction of what-must-have-happened rather than a log of what actually did happen.
The good news here is that Humboldt County’s opening of election records to the public paid off, when members of the public found important facts in the records that officials and the vendor had missed. If other jurisdictions opened their records, how many more errors would we find and fix?
How does something like this get past checking?
No, seriously. We’re past the “of course all the code in election-related machines should be subject to public scrutiny” to a thoroughgoing review of how voting-machine companies produce their software and why they shouldn’t be considered criminal frauds for taking public money under false pretenses.
( previous comment…. User error …Hit SAVE instead of Preview….please delete …(.last year.)….Humboldt….)
During the 2007 CA Top-To-Bottom – Review (TTBR) of voting systems, all systems failed to meet security feature requirements for certification. This includes voting systems from Premier/Diebold, Hart InterCivic, Sequoia, ES&S and InkaVote Plus. All voting systems were then Conditionally re-approved with the assurance that the state required audits would provide assurance of reliable and accurate election outcomes. The SOS, Bowen also required each county to create a citizen’s Observer Panel which puts a large burden on the public to play watchdog to a seriously defective voting system … ( I highly recommend reading all the TTBR reports and the Conditional Re-approval requirements.)
I agree with Professor Felten that “opening of the records to the public paid off when members of the public found important facts in the records THAT OFFICIALS AND VENDOR HAD MISSED.” However, in this particular instance the error was quite obvious to the human eyeball but it was completely overlooked by the audit. Ultimately, who is to be held accountable for machine failures, whether caused by faulty equipment or human error and how are we to prove purposeful machine fraud should that ever be the case? And what about the not so obvious errors? What about vote by mail paper ballot chain of custody ?
My response to the the final question “If other jurisdictions opened their records how many more errors would we find and fix ?” …………
As election integrity advocates we have assumed the responsibility of playing watch dog. But we have no authority. We are totally dependent on the voluntary cooperation of our officials. Our questions are often severely and critically challenged and will not be answered should they be considered an “impediment to conducting the election.” We have no guarantee that our public records requests will be responded to in a timely manner. We have only 5 days after the final election results are released to request a recount. We are fortunate to have an electronics tech working with our group but every county in the nation is not so lucky. We do not have the funds to sue to get electronic records such as the database.
I hope this plea will inspire you to look beyond the software issues and conclude that ultimately if the citizens are responsible to play watchdog they deserve to count the ballots in the first place. Professor Stephen Unger at Columbia University (engineering and computer dept.) has posted an essay on his blog which helps to explain our situation. We support Professor Unger’s solution. Lets go” Back to the Future” and let the public count the ballots at the precinct on election day. We hope you will agree.For more info/questions,comments please e-mail . Thanks for your interest.
Yes indeed Kim Zetter has written an amazing article about “lost ballots and the public advocates who found them”. As Chair of SAVElections Monterey County (an active election integrity advocacy group) I understand that your focus is obviously to address the software issues, However, I must ask that you please consider the broad picture in your analysis. What is unfortunately omitted from this important report exerpt is that the CA STATE MANDATED AUDIT DID NOT FIND THE ERROR before the election was certified. 58 counties in CA are depending on the state required audits to discover such problems. In AUG 2007 The CA SOS Bowen, Decertified all CA e-voting systems for comparable systemic voting system security failures. last year Humboldt County EI advocates must be applauded for their diligence and civic minded dedication to reviewing the e-voting systems incorrect results with the innovative digital scan. . However, the election was in fact certified by the Registrar of Voters before the error was found.
Looks like they know they’ve got a bug – they think they know how it works – but actually you never really know this until you’ve fixed it – and they haven’t bothered to fix it…oh dear
The interesting aspects of programming are, in the case of this defect, completely outside the point. You have a piece of software that the vendor knows is deeply flawed, but nothing is done about it. From the story, I can’t tell if the vendor simply said nothing or if the flaw was reported, but the operators (county workers) failed to do the right thing. No matter what, you have a situation where AT LEAST one person with a lot of responsibility has shown none.
What if nobody had noticed them on the web? How many other flaws in the machine lay unrevealed or, worse yet, undiscovered?
Stunning.
Michael,
Yes, as you have astutely observed there is no valid accountability .
I was not able to find the article in Wired but I have a copy that was sent to my e-mail if you would like to read the full report. I am also willing to share copy of our reports to our Board of Supervisors and to the Secretary of State if this is an issue you care to pursue… send e-mail to .
Some people think that arrays start counting at 1, while others prefer to start at 0.
Although 0 is a very useful concept in mathematics, it is also an unintuitive concept for most people (what happened to year 0?) and a great source of confusion (is year 2000 in the 20th century?). After many years in the computer industry, you learn never to trust anyone to get it right.
The audit log sounds like a bolt-on feature written to a deadline. I can imagine, “We need an audit log, it will boost confidence in our product, you have two days”. I would have thought that an obvious test of the audit log would have been to take copies of the audit log at various times during the process and then run a difference against them. Also, to re-run various test batches from scratch with subtle variations (e.g. a vote missing or a batch out of order) and save the audit log each time, then find differences in the logs. We all know these machines were not seriously tested.
Just think how much fun this would be if California were in the middle of a Minnesota-style squeaker.