Yesterday I explained the design error that led Diebold in 2005 to recall and replace the motherboards in thousands of voting machines, most of which had been used in the November 2004 election. Today I’ll talk about how the motherboard flaws might have affected the accuracy of elections.
Machines with flawed boards were normally identified when they “froze” on election day. When personal computers crash, they often manage to reboot themselves, but the Diebold machines don’t reboot themselves on a crash, so any kind of general system crash will make the system freeze. So the bug was usually identified when a voting machine crashed. Mystery crashes typically don’t happen at random times but are concerntrated at certain stages of the machine’s use, because the detailed technical conditions that trigger the crash are more likely to happen at some times than at others.
When did the flawed Diebold machines crash? Here’s the Montgomery County (Maryland) Lessons Learned report from the 2004 election (page 11):
Election judges and technical staff reported that many of these units froze when the voter pressed the Cast Ballot button. This leads to great confusion for judges and voters. The voter leaves the polling place with little or no confidence that their vote was counted. In many cases, the election judges are unable to provide substantial confirmation that the vote was, in fact, counted.
You’d be hard pressed to pick a worse time for a voting machine to crash. The voter has made his selections, confirmed them on the ballot review screen, and now wants them to be recorded. When the Cast Vote button is pressed, the machine reads the intended votes out of its temporary RAM memory and copies them into the official ballot record file, which lives in the machine’s flash memory. If the machine crashes just before the vote is copied, the vote is lost. If it crashes just after the vote is copied, the vote is recorded. It won’t be immediately obvious which case you’re in – hence the confused voters and poll workers.
The kind of design mistake Diebold made – timing errors in the use of RAM chips – crops up in other (non-voting) systems, so we know what kinds of problems it tends to cause. Sometimes it will cause system crashes, but sometimes it will cause data to be corrupted when it gets copied from one place to another. Which is particularly worrisome because the Diebold flaw tends to show up just at the time when the vote is copied into the official record.
And that’s not all. Some other machines failed with Ballot Exception Errors, which happen when the machine’s log file is corrupted – a file that is stored alongside the vote record file, and is also updated when the Cast Vote button is pressed. So we know that some of the records kept by the voting machine (either internally or on removable memory cards) were getting corrupted.
Were votes ever actually corrupted? We’ll never know. If we had a voter-verified paper audit trail, we could compare it to the records kept by the crashed machines. But with only the electronic records to go on, it’s probably impossible to tell.
The good news is that all of the affected motherboards have now been replaced. The bad news is that Diebold knew about these problems in March 2004, and yet they allowed thousands of affected machines to be used in the November 2004 election.
With the elections now upon us in full swing, its essential the lessons are learnt from this computing failure. Its mission critical and must more redunancy should be given to the systems such as dual motherboard configurations
i once believed that we the people of america actually had a say so in who will ultimately become the president. but there is a cold truth out there that most of us dont know. and that truth is that we do not have a say so. These Diebold machines can be haked and manipulated to swing the vote one way or another, and they will been hacked once again in the 2008 presidential election. the evidence is already out there people… just open your eyes and see it for yourself.
“But with only the electronic records to go on, it’s probably impossible to tell.”
Not probably, absolutely impossible to tell. Without any kind of independant data storage, and with softwarecode you can’t look at, it is impossible to confirm any connection between the input and output of these machines. You may as well ask everyone to whisper in my ear and trust the numbers I give you at the end of the day.
My imagination, or are some people here taking the plot of “Cats and Dogs” a little too seriously? fsm3ioaspd,3eh what? get off my keyboard fur face. hey! claws hurt! aaargh…
🙂
Sorry about the Cuyahoga virus post. I made some edits, and inadvertently changed the post’s status to private. That meant that other people couldn’t see it, but I could — so I didn’t see anything wrong.
It’s back now. Thanks for pointing out the problem.
Ronald Crane Said:
“Neko’s fast, but not fast enough for my eyes.”
Eyes are overrated. I can assure you that the Project for the Neko American Century has affairs well in paw.
Neko’s fast, but not fast enough for my eyes.
Ronald Crane Says:
“What happened to the post on Cuyahoga County’s potential virus exposure? The one that used to be at
nonexistant url redacted…”
I’m sorry, citizen. There is no such blog entry. There never was. Perhaps your eyes are tired. You should get some rest.
We can assure you that everything is perfectly secure and under control.
The Management
What happened to the post on Cuyahoga County’s potential virus exposure? The one that used to be at http://www.freedom-to-tinker.com/?p=1083 ?
My bad–the p=1080 posting had disappeared (it’s back now) and without being able to look at it any more I thought it had been an entry on AV-OS, but it was actually the initial motherboard implications posting. I got mixed up with Avi Rubin’s last blog entry.
Thanks for all your great work.
antibozo,
I haven’t posted anything (yet) about the AccuVote-OS report. It’s an interessting report, but my blogging schedule doesn’t allow me to write about every interesting thing I see. That’s especially true now, when so much is happening in advance of the election.
antibozo Says:
“Um, what happened to the blog entry about the AV-OS report?”
I’m sorry, citizen. There is no such blog entry. There never was. Perhaps your eyes are tired. You should get some rest.
The Management
Seriously, could be another WordPress glitch… hard for me to tell but I think they’ve been fiddling with it a bit the past couple of days.
Um, what happened to the blog entry about the AV-OS report?
Prepare for an onslaught?…
HBO’s “Hacking Democracy” goes live tonight and I think we can expect a virtual avalanche of (equally virtual) Diebold apologists 🙂
Also… A report just out that all Sequoia machines currently deployed can have somebody add infinite votes to their counts… not by anything as mundane as a hotel minibar key… but by just reaching around the back and pushing the yellow button.
Seriously.
Doug Says:
“Honestly, how hard is it to make a voting machine?”
That’s the primary problem with Diebold. They didn’t design voting machines.
Their machine concept takes cheap general-purpose computer components and shoehorns them into touchscreen-equipped boxes… apparently all without any thought as to computer security, human nature, electoral realities or much of anything else.
Honestly, how hard is it to make a voting machine? It seems to me that these things shouldn’t be more than a glorified pocket calculator that only needs to add! My TI-86 has never crashed, and it does alot more than increment an integer when a user pushes a button.
Ed et al… to help you adjust your spam filters: extensive commercial spammed at #1064
http://www.freedom-to-tinker.com/?p=1064
hate spam… first thought it was more Diebold backfilling… worse than splitters, that… 🙂
Ed, I think it is possible that you have conflated two separate types of Diebold crashes.
In the volume testing of 96 Diebold TSx machines (not the TS used in MD) done by the CA Secretary of State in the summer of 2005, we had an enormously high crash rate: over 20% of the machines crashed during the course of one election day’s worth of votes. These crashes always occurred either at the end of one voting transaction when the voter touched the CAST button, or right at the beginning of the next voter’s session when the voter SmartCard was inserted.
It turned out that, after a huge effort on Diebold’s part, a GUI bug was discovered. If a voter touched the CAST button a sloppily, and dragged his/her finger from the button across a line into another nearby window (something that apparently happened with only one of every 400 or 500 voters) an exception would be signaled. But the exception was not handled properly, leading to stack corruption or heap corruption (it was never clear to us which), which apparently invariably lead to the crash. Whether it caused other problems also, such as vote corruption, or audit log corruption, was never determined, at least to my knowledge. Diebold fixed this bug, and at least TSx machines are free of it now. I my colleagues (David Wagner, Matt Bishop, and Loretta Guarino-Reid) wrote a report on this which we sent to the SoS, but it was never published.
I bring this up because, while I don’t know for sure, I think this GUI bug may be the source of the crashes that the 2004 Montgomery County “Lessons Learned” report referred to, rather than the recently described motherboard problems. My hypothesis assumes that the same bug was present in the TS GUI used in Montgomery County as was present in the TSx GUI. You should be able to test this using the TS in your possession, since the version of the code you have dates from before the cause of the TSx GUI bug was found. Nothing I have seen positively indicates that the motherboard bug causes crashes at the moment of touching the CAST button. But I know that the GUI bug did just that.
If you want a copy of our report, I’ll be happy to send it.
But wouldn’t it also be likely that votes which are cast for one party but counted as votes for the other party would “GOP vote counted as Democratic vote” as often as “Democratic vote counted as GOP vote”? Yet so far, in many instances of this documented, it’s virtually always “Democratic vote counted as GOP vote”.
I also think that allowing workers to take home voting machines and store them in their garages for several weeks before the election is downright weird. But hey, that’s just me. 🙂
The problem with Scott Adam’s blog is that the maker of the machines are among the corporate s—bags that he decries. They may even have a strong political agenda, like so many corporations have anymore. If they are even half as corrupt as the maker of a popular operating system, elections are probably already being subverted–and not by some teenager in Finland.
Re DMC’s comment, it seems to me that if a machine in a particular polling place were to lose or corrupt votes, that might have a skewing affect towards one or another candidate, depending on the likely voting preferences of those assigned to that poll. To put it more directly, if a machine was freezing up in a polling place in a heavily Republican neighborhood, then it is more likely that Republican votes would be corrupted or lost.
Ever so slightly off topic, it would seem that diebold is getting a little testy about all of this adverse publicity. Earlier today they released this press release.
http://www.prnewswire.com/news/index_mail.shtml?ACCT=104&STORY=/www/story/10-31-2006/0004463570&EDATE=
Good point, Roger. I always find it ironic that a web site dedicated to the “freedom to tinker” thinks its *objectionable* when people play games with election machines…
Does anyone read Scott Adams blog? Why worry about this so much.
http://dilbertblog.typepad.com/the_dilbert_blog/2006/10/electronic_voti.html
The good news is that all of the affected motherboards have now been replaced. The bad news is that Diebold knew about these problems in March 2004, and yet they allowed thousands of affected machines to be used in the November 2004 election.
I would certainly hope that this could open the door to some type of criminal prosecution of Diebold, or at least several of their employees who knew about the flaw and knowingly allowed the machines to be used.
In this way, the movement towards e-voting could even be derailed, by making the likelihood of legal action and huge damages, possibly criminal prosecution, so probable that this line of work becomes so risky and unprofitable, that the shareholders will not permit Diebold to continue selling and marketing evoting machines.
Another line would be refunds for defective machines, or the jurisdictions which purchase these could write large penalties into their contracts with Diebold, so an instances would cause large penalties to be imposed, similar to liquidated damages in a construction contract.
It is discomforting to know that votes may have been lost.
The good thing, I guess, is that it seems unlikely that votes would have been lost systematically for one candidate or one party.