How did e-voting technologies hold up in Tuesday’s election? It’s too early to tell for sure, but it looks as if there weren’t any major disasters.

We saw the usual list of crashing, misbehaving, and non-functional machines. Some of these are just routine glitches or procedural problems. If somebody forgets to deliver power cords to the polling place, that’s just an isolated mistake. If a machine just won’t turn on in the morning, that’s probably just a maintenance issue.

But other kinds of “glitches” can indicate deeper problems. Experienced engineers know that certain behaviors, especially complex ones that are supposed to be impossible, are clues that something has gone badly wrong in the system’s internals. If the inside of your fridge is at room temperature, you probably have a simple problem. If the liquids in your fridge are boiling, you have an Engineering Issue.

The most alarming error report I saw from Tuesday’s election came from Avi Rubin, a respected computer scientist and e-voting expert who is a precinct worker in Maryland, where they use the Diebold AccuVote-TS, the same machine my colleagues and I recently studied. Here is Avi’s story:

So, while we were watching the last handful of voters cast their ballots … one of the chief judges came up to me and said that there was a “situation”. I was called over where a voter was explaining to one of the judges what had happened, and he repeated his story to me. The voter had made his selections and pressed the “cast ballot” button on the machine. The machine spit out his smartcard, as it is supposed to do, but his summary screen remained, and it did not appear that his vote had been cast. So, he pushed the smartcard back in, and it came out saying that he had already voted. But, he was still in the screen that showed he was in the process of voting. The voter then pressed the “cast ballot” again, and an error message appeared on the screen that said that he needs to call a judge for assistance. The voter was very patient, but was clearly taking this very seriously, as one would expect. After discussing the details about what happened with him very carefully, I believed that there was a glitch with his machine, and that it was in an unexpected state after it spit out the smartcard.

This is supposed to be impossible. Having examined a similar version of Diebold’s software, I know that when the Cast Vote button is pressed, the system is supposed to (1) invalidate the smartcard, then (2) record the vote, then (3) kill the voting screens, then (4) eject the smartcard. This voter saw Steps 1 and 4 happen, but not Step 3. (We don’t know whether Step 2, recording the vote, happened.) At least one voting screen was still there, and that screen was active: something happened when the Cast Vote button on that screen was pressed, but it wasn’t the something that would normally happen.

It’s hard to see how this can happen, absent a subtle, serious bug in this part of Diebold’s software. And by “this part” I mean the part that carries out the four-step procedure that includes recording the vote. Could this bug have affected vote recording for other voters? What other problems could it have caused? We don’t know. We could probably tell, given access to a Maryland voting machine.

Another thing we don’t know is how many times this bug showed up in Maryland on Tuesday. It’s hard to believe that the problem didn’t happen elsewhere too. If it were going to happen only once, what are the odds that that one occurrence would be in a precinct with an evoting-savvy computer scientist blogger election judge? Pretty slim.

Fortunately, Avi was there and was able to recognize the relevance of this particular machine misbehavior. How many other poll workers, not being experts in computer science, saw a similar problem and just shrugged it off as a routine glitch?

I was going about my business this morning when I was surprised to see some unattended electronic voting machines that had already been delivered to a polling place in advance of Tuesday’s election. I wasn’t looking for voting machines in this location, not knowing that it served as a polling place, but the machines were pretty hard to miss. They were Sequoia AVC Advantage machines, the most common model in New Jersey. I don’t know how long they had been sitting unprotected.

Here’s a photo, taken this morning, of me with one of the machines.

The Election Science Institute just released a statement revealing that the memory cards that will be used to store votes on Election Day in Cuyahoga County, Ohio were stuck into ordinary laptop computers in September.

The release points to an online video shot by Cleveland-area filmmaker Jeffrey Kirkby, shows a group of election workers sitting at tables, each with a laptop computer. An official explains that these laptops were gathered from around the office, and some are the personal laptops of election workers. Each worker has a laptop and a stack of memory cards, and is inserting the memory cards one by one into the laptop.

Our e-voting study) showed that the memory cards used in Diebold touchscreen voting systems can carry computer viruses that can infect voting machines and steal votes on the infected machines.

The risk here is that one of the laptops is infected with malicious software that could infect a memory card that will eventually be inserted into a voting machine. Safe procedures call for memory cards to be inserted only into computers that are carefully secured and never connected to the Internet. Using ordinary laptop computers, borrowed from offices and homes, to process memory cards is dangerous.

Voting machine vendors and election officials often argue that rigorous procedures can compensate for the technical weaknesses of voting machines. Some jurisdictions implement such procedures well, but many do not. Talking about procedural controls is easy. Putting them into practice is much harder.

Yesterday I explained the design error that led Diebold in 2005 to recall and replace the motherboards in thousands of voting machines, most of which had been used in the November 2004 election. Today I’ll talk about how the motherboard flaws might have affected the accuracy of elections.

Machines with flawed boards were normally identified when they “froze” on election day. When personal computers crash, they often manage to reboot themselves, but the Diebold machines don’t reboot themselves on a crash, so any kind of general system crash will make the system freeze. So the bug was usually identified when a voting machine crashed. Mystery crashes typically don’t happen at random times but are concerntrated at certain stages of the machine’s use, because the detailed technical conditions that trigger the crash are more likely to happen at some times than at others.

When did the flawed Diebold machines crash? Here’s the Montgomery County (Maryland) Lessons Learned report from the 2004 election (page 11):

Election judges and technical staff reported that many of these units froze when the voter pressed the Cast Ballot button. This leads to great confusion for judges and voters. The voter leaves the polling place with little or no confidence that their vote was counted. In many cases, the election judges are unable to provide substantial confirmation that the vote was, in fact, counted.

You’d be hard pressed to pick a worse time for a voting machine to crash. The voter has made his selections, confirmed them on the ballot review screen, and now wants them to be recorded. When the Cast Vote button is pressed, the machine reads the intended votes out of its temporary RAM memory and copies them into the official ballot record file, which lives in the machine’s flash memory. If the machine crashes just before the vote is copied, the vote is lost. If it crashes just after the vote is copied, the vote is recorded. It won’t be immediately obvious which case you’re in – hence the confused voters and poll workers.

The kind of design mistake Diebold made – timing errors in the use of RAM chips – crops up in other (non-voting) systems, so we know what kinds of problems it tends to cause. Sometimes it will cause system crashes, but sometimes it will cause data to be corrupted when it gets copied from one place to another. Which is particularly worrisome because the Diebold flaw tends to show up just at the time when the vote is copied into the official record.

And that’s not all. Some other machines failed with Ballot Exception Errors, which happen when the machine’s log file is corrupted – a file that is stored alongside the vote record file, and is also updated when the Cast Vote button is pressed. So we know that some of the records kept by the voting machine (either internally or on removable memory cards) were getting corrupted.

Were votes ever actually corrupted? We’ll never know. If we had a voter-verified paper audit trail, we could compare it to the records kept by the crashed machines. But with only the electronic records to go on, it’s probably impossible to tell.

The good news is that all of the affected motherboards have now been replaced. The bad news is that Diebold knew about these problems in March 2004, and yet they allowed thousands of affected machines to be used in the November 2004 election.

Diebold replaced the motherboard (i.e., the main electronic component) on about 4700 of Maryland’s AccuVote-TS voting machines in 2005, according to Cameron Barr’s story in Thursday’s Washington Post. The company and state officials kept the recall quiet – even some members of the state’s Board of Elections were unaware of it until contacted by the Post. (“If they had asked, we would have told them,” an official said.)

The original motherboards had a design error that caused the machines to become unresponsive, or “freeze”, sometimes during elections. In the 2004 general election, about four percent of Montgomery County’s machines had this problem, according to the county’s 2004 Presidential General Election Review: Lessons Learned report (page 11).

In March 2004, Diebold had sent the state a memo describing the problem in the original motherboards. The memo says that “stack-up of component tolerances” led to timing errors in accessing RAM memory.

Let’s decode that for non-engineer readers. A circuitboard uses many chips or components. The technical specifications for each chip give a set of tolerances, which might say something like this: “If the temperature is between 40 and 140 degrees, and the supply voltage is between 2.9 and 3.1 volts, and a stable signal is delivered on pin 13 for at least 30 nanoseconds, then the chip will respond by sending a signal on pin 19, between 30 and 70 nanoseconds after receiving the pin-13 signal.” This is a promise from the chip’s manufacturer to the system designer. Designers rely on promises like this to make sure their systems will work.

When the designer connects different chips together – when a signal produced by one chip is fed into another one – the designer has to make sure that the signal provided by the first chip falls within the tolerances accepted by the second chip. Otherwise the second chip might not work as advertised, and the overall system might be flaky or simply fail.

But sometimes design errors like these turn out not to cause trouble. If tolerances are just a little bit out of whack, you might just get lucky. Maybe a chip that is guarantted only for voltages over 2.9 volts will still work at 2.88 volts. Maybe a delay guaranteed between 30 and 70 nanoseconds tends to come out on the low end of that range in the batch of chips you got. Or maybe everything works fine, except when something unusual happens – a hot day, or a glitch in the building’s power supply, or an unusual sequence of button presses on the screen. A designer might choose to risk such problems to save money, in an application where reliability isn’t critical. But it shouldn’t happen in a voting machine.

Diebold’s March 2004 memo explains their design problem and says that they redesigned the motherboard to fix the problem. Newly manufactured machines were getting the redesigned motherboards, and any old machines that exhibited problems would have their motherboards replaced. But at that time old machines that hadn’t been seen malfunctioning were left in the field. Diebold estimated that fewer than one percent of the old machines would have problems.

In the November 2004 election, about four percent of Montgomery County machines had screen freezes. Afterward, Diebold decided to recall the old motherboards, replacing them all with new redesigned boards. Today, every Maryland voting machine has one of the new motherboards. Will we see further problems with Diebold’s motherboard design? Only time will tell.

(You may be wondering how these design problems might have affected the accuracy of vote-counting in the 2004 election. I’ll consider that question in the next post.)