On Apr 14 2015, the Virginia State Board of Elections immediately decertified use of the AVS WinVote touchscreen Direct Recording Electronic (DRE) voting machine. This seems pretty minor, but it received a tremendous amount of pushback from some local election officials. In this post, I’ll explain how we got to that point, and what the problems were.
As one of my colleagues taught me, BLUF – Bottom Line Up Front. If an election was held using the AVS WinVote, and it wasn’t hacked, it was only because no one tried. The vulnerabilities were so severe, and so trivial to exploit, that anyone with even a modicum of training could have succeeded. They didn’t need to be in the polling place – within a few hundred feet (e.g., in the parking lot) is easy, and within a half mile with a rudimentary antenna built using a Pringles can. Further, there are no logs or other records that would indicate if such a thing ever happened, so if an election was hacked any time in the past, we will never know.
Now for some background.
The AVS WinVote is a Windows XP Embedded laptop with a touchscreen. Early versions of the software ran the Windows 2000 (an election official told me about playing solitaire on the device, to demonstrate just how complete it was). Later versions ran a somewhat cut-down version, although it’s not clear to me how much it was actually cut down. The WinVote system was certified as meeting the Voting Systems Standards (VSS) of 2002, and was approved for use in Virginia, Pennsylvania, and Mississippi. (It was decertified a few years ago in Pennsylvania, and Mississippi also stopped using theirs a few years ago after some malfunction that I can’t recall in Hinds County.) [A later version of the software was submitted for certification to the Election Assistance Commission, but never approved. I don’t know if that version solved any of the problems described here.]
So how did Virginia get to decertification? It seems that in the November 2014 election, voting machines in one precinct were repeatedly crashing, and it was hypothesized to be due to some interference from someone trying to download music using their iPhone. (There were other problems with other brands of voting machines, but I’m going to focus on the WinVote.) The State Board of Elections invited the Virginia Information Technology Agency (VITA, the agency charged with providing IT services to the state government) to investigate the problem. The report, which was released on Apr 14, includes a litany of problems. [I still don’t understand how the iPhone interfered with the system, but that’s not really important at this point.]
I’ve been in the security field for 30 years, and it takes a lot to surprise me. But the VITA report really shocked me – as bad as I thought the problems were likely to be, VITA’s five-page report showed that they were far worse. And the WinVote system was so fragile that it hardly took any effort. While the report does not state how much effort went into the investigation, my estimation based on the description is that it was less than a person week.
Among the goodies VITA found:
- The wireless connection uses WEP (which we knew). What we didn’t know is that a few minutes of wireless monitoring showed that the encryption key is “abcde”, and that key is unchangeable.
- The system hasn’t been patched since 2004 (which we knew). What we didn’t know is that the system is running a whole bunch of open ports with active services. The report specifically notes that ports 135/tcp, 139/tcp, 445/tcp, 3389/tcp, 6000/tcp and 16001/tcp are all running unpatched services. (Layman’s explanation: the voting machines aren’t just voting machines, they’re also servers happy to give you whatever files you ask for, and various other things, if only you ask. Think of them as an extra disk drive on the network, that just happens to hold all of the votes.) (Obdisclosure: In retrospect, I *probably* could have figured this out a few years ago when I had supervised access to a WinVote with a shell prompt, but I didn’t think of checking.)
- The system has a weak set of controls – it’s easy to get to a DOS prompt (which we knew). What we didn’t know is that the administrator password seems to be hardwired to “admin”.
- The database is a very obsolete version of Microsoft Access, and uses a very weak encryption key (which I knew a couple years ago, but didn’t want to disclose – the key is “shoup”, as also disclosed in the VITA report). What we didn’t know is that there are no controls on changing the database – if you copy the database to a separate machine, which is easy to do given the file services described above, edit the votes, and put it back, it’s happy as can be, and there are no controls to detect that the tampering occurred.
- The USB ports and other physical connections are only marginally physically protected from tampering. What we didn’t know is that there’s no protections once you plug something into one of these ports. What this means is that someone with even a few minutes unsupervised with one of the machines could doubtless replace the software, modify results, etc. This is by far the hardest of the attacks that VITA identified, so it’s almost irrelevant given how severe the other problems are.
- And so on.
The amazing thing is that to find all this, VITA just scratched the surface, and mostly used off-the-shelf open source tools – nothing special. They didn’t have access to source code, or any advanced tools. Or said in other words, anyone within a half mile could have modified every vote, undetected.
So how would someone use these vulnerabilities to change an election?
- Take your laptop to a polling place, and sit outside in the parking lot.
- Use a free sniffer to capture the traffic, and use that to figure out the WEP password (which VITA did for us).
- Connect to the voting machine over WiFi.
- If asked for a password, the administrator password is “admin” (VITA provided that).
- Download the Microsoft Access database using Windows Explorer.
- Use a free tool to extract the hardwired key (“shoup”), which VITA also did for us.
- Use Microsoft Access to add, delete, or change any of the votes in the database.
- Upload the modified copy of the Microsoft Access database back to the voting machine.
- Wait for the election results to be published.
Note that none of the above steps, with the possible exception of figuring out the WEP password, require any technical expertise. In fact, they’re pretty much things that the average office worker does on a daily basis.
Was it really necessary to decertify immediately? As quoted in the Washington Post, Richard Herrington, secretary of the Fairfax City Electoral Board said “No matter how much time, money and effort we could put into a device or a system to make it as secure as possible, there is always the possibility that someone else would put in the time, money and effort to exploit that system”. Herrington is wrong – this isn’t a remote possibility, but an almost certain reality. A high school student could perform undetectable tampering, perhaps without even leaving their bedroom. In short, the SBE’s decision was right. Now that the information is public on just how weak the systems are, it is inevitable that someone will try it out, and it will take only minutes to manipulate an election.
Why doesn’t the vendor just fix the problems? Well, they went out of business five years ago. Their domain is now owned by a Chinese organization of some sort. And even if they were still in business, this isn’t a matter of fixing a few problems – what VITA found was undoubtedly the tip of the iceberg.
Bottom line is that *if* no Virginia elections were ever hacked (and we have no way of knowing if it happened), it’s because no one with even a modicum of skill tried. The Diebold machines that got lots of bad press a few years ago were 100 times more secure than the WinVote.
Replacing these machines in time for a primary in two months will not be easy. I feel for the local election officials who will have many sleepless nights to replace the WinVote systems. But once the State Board of Election learned just how vulnerable they are, they had no choice – it would have been criminally negligent to continue to use a system this vulnerable.
[Updated 4/15/15: Added step to possibly provide an administrator password.]
[Updated 4/15/15: Added a mention that the vendor is out of business, and so can’t fix the problems.]
Software engineer here.
The discussion that “any system can be hacked” and “every system is imperfect” is crazy.
The description of vulnerabilities is extraordinary. The idea of open ports on any embedded device is crazy. The use of a single fixed unchangable password on wifi and a superuser account is incredible.
Do you have an ADT Pulse system in your house? I worked on that. It has a gateway and a touchscreen. I personally was in china making sure that system has a reasonable security system with large randomly generated keys using WPA2. They didn’t want to do it, I had to bet an engineer $100 (US) that he could get a driver up and running in 48 hours, which he did.
We took home security seriously, and that’s one person’s house, not an entire election.
It might be hard to explain how bad that report is to a non professional. Maybe like buying an armored car, filling it full of cash, driving it to the bad part of town, unlocking the back, leave the keys running, and shout “HEY! WILL SOMEONE WATCH THIS LARGE PILE OF CASH WHILE I TAKE A LONG LUNCH?”
Yes, any armored car can be stolen — but you don’t just leave them unlocked in the middle of a road. Especially when you’re a hired and bonded guard company. As the man in the movie said: “Inconceivable!”
I also happen to have worked on a small company (we had about 6 people) that innovated in mobile wifi. I took a mass market $40 device and wrote a few key scripts that would automate attaching to networks and downloading and sending emails. I could literally attach to a garden variety access point inside a house and get a few recent emails — while driving at 70 MPH — from houses with open networks near the freeway. With a lousy embedded antenna. ( this was only for demonstration purposes, our business plan had us partnering with commercial wifi companies so you just had to drive past a starbucks or macdonalds and your email would be downloaded & fresh when you next looked ).
The matter of time? Could you drive slowly through all the polling places in my town during the course of a day? You bet, although you’ll have to do it toward the end of the election day. You can’t change all the vote in a polling place, obviously.
USB sticks were used? Then an actual sophisticated hacker could infect the embedded device, cause the inserted USB stick to be infected, and infect the central vote counting machines. This system is not sophisticated enough to withstand that kind of attack – you can build that.
The sanctity of the vote is supposed to be so great that we have disenfranchised hundreds of thousands of Americans because of a claimed threat of people falsely voting, when that attack would take too much time, be to easy to detect, and has never been seen to have happened – yet we pass voter ID laws all across America, to protect each sacrosanct vote.
My country has a virtually unhackable system… we get a piece of paper with the candidates name on it and we put an X next to the one we like. You should try it some time!
This is a matter of political will. The GPL / Open Source solution systems have been available for build out- but the officials have stood silent with the current vendor models. The Microsoft / Intellectual property lobby has remained steadfastly in charge.. Now this. For more info- see http://www.cavo-us.org
Open-source software is a great idea, and the Microsoft/intellectual property lobby should be ashamed of itself. In the meantime, if we demand our election officials in those jurisdictions with op-scans begin routinely to audit the voting-machine output, we will be able reliably to protect our final election results from electronically miscounted results regardless of whether the software is proprietary or open source, and regardless of whether it it pristine or compromised. We will also protect our final election results from voting-machine output compromised by paper-lint dust bunnies, overheating, and other unpredictable malfunctions.
Even after we win the open-source programming battle, we will still need to verify the machine output to ensure the open-source software operated as intended, so we might as well start now to implement the routine post-eletion audits that have been recommended since the inception of electronic elections technology.
The wonderfulness of more technology and better technology is and always will be limited by the extent to which we remain unwilling to check to make sure it works correctly.
Software engineer here: Absolutely no electronic voting machine is trustworthy unless it has the following minimum characteristics:
– Runs a publicly available open source operating system (i.e. running Windows is an automatic disqualifier) which can be inspected by any member of the public at any time
– That operating system should be as minimal as possible (i.e. any software, services, drivers not directly needed by the voting software should not be present on the machine at all, down to the OS kernel) – i.e. you can’t grab a handy Linux distro and throw it on a machine as-is.
– Runs on open-source hardware (say, Arduino or Raspberry Pi or similar hardware where any member of the public can examine all aspects of the hardware’s design)
– Available for inspection by the public – i.e. anyone with an interest to can verify that the hardware matches the spec, that the bytes of the OS are what they should be
– Produces a *paper audit trail* – i.e. electronic voting + paper that the voter can verify reflects their intended vote, and which can be audited by government or watchdog agencies
None of this is hard to do, or expensive. An open spec for an open voting machine is not a hard thing to do.
The general public tends to have the intuitive impression that voting machines will be safer if the software and hardware is secret. The exact opposite is true – it’s like encryption. Anybody can write something *they* can’t hack. What you want is something that nobody on the planet can hack, and you only find that out if the whole world can *try* to hack it.
If your polling place uses electronic voting machines, and they do not have ALL of the above characteristics, you should get an absentee ballot and use that instead. That has a much better chance of being counted correctly.
30-year veteran of public administration quality assurance here. Absolutely no human-run endeavor (and computers are run by humans, no matter what our wishful thinking might tell us) is trustworthy unless it has the following minimum characteristics:
Okay; I can’t go much farther in directly imitating your post–here’s my point:
The technological options you suggest are solid gold (though I’d substitute a voter-marked paper ballot for a machine-created paper trail). We should implement them. BUT: None of those technological solutions will make any difference if we are not able to deploy the human resources, skill, and procedures needed to produce their intended benefits. The one intervention that IS within our grasp is the actual performance of routine, transparent post-election verification of voting-machine output. It is not enough for either deterrence or detection to have a paper trail that CAN be audited. We need a paper trail that IS audited.
Open-source software is protective only if there are members of the public who are willing and able to inspect it. Given the difficulty I have as an election-integrity activist in Wisconsin getting volunteers even to observe the pre-election machine tests, I am pessimistic that knowledgeable programmers would come forth in every jurisdiction to examine the open-source software for each election.
And even if those citizen volunteers did come forward, their work will be worthless unless their OBSERVATIONS ARE HEEDED WHEN THEY DETECT ANYTHING WRONG. If you know of one jurisdiction anywhere in the US that has procedures in place to allow for citizens to come forward with that kind of feedback on elections administration AND BE TAKEN SERIOUSLY, please let me know. I have never seen it, and what I do see leads me to believe the citizens would be blown off and their observations dismissed or covered up. Exhibit A: Read through David Bjerke’s comments above, and notice how hard he fights back against the implications of a major, credible, professional, official study and a one-on-one exchange with an IT professional of the caliber of Jeremy Epstein. Using his comments above, imagine how Mr. Bjerke–very typical of the election officials I’ve dealt with–would respond to a volunteer citizen programmer who detected a flaw in the open-source software she had examined.
I think the most realistic, straightest path between where we are now and reliably accurate election results is to pressure our election officials to perform timely transparent verification of the voting-machine output routinely after every election. This won’t prevent the machines from producing miscounts–nothing ever can, really–but it will prevent the miscounts from being certified as final election results, and that’s the bottom line. For the moment, we can even write off the jurisdictions that don’t have an audit trail–as soon as the jurisdictions that do have an audit trail start to actually use it in audits, its value will be apparent and jurisdictions without paper trail (and with responsible election officials) will see the light regarding the need for open-source programming and post-election verification of voting-machine output.
The Pringles can antenna is merely illustrative. A more reasonable threat is a 6-foot dish antenna in a RV, camper, or box truck with a large window (glass window with cotton curtains or fiberglass panel) parked on the other side of the street or a half-block away. An antenna like that should give about 30 dB gain. If the range of the Wi-Fi were 100 feet for a device using a normal antenna to connect, such an antenna would extend the range to 500 to 1000 feet.
There’s usually nothing suspicious about such a parked vehicle.
Observer
Thank you for making that point. I used a Pringles can as a way to demonstrate the low cost and technical sophistication required. But you’re right, the threat could be more sophisticated technology. I think you’re understating the distance with a good antenna – I remember reading some years ago about accessing WiFi from a mile away with a good antenna.
How soon we forget… Please tell me that anybody here remembers the untimely demise of Michael Connel, who was set to testify in court against Karl Rove, et al, that he rigged the 2004 Ohio vote for Karl Rove, who received threats from Karl Rove, and who then had a mysterious and fatal light plane crash? I know the corporate media pretty well buried the story at the time, but anybody who has been paying attention to the issue of using computers to vote has usually heard about this. It seems either disengenuous or outright clueless to claim that there is no chance that voter fraud through the use of computers has already happened. Ohio 2004 exit polls differed from the tallies by nearly 6%. When even a 2% variance happens in foreign countries, we tell them they’ve had a fraudulent election, but here, it’s blissful silence and ignorance. There was a computer programmer that testified before a Congressional subcommitee that he had been asked before the 2000 elections, by the Florida Secretary of State, whether he could in fact write “vote-flipping software”. Another story that was dropped by our wonderful corporate media. What truly boggles my mind about this present discussion, though, is the “too much risk for too little payoff” argument. To political fanatics, there is no risk too high. For those looking only for payoffs, we are looking at control of State budgets often in the billions of dollars, and a Federal budget in the hundreds of billions. Is the argument that in a populous nation like ours, there is nobody immoral enough to do something illegal or unethical for even partial control of that kind of money? -Not to mention partial control of that level of regulatory power? And if we ever “graduate” to internet voting, that pool of immoral and/or unethical people would expand to include hostile foreign governments, well-heeled crime syndicates, and religious fanatics. The safest course would be to keep elections as local as possible, and as transparent as possible. It is no mistake, and no case of Luddite over-reaction, when so many people conversant with computer security say we should go back to paper ballots and hand counts, like so many other advanced nations still do. Are we really such children that we couldn’t wait for several hours, or even a few days, for election results, if it meant they were accurate and reliable?
A curious question.
Does the SBE keep track of the precincts these machines were used? I’m thinking yes, but I ask just in case.
It would be interesting to compare 90’s election data and compare to the 2000’s election data in the precincts these machines were used. Are you aware of anyone doing this?
I recall George Allen and Creigh Deeds elections in particular with many Virginian’s speculating whether the votes in key precincts were tampered with.
“laptop in a parking lot” is way too generous. Since 2007 at least (with the n800) people have been able to buy gadgets the size of a modern smartphone that connect to wi-fi networks, run user scripts, browsers, file-sharing software and so forth. With such a tablet or a smartphone capable of running a side-loaded app, or any of the dozens of models of battery-powered scriptable wifi hotspot boxes, a person could simply walk into a polling place to vote (or make inquires about voting) and let their device do the dirty work while they did their legitimate errand.
Hello mates, good article and good urging commented at this
place, I am truly enjoying by these.
Mr. Bjerke,
Can you be sure that the voting machines weren’t tampered with previous to their use? That would certainly address the ‘lack of time’ you claim as an impediment. This blog used to delight in taking pictures of unattended voting machines stored in the back halls of polling places – perhaps our host, Mr. Epstein should resurrect that practice?
Reminds me of a short story from a couple of years ago…
http://www.princetoninfo.com/index.php/component/us1more/?Itemid=6&key=7-25-12cohen
I agree with Dave that the machines should have been decertified when the vendor went out of business. As a long time IT professional, I never thought Windows XP was an acceptable platform for a mission critical real time system like voting. But in defense of the use of WinVote machines in the past, I’d like to emphasize one thing Dave mentioned. VITA had physical access to machines for an unlimited amount of time. While it didn’t take them long to find vulnerabilities, they didn’t have to worry about someone noticing a slowdown in processing or a disk activity light or a suspicious voter wondering about the person parked in the parking lot. Hacking the machines required a two step process–the first was access to machines to figure out the technology and plan the attack. No one has really addressed how difficult this would have been given the strict security of handling machines in storage, in transport and in the polling place. And this first level access would have to happen prior to the election if any large scale attack at multiple polling places was to occur. The amount of time and skill required to make just the first stage access would hardly be worth the risk of felony time in jail unless the results could be changed in an election. And most likely the total effort would only be worth the risk, expense and effort if the election was for some significant office. While we can’t rule out the possibility some group would think it worth the effort, I feel certain the people and systems in place would make it much more difficult than this report and discussion makes it sound. Again, decertification is the right thing to do, but lets not take one situation (VITA’s examination) and project it back into the past as if the two situations are the same. They are not.
I’ve provided technical support on other DRE and optical scan equipment since the DREs first came into use, and as a private citizen, I advised our locality against WinVote. For the past four years I’ve been the General Registrar in Bath County. All voting systems can be compromised, including plain paper ballots. The average person would be amazed at the things our part-time, under-paid, under-appreciated Officers of Election catch, and how sensitive our multi-level system of tabulation is to detecting errors. Good people can run good elections with less than perfect machines. No machine will ever be secure enough to protect the security of an election if we don’t train, respect and compensate the good people who manage elections.
Mr. Garrett, please see my response a few minutes ago to Mr. Bjerke – despite the good faith efforts of local election officials, I had the opportunity to buy hundreds of these machines a few years ago in Pennsylvania – and someone else, unknown to me, actually did. I have no idea what they did with them – do you? So getting access to the technology isn’t that hard. And who knows if there were other machines from other states (e.g., Mississippi) that were resold, or copies that were still in the vendor’s warehouse when they went out of business, or …
Relying on the secrecy of the devices is a losing proposition. If it worked, we wouldn’t see web sites hacked every day of the week – those systems are also secret, but yet they get hacked.
This is not to criticize the hard work of election officials, who are overworked and undercompensated. It’s a recognition of the reality of technology.
Mr. Epstein,
First, let me say again, I never was a fan of WinVote and I agree with the decertification and that it should have been done years ago. And I read your response to Dave. You are totally right about the lack of security in the disposal of voting machines. Last year, I personally bought an AutoMark machine from a locality through an online government auction site. I wanted the big case. I imagine lots of equipment is available either directly from local governments, or from the reclaim/resell houses which probably buy most of it.
My concern is the impression being given by yourself and others that any high school kid with a laptop and a Pringles can would be able to sit in the parking lot of a voting precinct and hack an election in fifteen minutes. I can say with certainty that not even a skilled hacker would be able to change the contents of a WinVote machine with only 15 minutes of wireless access to machines he/she had never seen before. And even if they did, the only way that would change an election is if the office was only on the ballot in that one precinct.
Let me address just a couple of issues. One, the Pringles can antenna was designed to go on a wireless router or similar device to increase the range to all connected devices. The design requires N-type antenna connections on the device. While it is possible to buy wireless cards for computers with external an antenna, there really would be no point since the Pringles can is designed for the higher power of a router, and better routers are available today which could be carried in the car. I would call the Pringles can a red herring. The range of access to the WinVotes would depend almost entirely on the power of the WinVote, its location in the building, what the building was constructed of, and all the factors which impact Wireless. No external antenna in a car, or even powerful router in a car, can change the level of signal coming from the WinVote.
Next is the idea of copying, editing and reloading the database while it is in use. Since I haven’t been into the WinVote hardware and software, I can’t say anything for sure, but I have a couple of well educated guesses. First, anyone coming into a 10 year old Win XP machine through wireless is going to impact the performance on that machine. I would guess it would definitely halt voting while any hacking was happening. And while old Access databases weren’t designed for really robust transaction applications, they did have database and record level locking. Since the WinVotes were designed to communicate and stay synced, there had to be some level of transaction locking of the database. At the very least, the database would be opened for the full time the WinVote application was in use. Anyone who remembers trying to copy a file in Win XP which was left open by an application which crashed can appreciate the difficulty one might have trying to copy, change, and replace the database while it is in use. My guess is Windows would prohibit the copy in either direction until the database was closed, which would require killing the WinVote application. Tell me the voter and OE wouldn’t notice this.
Again, I’m not saying the machines couldn’t have been hacked. But, I think the press and Mr. Epstein make it seem much easier than it likely is, and are also discounting the risk vs penalty, and finally underestimating the difficulty of impacting enough WinVotes to change an election. Could it happen, yes. Has it happened, I’d say no. And I think we do a disservice to the election community and to the voting public to make this operation sound so simple and easy as to call into question all the elections over the past ten years.
Decertifying the WinVote is a step toward assuring the integrity of elections in the future. Calling into question the integrity of elections in the past based on suppositions and Pringles cans seems ill advised.
Mr. Garratt,
While you do raise a good point about the possibility of detection, you should consider a few things:
1. Even though the devices might not be that powerful, with a good antenna you can still maintain a (weak, but still present) connection over a decent length; look at what these guys are doing in the back o’beyond: https://www.youtube.com/watch?v=hH8CORDow3o
2. Even though cinderblocks might block a signal altogether, keep in mind that if you have a number of polling places, you can pick and choose which ones to manipulate. Also, voting places might be optimized for outside interference, simply because polling stations tend to be close to the entrance of buildings. Poll workers might be a factor here as well, as they might choose to locate booths in a place where they can use their cell phones.
3. Even though it might take longer than 15 minutes to set up the first time, an attacker might have a way to case the polling place initially by either going in to vote themselves (or going in but pretending they have the wrong place), or by going to use an alternative function of that building (If your polling place is in a library, people can be within 30-50 feet of the booths for an extended period of time).
4. Even though these machines might be detectable if they had to reboot, an attacker might be able to exploit a lull in voting to try their exploit. If the performance of these machines is already questionable, some sort of sluggishness might be expected. But in any case, I don’t imagine that a simple table of vote totals or records would be too resource intensive.
I suppose I need to start looking into Pringles can technology when I go on trips, and based on these assertions, I’m surprised I don’t see more of them around used in this manner. Yes, “security by obscurity” was one part of my argument, and a decent argument considering the turnout of most of the elections in which these WivVotes were used. However, the other and more compelling argument is the process that happens beyond the voting equipment. From an election administrator’s vantage point, all voting equipment is hack-able. Nothing that we will ever get will be certified 100% secure from all outside forces. Therefore, we must build processes above and beyond the voting equipment to ensure that any nefarious acts can be caught to the best of our ability. We all agree that that these machines were flawed, but can we also agree that all machines ever used by election administrators will also be flawed to a certain degree? That is what makes these arguments so challenging. You are not giving any credit to the security efforts employed by the election administrators above and beyond the voting equipment. By not doing so, there is an insinuation that we are not doing our jobs, as one of the commentators to your post observed above. They will continue to have little faith if election administration lives and dies by only the security protocols of election equipment that needs to be procured by outside vendors. You may criticize the equipment to your heart’s content, but please take account of the processes enacted by the election administrators above and beyond the voting equipment to protect the integrity of the election before alleging the possibilities of past elections being compromised. And if you don’t know what those processes are please feel free to attend a Logic & Accuracy testing and/or Canvass before and after an election. The administrators will be happy to inform you.*
Thank you for allowing this discussion. I think it has been an important conversation to have.
*Assuming you are there for purely educational reasons. They will not necessarily be happy, though they will still inform you, if you are looking for a way to condemn the overworked and under compensated election administrator and the service they provide.
I’ve been an election officer (pollworker) in Fairfax County VA for about five years, so I understand how hard election officials (and election officers) work, and how understaffed and underpaid they are. I’m not in any way criticizing them. These machines were foisted on them, and no one at the state or Federal level really looked at how bad they were. The vendor did a good job of selling snake oil, and because the election officials are experts at elections and not experts in security technology, they didn’t know it was snake oil.
Election officials put in a lot of effort to making sure elections are accurate. The problem is that none of the measures are effective against a system this broken.
When I was a kid, the local pizza place had a sign “we have a deal with the bank – they don’t make pizzas, and we don’t cash checks”. In assessing the security of these machines, you’re the expert in elections, and I’m the expert in security. And these machines are not secure. I never said you can figure out how to break one of these in 15 minutes; but once the first person figured it out (and we have no idea if that was VITA or someone else), anyone else can get in within 5 minutes. And figuring it out the first time isn’t that hard – try out the experiments on primary day, and on general election day you’ll be ready to subvert the system.
My hope is that all concerned (nationwide) have learned about the fragility of voting technology, and will back off the headlong rush we’re seeing in many states towards internet voting. We have no idea whether those systems are as fragile as WinVote, because no one is looking at them, in the same way that no one looked at WinVote for over a decade. Let’s make sure we’re not having this same conversation in a few years about elections that may have been undetectably compromised by internet voting.
David and Charles,
We’re getting hung up on technology terms here, but I’d like to add that much less of his proposed exploit is hyperbolic than you’d think. Pringles can antennas are somewhat well-known within the tech community; they’re the modern-day equivalent of a jury-rigged ham radio antenna, and are dead easy to make with a piece of TV cable, a soldering iron, and a few parts from radioshack. It’s a quick and simple project you could make at a kid’s summer camp.
More importantly, Jeremy isn’t kidding when he says that the wireless encryption is breakable by a high-schooler. I know, because I did something similar in 10th grade when my grandmother forgot her wi-fi password. It took 5 minutes, including setup time.
Mr. Bjerke, you’re absolutely right that it is difficult to create a system that is secure, reliable, and accessible to voters. But there are nuances. If a 15 year old could have broken into this network almost a decade ago, imagine what an adult could do today with a $40 laptop, sitting out in his car.
The only hope is that no one is petty and malicious enough to have changed the results of local election races in Virginia. As a public official who may have sat through a public comment session or two, I’m sure you have met enough petty and malicious people to know better.
I live in northern Virginia. In view of the above information from Jeremy Epstein, I have little faith in the results of previous elections in which these machines were used (apparently more than just local elections). Given the incompetent response by Dave Bjerke (if that is really Dave Bjerke), I have little faith in the administrators of our voting system.
On the other hand, it is only our democracy that is at stake (or faith in our democracy, which is already shaky).
Best regards,
Victor Scott
Yes, it is really me and you can find me here: http://www.fallschurchva.gov/vote or on twitter @davebjerke if you have any specific questions concerning your lack of faith in Virginia election administration. All of us are happy to discuss your concerns.
Mr. Bjerke, you’re partially right that the attackers would have to be out in the parking lot of each precinct. But luckily (?) for us, that’s good enough, if you have a long-term view. Once I’ve connected to one machine, I can put software on it that infects the USB drive. That USB drive will get carried back to the central election office, where it’ll get put into the main server to tally vote totals. And it can infect that machine. And when the next election is programmed, that infection will spread to all of the other machines. That’s how a virus/worm operates.
Harder certainly than the simple case I described of doing it one machine at a time, but just a few weeks work for someone who’s interested. And of course this whole attack can be completely automated, so an attacker with just a little bit of work would only have to drive past each polling place, park for a minute, launch the attack against each machine in the polling place, and then move on to the next location.
I didn’t know about many of the issues that VITA found. But I knew in 2008 that they were using WEP, because I questioned the vendor in an open session of a legislative study group, and pointed out that WEP was already by then known to be fundamentally broken. And I questioned the SBE’s outside expert on whether he had done any sort of penetration testing, and he said he didn’t know how. And several other items as well – it was clear to anyone who looked that they were broken; it just wasn’t clear how bad it was.
I am very pleased that the new administration led by Edgardo Cortez at the SBE has turned over a new leaf, and investigated the problems, and responded by decertifying. But to say the problems were unknown is only because past state and local election officials refused to examine the problems for nearly a decade, and insisted they needed to buy more. As recently as a few days ago, local election officials were insisting that there was no problem. It was only when presented with incontrovertible proof by VITA that there was a change of heart.
Mr. Epstein,
Having been in charge of these machines, I think you have a misunderstanding of how WinVotes work. While you are correct that USBs could become infected with a virus/worm, there is no main server that tally’s votes on these machines in the central office. There was a capability to use a modem to send results to a central server but we never used that function when I was there. The only time there was a main server is when a specific WinVote is picked to be the “Master” in a specific polling place. So again, the person would need to go to each polling place, hope that the signal was strong enough (doubtful?) and then perhaps, under the perfect conditions, this person or group could be successful, and if executed perfectly, all precincts would show numbers and percentages that made sense to the election officials so there was no concern (also doubtful). This assumes someone understood all of this prior to the VITA report. You have been following this for a long time. Other than the manufacturers, who would be able to come up with this type of plan and execute it without having access to these machines to test their work for a few weeks? The entire hypothetical is completely improbable prior to this VITA report.
Further, after that election is over, the WinVotes are cleared and we conducted 100% testing before the next election. Our scripts to test these machines are such that any virus/worm that changed our scripts would be discovered. At least in Fairfax County, none ever were. I have the utmost confidence in my fellow administrators that none were ever found in their localities either. Ask any representative of the Democratic or Republican parties that attended the Logic & Accuracy tests that we conducted prior to every election.
These machines were certainly flawed, as the VITA report shows. But election administrators have the necessary procedures in place to flesh out this kind of fraud and these kind of flaws beyond the capability of the machines. In my personal opinion, the WinVotes should have been decertified after the vendor ceased to exist. Now that this VITA report is out, there is clearly more than enough reason to decertify these machines. My only point is that of perspective: It is highly unlikely and improbable that any election in Virginia was affected with the kind of fraud you say was theoretically possible before the results of the VITA report was published.
Mr. Bjerke, my understanding was that the official totals from each machine were copied via USB stick, after the preliminary results were phoned in. But perhaps I’m confusing WinVote with one of the other brands on that point.
Regarding difficulty of connecting to a WiFi signal, a Pringles can has been shown to be a very effective antenna, allowing access to signals from quite a distance. I explained this to the legislative commission in 2008. It’s not my invention, just explaining the reality of technology.
Regarding improbability, 30 years of experience in security tells me that “security by obscurity”, which is what you’re describing, provides minimal protection. This is well understood by everyone who works in the security field. I’ve often heard the suggestion that “no one but the vendor could perform this attack”; even assuming that none of the machines have been obtained by an adversary (*), it’s the same argument used by many over the years to explain why their websites or other computer systems can’t be hacked – until they are. Just ask Sony.
It’s irrelevant now, but it’s well understood how to completely bypass the protections used in election offices such as the “100% testing”, simply by reprogramming the memory where the software is stored. How difficult it is differs based on the design of the system; given what we now know about the primitive design of the WinVote, such an effort would be something well within reach for an undergraduate student, and certainly for any professional programmer.
I stand by my statement: if these machines and elections weren’t hacked, it’s because no one tried.
(*) About five years ago, Pennsylvania was surplussing their WinVote machines. If I was willing to buy them all (I think 500 of them, but not sure), I could have bought them cheaply. Unfortunately I couldn’t find enough warehouse space to hold them all until I distributed them to researchers, and the government wasn’t willing to sell me a partial lot. Had it not been for that quirk, every researcher who wanted one could have had them, and these breaks would have been discovered years ago. I don’t know who actually acquired them – perhaps someone else with nefarious purposes did, and has been hacking the elections without our knowledge. We’ll never know – and there are no logs in these systems to give a hint.
“Other than the manufacturers, who would be able to come up with this type of plan and execute it without having access to these machines to test their work for a few weeks?”
There are a fair number of people who would be skilled enough to do this all in one day, while the election was occurring. That task is above the “script kiddie” level, but not by a large amount. A five-or-six-figure budget would go a long way towards finding that kind of skill.
The flaws demonstrated here are enough to show that someone who didn’t mind being detected could absolutely destroy all data on the devices near the end of the day. That kind of chaos and confusion might have it’s own appeal to a certain element.
Perspective is necessary. Yes, the security issues VITA found are serious. The seriousness is compounded by the fact that we now know about them. Before the VITA report was published on April Fool’s Day, we did not know about the seriousness of the security issues. Not even Jeremy Epstein knew, and he is an expert in technology security who has been railing against these types of machines for many years. So now that the report is published, decertification is the right course to take. However, for perspective, this does not mean that all, or even any, of the elections in which the WinVote system was used were flawed.
Keep in mind, that as easy as Mr. Epstein claims it could be for someone to hack a WinVote from the parking lot of a polling place, manipulating just one machine would likely not affect the outcome of an election. Election administrators and officers ensure the integrity of the vote by taking specific procedures during the tallying of the results and during canvass (which is like an audit) and will notice if the results of one machine are vastly different from the results of the rest of the machines. That means that the hacker in the parking lot would need to hack all of the WinVotes in the polling place. That could be anywhere from three to twelve machines and their percentages would need to be similar to get past our own procedural checks. And that is just for one polling place. That means, in order to rig an election, another person would have to be out in the parking lot of another polling place doing the same thing. If the results were off, election officials conducting the canvass would notice the percentage differences from polling place to polling place. So while one person with “a modicum of training could have succeeded” many people with the same modicum of training would be needed to also organize and essentially conspire, further breaking the law, in order to rig the election without detection. Also something to keep in mind, is that this is wireless technology that is over a decade old. I don’t think it’s better than our current wifi technology which we already have problems trying to use from outside of a building. The only elections this could have been successful are ones that literally only use one to maybe three polling places. Basically, since we’re talking about Virginia, it could only have happened in town elections without getting caught. But if you have ever studied the history of town elections, you’d know that most have trouble even getting enough candidates to run, much less having a contest worthy of breaking the law in this way. But for the ones that did have high turnout contests, I am curious what power a town council has that would be worthy of this James Bond style effort of very high risk for very little reward. No, the probability that any election was affected by these WinVote security flaws is as infinitesimal as voter identity fraud. But now that the security risk is known, the Virginia State Board of Elections has taken the right course of action towards the future of elections in Virginia.
– Dave Bjerke, Director of Elections and General Registrar of Voters for the City of Falls Church
I was also the Voting Equipment Custodian in Fairfax County from 2007 to 2009 when I was in charge of all aspects of approximately 1200 WinVote voting machines.
David,
The very fact that this machine was produced with unchangeable passwords of “admin”, “abcde”, and “shoup” (the company that preceded Advanced Voting Solutions) is outrageous in itself. Having my Comcast router come with a default password of “admin” is just stupid. Having machines charged with the all important task of counting votes to determine the leaders of our state and country have “admin” as a password defies all logic and reason. The auditors cracked their password in ten seconds!
Add to this the sad fact that Advanced Voting Solutions chose the WEP security protocol- which is so unbelievably crackable, that the IEEE banned it in 2004 and NO ONE should be developing wireless devices with it at all, is mind blowing.
Along with accuracy and ease of use, voting machines need to have absolutely have the best security practices and methods available at the time to guarantee the integrity of the vote and prevent tampering.
It appears Advanced Voting Solutions didn’t even try to secure their hardware or give security a passing thought. Reckless and irresponsible behavior that can threaten the foundation of the American voting system.
Read this article, it may change your opinion a bit. http://arstechnica.com/tech-policy/2015/04/meet-the-e-voting-machine-so-easy-to-hack-it-will-take-your-breath-away/
Chris Kozlik, Average US Citizen
David,
Your assertion that large scale manipulation would require a large conspiracy is incorrect.
With under $50 of hardware, a moderately skilled individual could build a box the size of a cigarette pack which automated the entire process and only needed to be hidden in wifi range of each polling place a before the election. This isn’t James Bond stuff, it’s more high school prank level. Changing votes at hundreds of polling places appears to have been well within reach of a motivated individual, never mind a someone with the resources of a major political campaign or criminal organization.
The insecurity of the machines also suggests that there were many other opportunities to compromise the machines en-mass, particularly for insiders.
The State Board did the right thing. It is unfortunaley that some of the local election officials (Fairfax City) fought this and want to continue to use there machines – even after they have been decertified.