The recount of the 2008 Minnesota Senate race gives us an opportunity to evaluate the accuracy of precinct-count optical-scan voting. Though there have been contentious disputes over which absentee ballot envelopes to open, the core technology for scanning ballots has proved to be extremely accurate.
The votes were counted by machine (except for part of one county that counts votes by hand), then every single ballot was examined by hand in the recount.
The “net” accuracy of optical-scan voting was 99.99% (see below).
The “gross” accuracy was 99.91% (see below).
The rate of ambiguous ballots was low, 99.99% unambiguous (see below).
My analysis is based on the official spreadsheet from the Minnesota Secretary of State. I commend the Secretary of State for his commitment to transparency in the form of making the data available in such an easy-to-analyze format. The vast majority of the counties use the ES&S M100 precinct-count optical-scanners; a few use other in-precinct scanners.
I exclude from this analysis all disputes over which absentee ballots to open. Approximately 10% of the ballots included in this analysis are optically scanned absentee ballots that were not subject to dispute over eligibility.
There were 2,423,851 votes counted for Coleman and Franken. The “net” error rate is the net change in the vote margin from the machine-scan to the hand recount (not including change related to qualification of absentee ballot envelopes). This was 264 votes, for an accuracy of 99.99% (error, one part in ten thousand).
The “gross” error rate is the total number of individual ballots either added to one candidate, or subtracted from one candidate, by the recount. A ballot that was changed from one candidate to the other will count twice, but such ballots are rare. In the precinct-by-precinct data, the vast majority of precincts have no change; many precincts have exactly one vote added to one candidate; few precincts have votes subtracted, or more than one vote added, or both.
The recount added a total of 1,528 votes to the candidates, and subtracted a total of 642 votes, for a gross change of 2170 (again, not including absentee ballot qualification). Thus, the “gross” error rate is about 1 in 1000, or a gross accuracy of 99.91%.
Ambiguous ballots: During the recount, the Coleman and Franken campaigns initially challenged a total of 6,655 ballot-interpretation decisions made by the human recounters. The State Canvassing Board asked the campaigns to voluntarily withdraw all but their most serious challenges, and in the end approximately 1,325 challenges remained. That is, approximately 5 ballots in 10,000 were ambiguous enough that one side or the other felt like arguing about it. The State Canvassing Board, in the end, classified all but 248 of these ballots as votes for one candidate or another. That is, approximately 1 ballot in 10,000 was ambiguous enough that the bipartisan recount board could not determine an intent to vote. (This analysis is based on the assumption that if the voter made an ambiguous mark, then this ballot was likely to be challenged either by one campaign or the other.)
Caveat: As with all voting systems, including optical-scan, DREs, and plain old paper ballots, there is also a source of error from voters incorrectly translating their intent into the marked ballot. Such error is likely to be greater than 0.1%, but the analysis I have done here does not measure this error.
Hand counting: Saint Louis County, which uses a mix of optical-scan and hand-counting, had a higher error rate: net accuracy 99.95%, gross accuracy 99.81%.
First of all, thank you for your very nice evaluation of the numbers of this spreadsheet. However, I’d like to bring your attention to a much larger statistical variance. As of Nov 4th, 2008, the MN Sec. of State reported 2,887,337 votes counted. Their totals also show 1,212,206 for Franken and 1,212,431 for Coleman. However, this spreadsheet now says that there were 831 more votes for Franken and 841 more votes for Coleman on November 4th. I assume this was the variance that happens between the counts of Nov. 4th and the “official” election reports that come a few days later.
But the major problem I have is that the recount now shows a total of 2,914,535 votes counted in the recount, leaving us with a difference of 27,198 additional votes found or an accuracy of only 99.058%. This variance troubles me greatly and I think overshadows the statistics you pointed out to above.
John Columbus, CISA
I am not sure where the myth of the printers “seem to have a high breakdown rate and need lots of mechanical care” originated. I had a standard off-the-shelf $100.00 Canon ink-jet printer that printed thousands of pages with never a breakdown and never needed mechanical care. In fact, every standard printer I have ever used has provided excellent service. Given that each precinct may average 200-300 voters it would result in a standard printer lasting for multiple election cycles. There was a rumor spread as a talking point by electronic voting system vendors regarding printer problems but that was connected to their resisting any paper record being generated by their DRE units. Check your county’s bill from the vendors for a printer from them. You will find they are greatly overcharging, as they do for all items that they sell to the election departments.
There is a group that has been working on an open source code voting system that would generate a paper ballot. It is the Open Voting Consortium located in California. Check their efforts out.
I am glad that the SoS in Minnesota has reported such great success. It would be a good exercise to do a Google search on optical scan errors and see what you might find. They have historically not been so accurate. Also keep in mind these are the precinct based optical scanners as opposed to the central count optical scanners. As happened in Humboldt County in California, the votes from the memory card that recorded the votes from the scanner when uploaded into the central tabulation computer were lost from one precinct. Turned out the manufacturer had a programming bug discovered 3 years ago that had never been corrected.
The problem is that printers seem to have a high break down rate and need lots of mechanical care in order to produce the intermediary ballot to be scanned. That problem is eliminated in the paper ballot/scanner sequence, and the Minnesota experience seems to suggest there really is a lower level of ambiguous ballots than critics had suggested.
Presumably the voter intent issue is corrected by those machines that actually show the voter on a screen what was scanned from the ballot and requires the voter to approve before recording the vote. In fact, it seems likely that it would also address the issue of ambiguous ballots, since (presumably) if the ballot can’t be scanned, the voter would be told. (I assume that the MN machines aren’t of this type.)
Given that such a machine would be rather straightforward to build, is there really any reason to consider any other kind?
Point taken, but I think that the AutoMARK is not quite what the first commenter had in mind. For one thing, the AutoMARK is not really a voting system you would want every voter to have to use – as I understand it, it’s a little slow. For voters with disabilities, there are good reasons to believe that it is as good or better than any accessible-tasked voting system on the market. The California Top-to-Bottom Review concluded that unlike most of the DREs marketed for accessibility, the AutoMARK is “substantially compliant” with the Help America Vote Act’s accessibility mandate.
http://www.sos.ca.gov/elections/voting_systems/unity_3011_accessibility.pdf
I think that the commenter envisioned something like paper ballot generators, which print a complete paper ballot the voter does not see until it is printed. There was such a system, called Populex, but incredibly, the printed ballot card contained numerical codes for the candidates!
http://www.populex.com/DPB_Intro.htm
No reason that you can’t have paper ballot generators that print real, plain-language ballots. Now, I don’t think that paper ballot generators are really necessary, and they would likely be expensive. You’d need a machine for every 250 or so voters, and you would probably have long lines anyway.. Where current optical scan systems are used, let them be used until they wear out. But where the current yucky electronic machines are used – well, get rid of them ASAP. And since Populex is broke and there are not many new systems in the pipeline (or are there??), we should not wait for fancy paper ballot generators to be developed.
Well, those exist too, they’re called Ballot-on-Demand systems.
Ballot on Demand systems like the ones they have in Florida print blank optical scan ballots that voters then fill out…. I don’t think any of them generate a completed ballot.
http://www.zwire.com/site/news.cfm?newsid=20085976&BRD=2150&PAG=461&dept_id=403301&rfi=6
I think the original commenter had in mind a paper ballot generator that does not require inserting a pre-printed ballot, does not tabulate, and does not require further marking of the ballot after the ballot is printed.
I was thinking that since the paper trail could be such a huge deal, why not have the electronic voting systems print out optical scan sheets? You can have an example sheet that shows where each vote goes for candidate, and that goes into the scanner. I’m sure there are other problems with this, but it seems like a logical next step to me.
Actually, that’s exactly what a class of voting systems, called Ballot Marking Devices, does. Google for the AutoMARK.