Go to HHH home page.
Election Academy

New Equation for Voting Technology: Auditing > Testing?

Bookmark and Share


[Image courtesy of Joe Hall]

Berkeley's Philip Stark and David Wagner recently shared a paper they have submitted for publication entitled "Evidence-Based Elections". While subject matter is highly technical, the authors do a nice job of making it accessible to the informed layperson - and tucked into the piece is an observation that could significantly revamp the approach to voting technology at every level of government nationwide.

Stark and Wagner start with this assertion: "an election should find out who won, but ... should also produce convincing evidence that it found the real winners - or report that it cannot." Working from that premise, the authors describe various recent elections where voting technology failures created controversy about the validity of the results.

Some of the blame, they suggest, can be laid at the continued reliance of the field on a testing and certification process aimed at identifying and screening for problems before a given technology can be used. Indeed, they note that "the trend over the past decade or two has been towards more sophisticated, complex technology and much greater reliance upon complex software--trends [to which] that the voting standards and the testing process have been slow to react." Moreover, the continued consolidation of the voting equipment market and slow development of the federal testing process has resulted in a high cost of testing that serves as a barrier to entry and innovation in the field.

Stark and Wagner suggest that the better approach to take is an evidence-based "resilient canvass framework" approach that examines how voting technology actually performed - with "evidence" defined using the following formulation:

evidence = auditability + auditing

Auditability, they say, is the ability to "produce a trustworthy audit trail that can be used to check the accuracy of the election." This audit trail, then, would be subjected to a two-step auditing process aimed at validating the results:

a compliance audit and a risk-limiting audit. The compliance audit checks that the audit trail is sufficiently complete and accurate to tell who won. The risk-limiting audit checks the audit trail statistically to determine whether the vote tabulation system found the correct winners, and, with high probability, corrects the outcome if the vote tabulation system was wrong.

The real bombshell in the piece, though, is the suggestion that the proposed approach is not only preferable to testing and certification but can be effective even if such procedures did not exist or weren't used:

Currently, certification does not serve the interests of the public or local elections officials as well as one might hope. It erects barriers to competition, increases acquisition and maintenance costs, slows innovation, and makes risk-limiting audits harder, slower, more expensive, and less transparent. And risk-limiting audits provide more direct evidence that outcomes are correct than certification can provide. Requiring local election officials to conduct compliance and risk-limiting audits rather than requiring them to use certified equipment would give them an incentive to use the most accurate (and most easily audited) tabulation technology available, because more accurate initial counts require less hand counting during the audit, reducing labor costs and allowing the canvass to finish sooner ... Using voting systems that are designed to support efficient auditing--whether those systems have been certified or not-- can substantially reduce the cost of collecting convincing evidence that the official results are correct. [emphasis added]

This sentiment - favoring auditing over testing - is slowly gaining favor in some circles, and could have a huge impact on the way in which voting technology is developed, marketed and maintained by public and private actors alike.

Kudos to Stark and Wagner for a provocative piece - election geeks of all kinds should add this one to the reading pile, preferably near the top.


  • Thanks for highlighting this, Doug. I certainly hope that it has a major impact on election policy and technology. Paper ballots are critical, but we also need to look at a sample of them by hand to check up on the equipment and procedures - good audits are critical also.

    And there is an effort underway in Colorado to pilot this approach and adopt it if it proves out. We have first-hand experience with the benefits of audits, and the failures of certification. We've done innovative audits starting in 2004 in Boulder County and did the first risk-limiting audit outside of California (http://bcn.boulder.co.us/~neal/elections/boulder-audit-10-11/). In contrast, the state lost a lawsuit due to flawed security and inadequate certification of systems from several vendors. So we adopted a flexible framework for voting system testing and approval, and a state-wide requirement for risk-limiting audits by 2014. Pilots are ongoing, as part of the Colorado Risk-Limiting Audit grant from the EAC. See more at http://bcn.boulder.co.us/~neal/elections/corla/

    I highly recommend the presentation Philip Stark made at the Colorado Best Practices and Vision Commission on Dec 14. You can get his slides from


    and listen to the audio there also. There are several options for open source software to do state-of-the-art, easily auditable scanning and tabulation. The commission likes the approach, as do many other lawmakers and election administrators, including the secretary of state.

  • The paper by Wagner and Stark is a fantastic read and does a great job of challenging some of the beliefs of the current voting system testing and certification structure. As someone that has worked in the testing and cert. world and now works at the state level I enjoyed the concepts the paper pushes forward.

    Auditing has become a vital component of election administration. The amount of data collected, procedures improved, and confidence imparted by these audits cannot be underestimated. However, I do not believe it is the panacea that the paper holds it out to be. As the paper notes, security is virtually impossible to test completely. You can mandate and prove what a system should not allow or do but you cannot fully test that the system is "secure". The EAC's testing program has long recognized this and continues to take steps to limit the cost of testing while getting value out of those areas that are most effective.

    As the paper recognizes what audits can't tell you that requirements, testing, and certifications can is the level of functionality and usability of the system. For instance, in the case of the overvote problem recently identified in New York. While an audit can show that thousands of ballots were overvoted it cannot help to prevent those overvotes nor can it allow the person to correct the problem. Well written requirements regarding usability and functionality, that are well tested (at all levels) can help to mitigate these types of human error problems that lead to thousands, or in this case tens of thousands of votes to be called into question.

    In pointing out the shortfalls of DRE's the paper states, "...there is no way to produce convincing evidence that the electronic record accurately reflects the voters' intent." The fact is that audits cannot do this either because there is no way to know that intent after the fact. Who knows how many of the voters in NY intended to overvote? Proper usability and functionality would have at the very least greatly improved the odds of capturing the voters intent, while auditing did nothing but show a potential problem after the fact, which is of value for later elections but does nothing for that current election.

    Having requirements that help to build usability into the system and functional requirements that give the election officials and voters confidence that the system will function at a certain level help to mitigate risks before they happen instead of simply detecting them after if at all via an audit.

    An election official's job is to limit uncertainty and then deal with the the uncertainty that arises. Testing helps to provide a certain level of certainty (not absolute certainty which is impossible). Auditing provides a different level of assurance which is of value but occurs after the fact.

    The fact is that there is room in this world for both certification testing and audits, because both bring something valuable to the table. The key is identifying those requirements that bring the most bang for the buck and testing them well and eliminating those which bring little value and cost a lot (see line-by-line source code review).

  • Thanks for this excellent comment. I agree completely that both certification testing and audits are necessary for the reasons given.

  • BTW, that image credit should go to me:


  • Leave a comment

    Humphrey School Sites
    Humphrey New Media Hub