Go to HHH home page.
Election Academy

Everything I Need to Know About Ballots I Learned In ... GRAD SCHOOL? Readability as Usability

Bookmark and Share


[Image courtesy of flickr user stonehouse]

Pew's latest Election Data Dispatch looks at a new journal article by Shauna Reilly and Sean Richey which analyzed state ballot language for readability.

More specifically, the authors took state ballot language for initiatives from 1997-2007 and scored it using the Flesch-Kincaid scale, which assigns a grade level as a proxy for readability.

The results are somewhat surprising. Pew created an infographic of the results - and if you click through you'll see that every single state's ballot language exceeds the average U.S. 8th grade reading level - and over 60% required a grad-school reading level or higher. Indeed, four states (NM, MN, CO, SC) require a reading level higher than a doctorate(!) to understand.

Lest you cluck your tongue at the state of the American educational system, remember that for most voters - even that tiny handful those with multiple doctorates - the subject matter of initiatives is outside their everyday experience, and so you want the ballot language to "stay out of the way" of the substance so that voters can understand what they're being asked and choose accordingly. This is not unusual; many states require certain documents (e.g., insurance) to adhere to an 8th grade reading level for the same reason.

In other words, readability - i.e., the ease with which an individual can navigate written language - is just as much a part of the science of usability as layout and visual design, subjects I touched on last week.

Achieving readability is harder than it looks; state statutes and legislative language often create barriers by imposing impenetrable language which election officials and ballot drafters are powerless to change.

Fortunately, Flesch-Kincaid readability is easy to measure - I just scored this post using Microsoft Word (instructions here) and discovered that it's written at just above a 12th grade level. That kind of information could be helpful to an election office pleading with policymakers to ESCHEW OBFUSCATION!

Thanks to Pew for finding and sharing the Reilly/Richey article. The lesson for the election community is clear - don't be afraid to take a red pen to your ballot language before voters take a pen to their ballots.


  • Although it is undoubtedly true that ballots can be difficult to understand, using a readability formula such as Flesch-Kincaid is not the way to assess them.

    Flesch-Kincaid has many flaws that make it unsuitable:
    - designed for use on continuous prose, not the smaller chunks of text that are used on ballots
    - based on the vocabulary available to children, which is different from the vocabulary of adults
    - cannot tell you whether the concepts presented are meaningful to the people who are voting.

    To give a UK example: the sentence 'Are you in favour of 'free schools'?' would easily pass any readability formula. But that wouldn't expose the issue that the term 'free schools' is about the governance of schools and the way that their funding is administered, not about whether the children (or their parents) have to pay to attend those schools.

    The way to find out whether something, anything, is usable is to run usability tests on it. Not apply arbitrary formulae.

  • Of course - but given that such testing takes time and effort to do, a simple readability score can be one way to clean up language before usability is assessed.

    I'm certainly not suggesting that Flesch-Kincaid be the only assessment - in your "free schools" example the election official would presumably know what that means and might even be required to define it.

    Readability is quick and dirty but it gives one a diagnosis of potential issues. That's what I'm talking about here.

  • In research with Martha Kropf we have applied the same readability scale to the ballot instructions as one of our measures of ballot usability. We have examined over 1,000 ballots using these measures. While the average ballot in our sample is written at an 8th grade level, in some states the ballot instructions include a lot of legal language that makes them very difficult to comprehend. One of our earlier studies is here:
    More of our research on ballot usability is in our forthcoming book, Helping America Vote: The Limits of Election Reform (Routledge, 2012).

  • Thanks, David - just forwarded a link to that study on the @HHHElections Twitter feed!

  • The Washington Secretary of State's elections division is working mightily on readability, Design for Democracy styling, etc., but on the question of ballot propositions, that language is un-revisable as it arrives to us from the Legislature or citizens. Likewise, explanatory official ballot titles, effect and background statements, and fiscal impact statements from the state attorney general and the state budget office also are un-revisable. The education process needs to filter down to the attorneys and bill-drafters...

  • Understood; feel free to use this blog post - and, more importantly, the studies discussed - as a conversation starter with your "content providers" upstream.

  • I'm in passionate agreement with Caroline on this. Readability formulas may seem convenient, but they're not really reliable for the kinds of things that can appear in propositions or referendums. Better than nothing? Perhaps. Testing with real voters is always better.

    Two related things, though. The San Francisco Ballot Simplification Committee and a study I did with Ginny Redish for the National Institute of Standards and Technology on the language of instructions on ballots.

    The Ballot Simplification Committee in San Francisco is a citizens advisory committee that writes summary digests of all the measures on the city-county ballot. The 400-word digests appear in the Voter Information Pamphlet before each election. The volunteer committee members write the digests in plain, non-partisan, objective, neutral language in public hearings. (It's like Iron Chef for editors. I loved being on the committee and wish there was one in every state and county.) You can see examples on the San Francisco Election Department website. This link even shows the evolution of drafts for the November 2010 election: http://www.sfgov2.org/index.aspx?page=1668. Does it always succeed at getting to 8th grade reading level? Not always, but voters love the digests and participate actively in the public hearings when the committee is in session.

    Plain language does indeed make a difference in how voters perform on ballots. Ginny Redish and I compared "traditional" instructions on ballots to plain language instructions on ballots with 45 participants spread across 3 locations in 2008. NIST published the report in 2009: http://www.nist.gov/itl/vote/upload/NISTIR-7556.pdf. The plain language guidelines that came out of that work are here: http://civicdesigning.org/uncategorized/guidelines-for-a-plain-language-ballot/ Many of the guidelines apply to all types of writing, not just procedural instructions.

  • Thanks for chiming in - I am all for making usability a core function in election offices. But unless you or someone else can suggest a way for election officials to quickly assess readability that *doesn't* involve spending money and time they don't always have, I'm OK with quick and dirty shortcuts like Flesch-Kincaid.

    Again, I'm not suggesting that anyone rely solely on such a test to measure usability (for the reasons you and Caroline suggest) but I find it very hard to believe that doing nothing at all - which is a likely choice if the only alternative requires even a modest investment of time or money - is a preferable approach.

    Is there a Plain Language tool that is as quick and easy to use as the one described here? If so, I bet folks would *LOVE* to hear about it.

    Iron Chef is great - but sometimes people just want to reheat a frozen pizza.

  • Election officials can always use the LEO Usability Testing Kit to do a quick-and-dirty test that costs little or nothing.

    I don't think either Caroline or I said that doing nothing was better than using a readability formula. We just think that the users of them should be aware of the shortcomings of the formulas. Using them won't guarantee plain language. But it's a fine place to start.

    A plain language tool would be a great thing. I'll get working on that right away!

  • This is an awesome dialogue on a very important topic. In my opinion these usability issues combined with the additional usability questions regarding the voting systems themselves lead to far more issues than security issues that get so much attention (please don't take this to mean that security is not vital, just that experience shows me that usability rears its ugly head far more often).

    One of the major steps that EAC/NIST are attempting to take with the next set of voting system standards is requiring all documentation including pollworker documentation, election official instructions, and warning messages from machines to be in plain English. This means usability will be built in from the beginning.

    This made me wonder how would usability be built in from the beginning with ballot language. The answer is getting tools in the hands of the drafters (often not election officials) that they can use to draft this stuff. So the LEO usability test kit can only help so much with this issue. What we need is a resource that can be given to drafters as part of a kit supplied by SOS offices and LEO to help make the language more useable.

    Also, this topic takes on an entirely new dynamic when one thinks about making alternative language ballots more useable.

    Thanks for the discussion.

  • Leave a comment

    Humphrey School Sites
    Humphrey New Media Hub