Policy writing skills series: how good are your descriptive skills?
Image: @kanereinholdtsen
By Holly Jarman
We often hear that policy work requires a good command of quantitative methods. This is true for several key policy-related career paths. People seeking policy jobs in think tanks and research settings are often advised to take courses in econometrics, cost-benefit analysis or program evaluation methods, and you should certainly consider doing so.
But when it comes to policy writing, one of the most important skills you have to master is translational rather than directly analytical -being able to incorporate descriptive statistics into your work in a way that accurately conveys your desired message.
Using descriptive statistics in policy writing is deceptively hard. You’re trying to be brief, and also compelling, while remaining accurate. The writer not only has to make choices about language, but also about visual design and presentation of quantitative data. And your policy audience, as always, may not have a long attention span or specialist knowledge. Some of the most common mistakes include:
#1 Letting statistics make the argument for you
Many writers will place a graph or table in their brief, but don’t explain its significance in the main text, assuming that its message is self-explanatory. This can create confusion in the reader, who might gravitate towards the visual only to find it’s not that relevant to the main argument, or may not bother to read your text and come away with a different message than the one you intend. Don’t do this!
#2 Information overload
Following on from #1, sometimes writers assume, incorrectly, that adding more statistics will make their brief better. While using descriptive statistics can be really helpful in contextualizing problems and arguing for certain policy solutions, too many statistics can actually detract from the key message of the brief. In the worst cases, this leads to information overload, where far too many factoids and graphs confuse the reader and obscure the argument. You should always question whether a statistic or chart furthers your argument. If in doubt, throw it out!
#3 Including visuals that aren’t legible or are poorly labelled
Even well meaning readers with plenty of time on their hands get frustrated when they encounter a poorly labelled or illegible chart. There’s no better way to get someone to abandon your brief. First you should ask, can you read everything on the visual, including the title, axes, and any data labels? Without a magnifying glass? At arm’s length? If not, you need to make it bigger.
Once it’s legible, does it actually make sense? This is very easy to get wrong, and you should always plan on showing your visuals to multiple people before you finalize them. Some common mistakes include omitting a title describing the key point of the visual, giving the visual a title that does not describe it well, not labelling axes on a graph or segments on a chart, starting the axis of your chart at an odd or unexpected place, using unevenly spaced measurements on axes, trying to put too many data series in one chart so that it becomes confusing, putting more than one y-axis on a graph (just don’t), using a 3D data visualization on a 2D page (also don’t), failing to note the source of any data… You get the point. Oh, and when putting two charts next to each other inviting the reader to make a comparison, make sure they are using the same scale of measurement!
When using color in visuals, think about what the colors signal to the reader as well as their cultural and political meanings. For example, a reader in the US is likely to think of green as ‘good’ and red as ‘bad’. If the green line trending upwards on your line graph shows increasing numbers of deaths and the red line trending downwards shows decreasing numbers of recoveries, the reader might get confused. On a map, using traffic light colors might be read as you ‘approving’ of some countries or cities over others. It’s also important to use colors that don’t signify aspects of social or political identity, so be careful when you assign colors to countries, subnational governments, political parties, or groups of people.
You should also consider the format in which the brief will be distributed. Will it be printed for the recipient to read? Or will it be read on a screen? Or both? It’s a good rule to always check how well any charts and other visuals in your brief can be read in greyscale on a standard-sized sheet of paper.
Finally, consider accessibility. One issue is colorblindness. Red/green colorblindness, in particular, is a problem if your visual depends on the reader distinguishing between those two colors. Most mass market programs have easy access checks. Also be sure to include visual descriptors for people who cannot see visuals, such as tooltips.
#4 Using numbers that your audience won’t understand
Does your audience know what an odds ratio is? No? Then don’t include one. Do they understand what return on investment means? Really? If not, don’t frame your argument around it. If you’ve thought carefully about your audience and their likely expertise and experience, it should be easy to tailor your prose, but always err on the side of the simplest expression of your argument.
#5 Incorrectly reporting statistics
The last category of errors is perhaps the worst. Not only are errors in reporting descriptive statistics extremely common in policy briefs, but they are also not always immediately noticed by the reader, who may have no experience with using statistics. This can create all sorts of misinformation problems. Some examples of reporting statistics incorrectly include:
Presenting a percentage change, but not making clear how it was derived. If I tell you that co-pays increased by 20% between 2010 and 2015, then decreased by 20% between 2015 and 2020, they must have gone back to their 2010 level, right? No! A co-pay of $10 in 2010 would become a co-pay of $12 by 2015 and a co-pay of $9.60 by 2020. Using percentage changes is often misleading in this way, and you might want to think about other ways to convey the same information.
Presenting an average but not conveying total variation, or confusing median and mean. Suppose you state that the ‘average American’s income is $77,000’. That doesn’t sound so bad, does it? However, you’ve used the mean income to make this statement, and the distribution of Americans’ incomes is not even. The median income is a lot lower, and you don’t talk about the skewed distribution, so your statement is misleading.
Reporting expenditure, but ignoring the time frame. This bill commits Congress to spend $10bn on addressing homelessness? Great! But is that over one year or ten? Knowing the correct time frame might make a big difference in how much the reader likes this idea.
Omitting vital baselines and other relevant context when discussing risk. If I told you that eating deli meats with hot sauce doubled your risk of a heart attack, would you be concerned? Maybe. But what if I also revealed that your initial risk of having a heart attack after eating deli meats was really low? If only 1 in every million people suffered a heart attack after eating plain deli meats, and that doubled to 2 in a million when eating deli meat with hot sauce, how would you feel then? In this case, understanding the baseline level of risk really matters.
Avoiding errors like these comes down to your knowledge of basic statistics (so brush up!), your ability to think visually, and your ability to critically question your writing. Mistakes can often be avoided by adopting routine editing and reviewing practices, so make sure that you budget plenty of time to edit your work and have it reviewed by others. With practice, avoiding these mistakes should become much easier.
