How not to screw up poll reporting

This 1A tale from a favorite major metropolitan daily brings a few immediate thoughts to mind:

Polls: Democrats
have real shot of
winning Senate

1) Gee, isn't this more or less the exact same story you ran three weeks ago? The one on 8A Oct. 2 that began "Democrats are within striking distance of taking control of the U.S. Senate on Election Day, a series of new polls for McClatchy Newspapers and MSNBC showed today"? Are you just going to keep running it until you get it right?
2) At least this iteration doesn't openly ignore all the rules -- like, say, sampling error -- that make legitimate polls legitimate (and made the previous story a crock, as Doug so deftly noted).
3) But since it doesn't bother to give us any of the statistics that would demonstrate that it might be playing by the rules, it's hard to tell whether there's been any improvement at all.
(OK, 4, that hed preposition: You have a shot "at" winning something. You have a shot "of" something potent that makes you forget how teemortal dumb newspapers can get around election season. But we digress.)

Any second now our good buddy Strayhorn (with whom we started playing journalism back when baseball had the good sense to be done by now) is going to suggest that polls, being in essence horoscopes with better creditlines, aren't news. He has a point, sort of. A badly run or badly reported poll is the moral equivalent of a horoscope. So to cut down on the number of copy editors burned at the stake for witchery, here are some suggestions -- and some out-and-out rules -- for proper coverage.

First, the principles:

I) Only polls based on legitimate probability samples can be generalized to any form of public opinion. All others, including but not limited to call-in, click-to-vote and man-on-the-street, are crap by definition and must be ignored or ridiculed. If you don't want to say "crap" in the newspaper, you can say "nonprobability (convenience or self-selecting) samples."

II) A poll can only be generalized to the population that is sampled. Residents of the capital cannot be assumed to stand for residents of the whole state. "Registered voters" and "likely voters" are not the same thing. Undergraduates do not represent the entire adult population. &c &c &c.

III) Polls measure only what they measure and answer only the questions that they ask. A question about attendance at religious services can provide a reliable measure of self-reported attendance at religious services. It can't, doesn't and never will say anything about whether the population in question is "faithful" or "takes religion seriously." "Do you approve or disapprove of how the president is doing his job?" is not the same question as "How do you intend to vote?"

IV) Polls do not "prove" or "confirm." They do not say what's going to happen. They provide a snapshot of a population at a specific time. That's all.

V) Don't draw inferences by comparing separate polls unless they ask the same questions of samples drawn from the same population.

VI) A minimally accurate report on a poll will say how many people were interviewed and when, what population they represent, who conducted the poll (and/or who paid for it), and the margin of sampling error and confidence level.

Now at this point, some reader-friendly [insert your favorite term here] is going to emerge from its glass office and complain that we're letting the numbers get in the way of the story. Not so. The numbers are the story; they're what distinguish polls from horoscopes. Here's how to use them and how not to misuse them.

What we're measuring here is the accuracy with which a randomly drawn sample (everything in the population has an equal chance of being chosen) is likely to represent the whole population -- say, 500 registered voters (sample) against all registered voters (population). The conventional, and entirely arbitrary, confidence level we use is 95%, meaning that 19 times out of 20, a number falling within our confidence interval (the other name for "margin of sampling error") will accurately reflect the population.

Say we interview 500 registered voters in the hotly contested Crook vs. Liar Senate race. Our handy table* tells us that the margin of sampling error for a sample of 500 is 4.4 points (not, despite what the youth of South Carolina** are told, "percent"). If 47 percent say they plan to vote for Crook, we can say we're 95 percent sure his (or her) support among registered voters is somewhere between 42.6 percent and 51.4 percent.

Put another way, a poll that finds Crook at 47 and Liar at 53 could reflect a population that favors Liar. It could also accurately reflect a population that favors Crook 51-49. And 5 percent of the time, it could be reflecting something like a 60-40 Liar lead.

Does the 53-47 result show a solid lead? Sure, if you don't mind, oh, a one-in-three chance of being wrong. Just set your confidence level at about 67 percent and the margin of sampling error falls to 2.2 points.

Does the 53-47 result show that Liar has gained ground since last month's (Liar 51, Crook 49)poll? You do the arithmetic.

Some takeaway points:
1) The size of the population has nothing to do with the confidence interval. A random sample of 500 can represent a city; another can represent the state with the same accuracy.
2) Changes in sample size, though, are hugely important. Sampling error for subgroups is always larger for the group as a whole, meaning a gap that's significant in a sample of registered voters might not be significant for male or female registered voters.
3) "Sampling error" is only one kind of error. Question wording, question order and other factors can bias a sample; a classic example is the infamous double-negative Roper question that purportedly found a fifth of Americans ready to believe that the Holocaust hadn't happened. And don't even get started on social desirability bias.
4) The numbers say what they say. If the gap between Crook and Liar isn't significant, it doesn't become so just because the Washington Post or the New York Times declares it so. Period, end graf.
5) A story that doesn't provide the tools by which the poll's legitimacy can be judged is incomplete. It's as bad as leaving the attribution out of a cop story.
5a) It ain't that complicated. "The poll of 600 registered voters was conducted Oct. 18-20 by the Gallup Organization for the Daily Blatt. It has a margin of sampling error of 4 percentage points at a 95 percent confidence level" will do. Don't tell me readers won't understand it; you run earned-run averages every day.
6) "Polls" and "experts" are not the same thing. The hed above is misleading because of the attribution; the supporting evidence comes from political observers, not numbers.

Surveys are valid and useful tools as long as you keep them within their boundaries. But if you don't use the rules, the rules will have an annoying way of using you. Meaning you might end up looking dumb. Or biased. Or both.


* Many stats or methods texts will have such a table. If you don't have such handy, here's a formula for the 95% confidence level: 1.96√(.25/n), where n=sample size. Note that the population size doesn't enter into it.
** Oh, the humanity.