There Will be Polls and Rumors of Polls...

Now is the time when candidates and media start touting polls. Oddly enough the polls usually reflect good news for either the candidate or the editorial board stance of whoever sponsored the poll. Common sense tells one that when five or six different polls come out, each claiming to have a 3-5 point margin of error, yet with wildly diverging results, someone is lying to someone - perhaps themselves.

At best a poll is nothing more than a snapshot in time taken through a badly distorted lens. At worst it is an utter fraud. A few quick rules for reading polls:

1) Methodology is critical. If, during a primary, a pollster conducted a survey on Republican candidates and sampled only likely Democratic voters, the results would be interesting, but useless. Watch to see who is sampled. A very bad way to sample is to go by voter "households." Whenever this methodology is used it always understates the conservative vote - not because of bias, but because of a methodology flaw. I haven't looked at the numbers in a few years, but the last time I did, conservative households had a little more than .5 more voters on average than moderate households. Even if everything else is flawless, the poll ends up understating ative the voting power of conservative households.

Be careful also of polls that sample registered voters without a filter for likely voters. These polls tend to favor those with higher name recognition without measuring organization. The best polls are those that measure likely voters for a particular election. Even so, if some major sub-theme is running beneath the surface, these polls can be wildly inaccurate. For example, in 1992 Carol Moseley-Braun won the Democratic Primary for Senate and polls never showed it coming. The reason is there was a huge influx of voters in Cook County who did not normally vote. A poll that counts unlikely voters is always inaccurate in normal times; but a poll which counts only likely voters when there is a powerful sub-theme running is also wildly inaccurate.

2) How are the questions asked. It is very easy to design a poll to give you the answers you want to hear. That, in fact, is the specialty of a whole host of pollsters. Some of the ways of doing it are subtle; some are blunt. Most people are familiar with the nototrious push polls (i.e...'If you found that candidate A had committed various felonies, while candidate B is notable for philanthropic and charitable work, who would you be most likely to vote for?') But there are a lot of ways to guide respondents to the answer you want without them even realizing they have been guided. When a candidate or media outlet releases the horse-race question without releasing the exact number and phrasing of the questions used, be very suspicious.

3) What is the history of the pollster. This is a seeming elementary question that few bother to ever ask. Every pollster has a history - and checking that history would give one a clue how reliable their poll is. Among media polls in Illinois the Chicago Tribune Poll is consistently the worst. It always regurgitates whatever conventional wisdom is and never picks up on a brewing upset. It is so wildly wrong so regularly most professional politicians utterly ignore it as a source of useful information (though we're glad to tout it when conventional wisdom says our candidate is leading). The two most reliable Illinois media polls over the past decade have been the Copley Poll and the Mason-Dixon Poll. Other outlets have their ups and downs.

Among national media polls, Zogby was doing great stuff in the late '90s and at the turn of the millenium. They must have dramatically changed their methodology, though, because for the last four or five years they have been as useless as a Tribune Poll. One outlet I'll be watching closely is Rasmussen. They got all 50 states right in the 2004 presidential election.

4) People don't level with pollsters. In this case, you need to watch for routine behaviors that don't match up with reality. For example, black candidates in non-urban districts almost always poll 3-8 points higher than they actually perform on election day. There is a similar disparity for women candidates. From the mid-'70s to the mid-'80s women candidates actually outperformed men candidates just based on gender on election day. Since then, with the exception of left-leaning districts, women candidates poll better in most districts than they perform. A certain number of people like to be politically correct when talking to pollsters. A candidate needs to correct for that. A good way to check suspect information is to occasionally sit alone in a popular and busy coffee shop at breakfast. Listen to the conversations around you. People level with their friends over coffee. Hear what people say to each other candidly and take it into account.

5) Take care in interpretation. On this one I will give one final example. Every so often we hear charges of racism directed at lending institutions based on turndown rates. Without more information, turndown rates are meaningless in determining whether institutional racism is in play. If you want one meaningful statistic that would answer that question, look at default rates on existing loans. If any group, with a large enough sample size, has a statistically significant higher default rate, it strongly indicates that group is being held to looser standards in the application process. Often, a statistic, even if accurate, does not mean what a candidate or media outlet wants it to mean.

A pollster who tells you what you want to hear can be very tempting, but it is almost always a mistake. For a candidate, it prevents you from doing what you need to do to correct genuine problem areas - because you don't know those problem areas exist. For a media outlet, ultimately you will pay a price in credibility. In a campaign I don't want happy information or unhappy information; I want accurate information so I can act appropriately and develop a strategy that meshes with the facts on the ground rather than some dreamy blue-skies assessment.