Monday, January 30, 2006

There Will be Polls and Rumors of Polls...

Now is the time when candidates and media start touting polls. Oddly enough the polls usually reflect good news for either the candidate or the editorial board stance of whoever sponsored the poll. Common sense tells one that when five or six different polls come out, each claiming to have a 3-5 point margin of error, yet with wildly diverging results, someone is lying to someone - perhaps themselves.

At best a poll is nothing more than a snapshot in time taken through a badly distorted lens. At worst it is an utter fraud. A few quick rules for reading polls:

1) Methodology is critical. If, during a primary, a pollster conducted a survey on Republican candidates and sampled only likely Democratic voters, the results would be interesting, but useless. Watch to see who is sampled. A very bad way to sample is to go by voter "households." Whenever this methodology is used it always understates the conservative vote - not because of bias, but because of a methodology flaw. I haven't looked at the numbers in a few years, but the last time I did, conservative households had a little more than .5 more voters on average than moderate households. Even if everything else is flawless, the poll ends up understating ative the voting power of conservative households.

Be careful also of polls that sample registered voters without a filter for likely voters. These polls tend to favor those with higher name recognition without measuring organization. The best polls are those that measure likely voters for a particular election. Even so, if some major sub-theme is running beneath the surface, these polls can be wildly inaccurate. For example, in 1992 Carol Moseley-Braun won the Democratic Primary for Senate and polls never showed it coming. The reason is there was a huge influx of voters in Cook County who did not normally vote. A poll that counts unlikely voters is always inaccurate in normal times; but a poll which counts only likely voters when there is a powerful sub-theme running is also wildly inaccurate.

2) How are the questions asked. It is very easy to design a poll to give you the answers you want to hear. That, in fact, is the specialty of a whole host of pollsters. Some of the ways of doing it are subtle; some are blunt. Most people are familiar with the nototrious push polls (i.e...'If you found that candidate A had committed various felonies, while candidate B is notable for philanthropic and charitable work, who would you be most likely to vote for?') But there are a lot of ways to guide respondents to the answer you want without them even realizing they have been guided. When a candidate or media outlet releases the horse-race question without releasing the exact number and phrasing of the questions used, be very suspicious.

3) What is the history of the pollster. This is a seeming elementary question that few bother to ever ask. Every pollster has a history - and checking that history would give one a clue how reliable their poll is. Among media polls in Illinois the Chicago Tribune Poll is consistently the worst. It always regurgitates whatever conventional wisdom is and never picks up on a brewing upset. It is so wildly wrong so regularly most professional politicians utterly ignore it as a source of useful information (though we're glad to tout it when conventional wisdom says our candidate is leading). The two most reliable Illinois media polls over the past decade have been the Copley Poll and the Mason-Dixon Poll. Other outlets have their ups and downs.

Among national media polls, Zogby was doing great stuff in the late '90s and at the turn of the millenium. They must have dramatically changed their methodology, though, because for the last four or five years they have been as useless as a Tribune Poll. One outlet I'll be watching closely is Rasmussen. They got all 50 states right in the 2004 presidential election.

4) People don't level with pollsters. In this case, you need to watch for routine behaviors that don't match up with reality. For example, black candidates in non-urban districts almost always poll 3-8 points higher than they actually perform on election day. There is a similar disparity for women candidates. From the mid-'70s to the mid-'80s women candidates actually outperformed men candidates just based on gender on election day. Since then, with the exception of left-leaning districts, women candidates poll better in most districts than they perform. A certain number of people like to be politically correct when talking to pollsters. A candidate needs to correct for that. A good way to check suspect information is to occasionally sit alone in a popular and busy coffee shop at breakfast. Listen to the conversations around you. People level with their friends over coffee. Hear what people say to each other candidly and take it into account.

5) Take care in interpretation. On this one I will give one final example. Every so often we hear charges of racism directed at lending institutions based on turndown rates. Without more information, turndown rates are meaningless in determining whether institutional racism is in play. If you want one meaningful statistic that would answer that question, look at default rates on existing loans. If any group, with a large enough sample size, has a statistically significant higher default rate, it strongly indicates that group is being held to looser standards in the application process. Often, a statistic, even if accurate, does not mean what a candidate or media outlet wants it to mean.

A pollster who tells you what you want to hear can be very tempting, but it is almost always a mistake. For a candidate, it prevents you from doing what you need to do to correct genuine problem areas - because you don't know those problem areas exist. For a media outlet, ultimately you will pay a price in credibility. In a campaign I don't want happy information or unhappy information; I want accurate information so I can act appropriately and develop a strategy that meshes with the facts on the ground rather than some dreamy blue-skies assessment.

14 comments:

Anonymous,  4:19 PM  

zzzzzzzzzzzzzzzzzz

Anonymous,  4:55 PM  

So?

Bill Baar 4:59 PM  

I found it helpful. It would be nice to see some standards set for polls pushed out to the public.

Something like the Blogesephere demands the margin of error evertime a result cited.

A list of questions and the universe sampled.

Charlie Johnston 6:55 PM  

To anon 4:19, you certainly have a point. When I started writing here I told Rich I would sometimes do raw opinion and, at others, write about how a working consultant looks at these things. Those columns that fall in the latter camp will often be very dry to many readers, but interesting to a lot of the political junkies who do not make a living from it. It will not offend me at all if, early on in such a post, you decide it is a bore and move on. But I will continue to make such posts. Bottom line...if you like fireworks but have no interest in how they are made, these types of columns are not for you.

Anonymous,  7:43 PM  

You seem like a good person, but your post if filled with very inaccurate data. In other words, it's baloney. Or in polling terms, you have a bad sample. I don't have the time to correct you, but please examine a back issue of the Chicago tribune a few weeks before the Dixon/Braun primary. They had her closing in and many people pegged her as the winner.

Anonymous,  7:46 PM  

Charlie--I just read you are a policial consultant--can you tell us for who? What is your background?

Daniel Darling 7:59 PM  

Good stuff, Charlie.

Charlie Johnston 8:28 PM  

Gosh, Political Hack, you must be a newbie. My background is pretty well known to readers on this site.

Your comments reminded me of a conversation I had in the spring of'96. I was speaking to the City Club of Chicago about how we had pulled off Al Salvi's upset victory over Bob Kustra. Afterwards, the Tribune's pollster (who was in the audience) came up to me and started earnestly explaining how he had showed Salvi closing in and smelled the upset coming all along. I listened politely but was thinking to myself. "Ye Gods, if you had a margin of error of more than 20 points, you should have said so."

In '92 I was working for a Lake County Radio Station. On election night I recall everyone being surprised that Moseley-Braun won. Now I did not seriously start paying attention to specific polls until '96, so perhaps you are right about '92. If you would forward the archived Tribune article to me at charliej373@yahoo.com and I am wrong I will correct it.

I am hesitant to say who I work for in fear that, if I am right that you are the fellow I chatted with at the City Club 10 years ago, the Tribune Poll is liable to show him losing. On the other hand, if you do that, it will give me more evidence on election day on how often the Trib Poll is wrong. I am a senior advisor to 8th Dist. Congressional candidate David McSweeney this cycle.

Charlie Johnston 10:09 PM  

An alert reader sent me a copy of a Tribune article from March 8, 1992(about a week and a half before election day)accompanying the poll on the Democratic U.S. Senate Primary. The poll showed as follows:

Alan Dixon - 38%
Carol Moseley-Braun - 28%
Al Hofeld - 17%
Undecided - 16%

This is an excellent poll - and the accompanying article, written by Thomas Hardy, astutely notes that all the upward movement was with Braun and Dixon was on a steep decline. He correctly predicted there was a distinct possibility of an upset victory by Braun. This article and this poll was right on.

Anonymous,  10:13 PM  

zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

Pat Collins 10:44 PM  

I find this fascinating - thanks. Amazing that the Pal had warning it was coming and STILL lost. I still think if Hofeld didn't run the Pal could have pulled it off.

Charlie Johnston 11:12 PM  

Actually, the Hardy article noted that the rhetoric was so bitter between Dixon and Hofeld that people had developed strong negative feelings towards both of them. Braun was the unknown in the pack. Her main asset was that she was neither Hofeld nor Dixon. Without Hofeld in the race clawing at Dixon constantly, you are almost undoubtedly right; not only does Dixon win, but he wins easily.

Anonymous,  11:18 PM  

It sounds as though Charlie Johnston is as relevant as the Chicago Tribune polls he cites. If you're a consultant to someone's campaign, how come you have all this time to post on blogs? With the election only 7 weeks away, is your client getting their money's worth?

Anonymous,  8:26 AM  

Anon 11:18-Last I checked people have the capacity to work and leave time for other activities. If you're not that type of person, I highly suggest you get a time management course and learn how to do two things at once.

  © Blogger template The Professional Template by Ourblogtemplates.com 2008

Back to TOP