Philosophical vs Substantive Polling Questions

There’s often a disconnect between what public opinion survey results suggest and the public’s true position on an issue. To understand why survey results sometime imply one outcome and reality reveals another, the consumers of public opinion data need to learn the difference and identify philosophical position and substantive survey questions.

Philosophical survey questions, as philosophical questions in general, are not designed to arrive at a definitive answer about a specific issue. While most people could identify fundamental philosophical questions (What is the meaning of life? What is good/evil? What is beauty?), survey questions designed to evaluate philosophical positions are less easily distinguished.

Philosophical survey questions often ask respondents to evaluate the priority they place on various concepts or ideas, or to indicate whether they support or oppose a plan or idea in general terms. An example of philosophical position survey question may include a priority question about whether respondents feel adding air conditioning to existing schools without air condition is a high, medium, low, or not a priority. Another example could be when respondents are asked if they agree or disagree with the statement that the government should guarantee health insurance for all Americans. These questions do not discuss a specific plan, but rather probe how respondents generally feel about these topics.

Understanding the philosophical position of the voters is critical for developing a campaign’s strategy. If it is known, following our example, that voters are generally or highly supportive of adding air conditioning to schools where it currently does not exist, then the campaign for adding air conditioning has a starting point from where to understand their position. Supporters of adding air conditioning know that they are operating in an ‘environment’ that is ‘friendly’ to their cause. Supporters can focus their campaign on issues that will retain support, instead of trying to shift opinions in favor of adding air conditioning.

Philosophical opinions are often the poll results to which campaigns try to direct the public’s attention. They are the results about which campaigns generally want their issue debated, they focus on the big idea. Substantive questions, however, often reveal a schism in respondent opinion, and are less likely to be reported or the focus of a media release. Substantive questions, following our example, would ask survey responds if they supported or opposed adding air conditioning to schools in their district where it currently does not exist if they knew the proposal would require a bond costing thirteen million dollars. A substantive question injects the facts missing from a philosophical question and allows voters to make a more informed decision about their position on the issue, concept, or idea.

There are varying degrees of substantive questions. Respondents could be asking their opinions about an issue with only being presented the basic facts (Adding X number of air conditioners at Y cost). This type of question is often called an ‘initial ask’ or an ‘uninformed ask.’ It is generally asked in neutral language and, if the issue is about something likely to appear on a ballot, worded as closely as to the questions language on the ballot as possible. This sort of question gives a clear indication if the philosophical opinions of the respondents will differ with the reality.

Substantive questions may also be included to try to simulate the dynamics of an engaged debate about an issue (along with additional message testing techniques). Questions like these are typically called ‘informed asks.’ In an informed ask, balanced messages in support and opposition of an issue are cited after a statement of facts about the issue (Adding X number of air conditioners at Y cost, supporters say… opponents say…). Respondents are then asked, based on the information presented in the question, if they support or oppose the issue.

Through examining the differences between philosophical opinions and what respondents indicate through substantive questions is where strategy is developed. Campaigns need to know where the voters stand philosophically on their issue, idea, or concept and where support (or opposition) is lost (or gained) once additional information and messages are presented about the topic. Polling offers campaigns the ability to target demographic and attitudinal groups, discover where and among which respondents changes in opinions occur, and what needs to be done to keep or convert supporters to their cause.

To genuinely understand public opinion on an issue, more than the philosophical data needs to be presented. We have observed the philosophical fallacy in the current deluge of polling results presented in the media about healthcare reform. While the polls show wide appeal for change and reforming the system, specific plans, substantive questions, show a less consensus about how to fix the system. Consumers of healthcare survey data should be careful to seek out the results for substantive questions and not be clouded by reports based on the philosophical support of a plan.

Read more...

Informed Polling and Getting it Right

Cross Posted from Fako & Associates' Political Polling Blog.

Every pollster will tell you that political polls are a snapshot in time and at best consecutive polls can elicit trends, but should not be used to predict turnout or the outcome of an election.

While the national attention is on how the pollsters got it wrong in several contests with surveys taken only days before an election, few are talking about how early surveying often-times gets things right. We've occasionally written about the importance of informed trial heats in some of our past posts.

To refresh, an informed trial heat is designed to simulate the effects of an engaged campaign, presenting balanced positive and negative messages about each candidate (or multiple candidates). The end result of the election scenario tells a campaign what is possible with their messages and themes they plan to implement. Most importantly, this section will determine if the core message works in direct contrast to the opponent(s)' message and can identify movement among the various demographic and attitudinal groups -- helping refine strategy.

F&A recently conducted a benchmark survey for a client running in a Democratic primary for open seat in a multi-candidate open-seat race. The benchmark poll was designed to evaluate the political environment, determine voter’s top issue concerns, examine opinions of the candidates and other significant figures, and test messages in support and in opposition of various candidates. As we always do in comprehensive benchmark polls, we included an informed trial heat question in the survey. The client was on a tight budget and didn't want to include "minor" candidates in the informed trial heat. After some debate, we were able to convince the client to include the "minor" candidates in the question.

We accounted for the ability of the "minor" candidates to get their message out given their budget constraints. Their messages were curtailed in the informed scenario to a simple bio-statement, while "major" candidates received bio, supporting, and opposing information, simulating an engaged campaign.

In our poll, the results of the informed trial heat were unexpected; a "minor" candidate took a 22 percentage point lead above the assumed frontrunner in the informed scenario, an increase of 25% above the candidate's level of support in the initial trial heat (the uninformed horse race question). A "major" candidate jumped up 7% and the assumed frontrunner stalled with a gain within the margin of error. Undecided voters in the initial trial heat heavily sided with one of the "minor" candidates. The percentage of undecided voters was reduced by over 40% in the informed trial heat. At this point we recognized the minor candidate's growth potential and advised the client to pay close attention to this so-called "minor" candidate. We noted that this individual clearly had the basic background and simple message that would break through the clutter of a highly engaged multi-candidate race, despite an initial perception of not being viable.

As the campaign progressed, the so called minor candidate ended up raising some serious money and gained significant earned media attention in addition to their own paid activities. It became apparent that this minor candidate was not minor, something our polling has observed only three months before the election

The "minor" candidate ended up winning this election, slightly ahead of the "major" candidate that we also observed gaining traction through the informed trial heat. This highlighted the usefulness and importance of utilizing informed trial heat questions in polls and why clients should never ignore perceived "minor" opponents. Polls that include an informed trial heat are one of the most useful strategic planning tools available to a campaign. It gives campaigns the information needed to determine if their message works (in the above example, out client’s message was not working); provides detailed strategic planning information, particularly at the demographic sub-group level, and gives campaigns information to prepare and adjust strategy for unanticipated situation (such as an unexpectedly strong opponent).

Read more...

Open Ended & Listed Issue Concerns

Cross Posted from Fako & Associates' Political Polling Blog.

Our polling and several national polls by other organizations have shown that the Iraq War is no longer the highest issue concern among likely voters. While still a major issue that could quickly rise to the top again given sufficient media attention, the War is being trumped by financial concerns about the stalling economy, foreclosures, raising taxes, and health care and prescription medicine.

"The economy, stupid"

Different Bush, different Clinton, same message... new Obama?

Is it really 1992 all over again?

"It did take a Clinton to clean (up) after the first Bush, and I think it might take a second one to clean up after the second Bush..." -- Hillary Clinton, January 31, 2008

Some seem to think so.

Carville's famed sign on the wall of the Little Rock office in 1992 also included two other important phrases:

1. "Change vs. more of the same"

2. "Don't forget health care."

While the relevance of 1992 is up for debate, we're finding that health care is becoming ever-more defined as an economic concern, and undoubtedly, an underlying component of the current feeling of economic uncertainty. According to a 2005 Harvard University study*, 68 percent of those who filed for bankruptcy in the US had health insurance. In addition, the study found that 50 percent of all bankruptcy filings were partly the result of medical expenses. Indeed, don't forget health care.

Simply talking off a bullet point isn't enough to relate to the voters. Candidates have to speak directly to the concerns of the electorate. While all challenger campaigns are inherently running on a message of "change," even if that message is never directly communicated, the nuances of the message behind their version of "change" will determine if they connect with the voters. As we've seen in the recent debates, none of the candidates were shying away from the word "Change..."

Undoubtedly the word "change" tested well.

Pollsters sometimes use listed issue concern questions instead of open-ended issue concern questions to save time (and money) in a survey. We find that listed issue questions often miss the way how issues are being discussed. While F&A, Inc. hasn't recently conducted a nation-wide survey that included a top issue concern question, some of our recent surveys in several mid-west and east coast state legislative campaigns included open-ended verbatim response questions about the voters' top issue concerns.

For example, in one of our recent surveys, a respondent offered the following when asked about his or her most important issue concern:

"I WOULD SAY HEALTH CARE SHOULD BE MORE AFFORDABLE TO PEOPLE AND THAT DRUG PRESCRIPTIONS PRICES SHOULD BE LOWERED. I THINK THAT EVERYBODY SHOULD HAVE AFFORDABLE INSURANCE"

This response was typical of all responses that related to health care. A campaign in this district that focuses it's health care message on concerns other than making health care affordable and lowering the cost of prescription drugs will not be connecting with the concerns of the voters in this particular district.

1992 or not, candidates who speak directly to the concerns of the voters in a way that addresses their concerns will fair far better than a candidate who elaborates off a bullet point without qualified direction.

Early benchmark surveys should be as comprehensive as possible and include open-ended issue concerns whenever possible and appropriate for a campaign's budget. These types of questions help drill deeper into the how and why a voter thinks and cares about a particular issue and provides better strategic direction on how a candidate can address the issue.

In a presidential race, 1992 or 2008, there simply is no excuse for not having the message right. Regardless of the level of campaign, we always say it is better to have the message designed right the first time, than to spend the rest of the campaign correcting it.

* Himmelstein, D, E. Warren, D. Thorne, and S. Woolhander, "Illness and Injury as Contributors to Bankruptcy, " Health Affairs Web Exclusive W5-63, 02 February , 2005.

Read more...