It’s all in how you ask the question

Posted by padams under Election 2008, Election 2008 Campaign strategy, Election 2008 Media commentary

Paul Adams

People often ask why polls released on the same day produce somewhat different results. Of course, some of this may just be the result of statistical error. That is, the differences between two polls may be within the margins of error of the polls. On other occasions, if you look closely, you can see that one poll was actually conducted on different dates, or over a different period than another, even if they are released the same day. This has been true, for example of the EKOS (conflict alert), Nanos and Harris-Decima daily tracking polls.

At EKOS, we think we have been a little ahead of other pollsters because the large sample sizes we get using a new methodology called interactive voice recognition (IVR) — a kind of robotic pollster. Of course, don’t expect other pollsters to agree with this, and quite rightly some people are asking how much we should trust this new technology. But that’s another story.

Sometimes, though, the effect of methodological differences in gathering poll data are perfectly clear. This has showed up in Nik Nanos’ daily tracking polls, which have consistently shown the Liberals running more strongly than other polls have, and have tracked the Green Party at significantly lower levels. 

This is important, of course. If Nanos is right, the Liberals are much more in contention in this election than the media and punditariat have given them credit for. And the attention to the Greens is disproportionate to their actual strength. Clearly, the media have taken their cue in framing this election from the consensus in the other polls that the Liberals are well behind the Conservatives, and dismissed the Nanos polls, in this respect, as an anomaly.

These media orientations may be crucial to the prospects of the parties, so it is important to understand why Nanos is so different. After all if he is right, and the rest of us pollsters are wrong, the media are missing the story.  

The difference in the Nanos numbers and those of other polling companies was explained quite well yesterday in an article by Glen McGregor appearing here in Ottawa in the Ottawa Citizen.

Unlike most other pollsters, Nanos is using an “open” question when asking respondents about their voting intention. What that means is that his phone operators ask people how they plan to vote, but don’t present them with a list of the parties from which to choose.

“If they don’t get the list, you get the cleanest read because they have to articulate their support,” Nanos told MacGregor.

This is an old debate in polling circles, and has merit on both sides.

I first encountered it in the early 1990s when I was working with Elly Alboim and Christopher Waddell among others on what was then the CBC-Globe and Mail poll. (I mention them because we are now all on staff at the Carleton J-School).

We published a national poll the same day as La Presse published one of their own focusing on Quebec. The La Presse poll showed the relatively newly-formed Bloc Quebecois running much more strongly than ours did, which led to accusations that the CBC, as a federal institution, was trying to deprecate the separatist BQ.

Of course, there was nothing to that accusation. We wondered however, whether the difference between our poll and La Presse’s could be explained by the fact that at the time we were using an open-ended question, without prompting respondents about the names of the parties. Perhaps without a menu to choose from people were “forgetting” that the BQ was an option; and the same might hold for the newish Reform Party in English Canada.

So in the next poll we did, we split the sample — asking half the respondents an open question, and half a closed question. Bingo! The closed question, in which all the parties were named, registered much higher support for both the BQ and Reform than the open question. Since these parties were going to be on the ballot, we decided to go with the closed question in future.

Similarly, at EKOS a few years ago, we decided to replace our old open question with a closed question — this time because of the emergence of the Greens. Sure enough, Green support popped up when we did that. And our results using the closed question actually tracked quite well to the final result.

Of course, none of this is to deny that there is an argument on the other side. After all, if you can’t even remember that a party is in the race without an operator reading its name out to you, how deep can your support for it be? 

What may be happening in this election is that many faint-hearted Liberals tell Nanos they are voting that way, but when presented with the Green option in other polls, they flip the other way. On election day, of course, they need to make a decision. And it may also be that as election day approaches, and people’s decisions become more firmly grounded, the difference in the methodology will start to wash out.

So what to make of all this, journalistically? I think it is wrong to reject the polls as meaningless, as some claim they do. I don’t think they actually do anyway. The polls agree on the ordering of the parties, at least at the top of the race: Tories first, Liberals second, NDP third. Moreover, all of them suggest that the Conservatives are in sight of a majority, but don’t have it in the bag.

Those are important issues for the media, and for voters who may choose to vote tactically.

But what journalists should do is look carefully at the differences among polls and why they exist, and not rely heavily on any single one. A couple of weekends ago, the Globe put great weight on its front page on a poll showing the Tories had had a strong first week. By Monday, the same pollster was saying there had been a softening in Tory support.

Polls provide interesting information bound by statistical and methodological limits. We don’t know which one will be closest till after the election is held. A record of success (or of past failure) by a pollster is worth bearing in mind, but even past success doesn’t guarantee the best results next time. 

Paul Adams is a former political reporter with the CBC and the Globe and Mail, and is now a member of Carleton’s journalism faculty, and executive director of EKOS Research Associates.