| Adam Geller, who polled for Chris Christie, has written a great analysis of the 2009 New Jersey polls for pollster.com. I really think you should read it if you had any interest in following the campaign here. Let me just highlight the parts I found most interesting.
Before the election, we discussed how the robotic (IRV) polls consistently showed a Christie lead while the human interview polls often showed a tied race or a Corzine lead. The election results obviously make the IRV polls look great. That's worth remembering on its own since we often hear that they're a new methodology that is not reliable. Well, that piece of folk wisdom is wrong. So what does Geller have to say about it? He acknowledges the issue but says it is a matter of other choices made the pollster rather than the humans. I wonder. I get the impression he is going to great lengths avoiding the conclusion that the IRV polls are better.
He argues that public polls should report their partisan spread and that they oversampled non-voters. He discusses the huge problem of the undeclared voters. He cautions that some pollsters (Carville, cough, cough, Shaftan, cough) may have an agenda. I think all of that is straightforward and we at least touched on these issues during the campaign.
What was new to me was his criticism of "random digit dialing" (RDD), where the pollster dials random digits, to make up random phone numbers, and then reach random potential voters. Here's what he says:
In general, RDD methodology is a bad choice in New Jersey, if the goal is predictive accuracy.
In New Jersey, there are many undeclared voters (commonly but mistakenly referred to as Independents). These undeclared voters identify themselves as Republicans or Democrats - even though they are not registered that way. In our polls, we frequently showed a Democrat registration advantage that matched their actual registration advantage - but when it came to partisan ID, the spread was more like a six point Democrat advantage. By using a voter list, we knew how a respondent was registered - and by seeing how they ID'ed themselves, we gained insight into the relative behavioral trends of undeclared voters and even registered Democrats who were self identifying as Independents. Public pollsters who dialed RDD missed this. Partisan identification in New Jersey is not enough, if the goal is to "get it right."
That makes a lot of sense to me. It must be more expensive and difficult to work from voter lists, but basing your turnout models on actual voters who really voted might work better. It also helps with this huge problem of people who think they are (say) Republicans but don't register that way. We also know that the turnout model was critical just from what the public pollsters told us: that Christie had a huge lead (bigger than the win, by the way) with "definite" voters but the less likely voters brought Corzine closer.
Of course, at the end of the day, we don't know that Geller's polls were any better, when we recall we've never seen his numbers and I read between the lines. He invokes "insight we gained" which is rather different than 'results we measured' and when you read the article critically you'll notice there is no actual claim he did better. Still, I'd like to see what the public pollsters say about working off of voter lists instead of random digits. There doesn't seem to be much doubt that polling in New Jersey is more difficult than in some other states. |
