How 538 Rates New Jersey Pollsters

by: Hopeful

Thu Jun 10, 2010 at 11:00:00 AM EDT

Nate Silver at FiveThirtyEight.com has put out a new set of pollster rankings (methodology here and reponse to complaints here) and I notice our New Jersey pollsters did off pretty well. His "Pollster Induced Error" (PIE) statistics is meant to capture "the amount of error that a pollster introduces above and beyond that which is unavoidable due to things like sampling variance. The lower a firm's PIE the better." Farleigh Dickinson (1.75), Rutgers (1.76), and Gannett/Monmouth (2.17) are all better than the default rating. Ratings are primarily based on how well polls within the last 21 days predict the outcome of the election over the last decade.

I should note, though, there is a statistical correction ("reversion to the mean") in which:

...the ratings for polling firms which were members of NCPP, or which had signed onto the AAPOR Transparency Initiative, as of June 1, 2010, are regressed toward a different mean than those which hadn't. Essentially, then, polling firms are rewarded for having made a public commitment to disclosure and transparency -- but the basis for rewarding them is statistical rather than ideological.

FDU and Rutgers both have met this criterion, but to my surprise
Gannett/Monmouth (according to 538) has not, and so I imagine it suffers slightly in this rating system. If this is an error by Silver, Patrick Murray might want to correct it. There certainly seems to me to quite good details in their releases.

Hopeful :: How 538 Rates New Jersey Pollsters

Tags: polls, pollster ratings, (All Tags)

Print Friendly View

Send As Email

How 538 Rates New Jersey Pollsters | 2 comments

Transparency (4.00 / 1)

Someone just alerted me to the list. Yes, there was an oversight (but not on Nate's part). Due to a miscommunication, we were left off the list (we're there now -- see AAPOR dot org). We are absolutely committed to transparency in the effort to demystify polling (and believe we share a lot more information about our methods than most pollsters do).

I've asked Nate how this would affect our rating and am waiting to hear back from him.
If you examine the raw scores on his list, you'll see we're the only New Jersey pollster (including Qpac) with a score below the typical pollster error. I think that extra credit for transparency may have a sizable impact on the rankings (albeit not so much on the adjusted scores, which actually cluster pretty closer together). I also think polling in volatile New Jersey probably gives us a higher "degree of difficulty" score too.

by: Patrick Murray @ Thu Jun 10, 2010 at 20:59:27 PM UTC