- The American Conservative - https://www.theamericanconservative.com -

Polling Averages

This [1] is a good example of why creating averages of polls can be a problem.  First of all, the averages for the presidential race, whether at Pollster or RCP, incorporate polls (such as the methodologically suspect June Newsweek survey and the LAT/Bloomberg poll) that are entirely unlike all others that skew the average in a certain direction.  Second, the “average” acquires a strange, almost canonical authority because it includes a number of polling results rather than being subject to the flaws of any particular poll, and people then invoke the average as if it were more authoritative rather than being correctly understood as less precise.  For the purposes of measuring overall momentum, averages can be useful, but when you include everything from typically reliable Rasmussen and Gallup numbers to the volatile Zogby results to the ridiculous PPP you are going to have an average that splits the difference between meaningful and useless.  This is not really to defend Dick Morris, but when he says [2] that Obama’s lead had virtually disappeared in the Rasmussen tracking poll [3], he was stating something that is demonstrably true.  In the days since he wrote that, a small Obama lead has reappeared, but Obama’s position remains virtually identical to what it was six weeks ago at the end of the Democratic primaries, and for three consecutive days last week the candidates were separated by no more than one point.  Dick Morris is wrong about enough things that we don’t need to impute more errors to him than he already makes on his own.

Update: Sullivan [4] draws attention to Mr. Koffler’s comment below.  I should have made clear in the post that I don’t attach significance to the movement of one tracking poll by a couple points that lasts for two or three days.  I certainly didn’t agree with Morris’ larger contention that this movement was a consequence of various Obama reversals.  Mr. Koffler explains where I’ve gone wrong here [5], and I appreciate the useful correction to my mistake.  Finally, I stated things in this post quite clumsily, particularly in the line about difference-splitting.

15 Comments (Open | Close)

15 Comments To "Polling Averages"

#1 Comment By kitstolz On July 16, 2008 @ 11:03 pm

Arguing the latest numbers on Obama is enjoyable, I think.

A contrasting view would point out that Obama’s lead is outside the margin of error in Iowa, Michigan, Minnesota, Wisconsin, and of course Illinois. People who should know Obama, like Obama.

Meanwhile “red states” such as Missouri, Florida, and Virginia are neck-and-neck. Obama is tied with McCain in Virginia, and is making a huge push for the state, with the total support of the popular Democratic leadership.

And to toss aside the LA Times poll and the Newsweek poll as somehow beneath inclusion is arbitrary. The latest Newsweek poll is much closer to the average of about 3.5 percent point lead for Obama, but who is to say that this snapshot of the national electorate is right and that one was not?

Not to mention pollster George Barna, whose very experienced firm frequently polls for religious organizations, and gave Obama a 15% lead shortly before July 4th.

(Hope that link is live!) Here’s another one, from my interview with Barna:

#2 Comment By Daniel Koffler On July 16, 2008 @ 11:18 pm

Polling averages tend to be less precise but more accurate than individual polls (there’s frequently a tradeoff between accuracy and precision). That’s also demonstrably true. The Newsweek and LAT/Bloomberg polls are methodologically suspected by the McCain campaign, but the grounds for that suspicion is suspect. (We’re really shooting in the dark about party-ID breakdowns and effects and turnout models, even more so than usual this year.)

It’s at best misleading to make anything of leads in polls “disappearing” and “reappearing” on a day-to-day basis. Sure, it’s possible that those shifts are significant of daily shifts in voter preferences, but overwhelmingly more likely (and militated for by Occam’s razor) that nothing at all changed in that period, and the poll had simply captured statistical noise. That’s why polling averages are useful. They help discriminate between real shifts in opinion and noise. Writing an article about fluctuations in a single daily tracking poll is a good sign of an innumerate hack, and cherrypicking a single poll that conforms to one’s biases before any verifiable trend can be adduced is breathtakingly mendacious — i.e., exactly what you’d expect from Dick Morris. True, we shouldn’t impute more errors to him than he makes on his own, but an epistemology that extends Morris the benefit of the doubt we are accustomed to extend to reasonable and honest people is bound to lead to error.

You’re right to object to treating the RCP average as canonical, though, since despite the fact that it does correct somewhat for inaccuracy in individual polls, it’s a crude, error-ridden first stab at doing so. Fortunately, there’s a much more sophisticated 2.0 version of what RCP is trying to do, namely FiveThirtyEight’s popular vote and electoral vote projectiions, based on not just on taking the arithmetic mean of incommensurate data, but on a fairly rigorous assignment of weights to various polls, demographic analysis of states on a district by district basis, nearest-neighbor analysis of states and districts (since information about a state or district gives us information about states and districts that closely resemble it), etc. As with any polling analysis, FiveThirtyEight’s method is imperfect. But it’s orders of magnitude more deserving of canonical authority than anything else in the general sphere of political opinion polling.

What does it show? It shows Obama winning the national popular vote by something in the neighborhood of 4 points and clearing 300 electoral votes since the general election campaign began. His numbers dipped slightly, reflecting the tightening Gallup and Rasmussen showed last week, and should tick back up with the latest tranche of polls showing high single digit leads. The point being that Obama’s lead has been consistent and stable since the general election began.

#3 Comment By Grumpy Old Man On July 16, 2008 @ 11:26 pm

Obama’s lead did narrow, and in the polls I tend to trust more (Rasmussen and Gallup), and then it widened a bit.

On the popular vote polls the gap has remained quite narrow, with BHO in the lead, and BHO leads a bit in most electoral vote projections.

Unless McCain comes up with something remarkable, the election is Obama’s to lose. As long as Jennifer Rubin and Abe Greenwald keep trashing him, I will confess to a strange new respect for the man.

And no, a toe fetish does NOT disqualify one from the punditocracy.

#4 Comment By conradg On July 17, 2008 @ 6:44 am

Mostly, what DK said. I’d definitely second the idea that you rigorously study [8] for a better understanding of the statistical realities behind these polls, and how to analyze them. Most journalists and Byzantine scholars are understandably not very good at math, and could use a little guidance. Unfortunately, most are uninterested in learning, so big misunderstandings tend to get spread across the board and aren’t easily cleared up.

BTW, it’s still far from clear that the big leads Newsweek put up early were false, or that the current narrowing is true. Voter models are in great flux right now, and results always vary, which is why there is such a thing as “margin of error”. Also, tracking polls have their own biases which are hard to eliminate.

#5 Comment By John Schwenkler On July 17, 2008 @ 9:17 am

Sheesh, conradg – do you think you could be a little more snide there?

#6 Comment By Daniel Larison On July 17, 2008 @ 10:04 am

It’s true that I’m not especially good at math, but I do have some understanding of statistics. As for what “most Byzantine scholars” can or cannot do, I’m fairly confident that you have no idea. If you could actually bother to point out what I have gotten seriously wrong, that might aid in the learning process in which I allegedly have no interest.

The Newsweek and LAT polls were not “false”–they really did find respondents who said the things they reported. However, they were probably unrepresentative of the state of the presidential race at that time. Once they had a sample that was much more like others in terms of party ID, their results were much more in line with what everyone else has been finding.

There’s a margin of error? Really? Thanks for the tip!

#7 Comment By Daniel Larison On July 17, 2008 @ 10:28 am

Thanks for the useful correction, Mr. Koffler. Clearly, I hadn’t thought about this nearly enough. I will keep your post foremost in my mind should I have something else to say about polling.

#8 Comment By conradg On July 17, 2008 @ 2:54 pm


First, you would benefit by taking yourself a little less seriously, If you want to be known as someone who dishes out harsh and unrelenting criticism, you have to welcome the same in return. If you can’t take the heat, get out of the blogosphere.

Second, DK does a great job of pointing out the problems with your analysis, which is just at the junior high school level, to be honest, and there’s so much wrong with it the best I can recommend is that you read assiduously through sources like 538 and study the general literature on the subject. I’m sure there are some Byzantine scholars with good statistical math skills, but clearly it’s not a job requirement, and it’s also clear you are deeply lacking in those skills by both your own admission and demonstration. If you want to get on a high horse, don’t use a pony.

#9 Comment By Daniel Koffler On July 17, 2008 @ 3:47 pm

Ah, don’t mention it Daniel. I wouldn’t have spent this much time on this if I weren’t a big admirer of your stuff in general and cheering as loud as anybody can from the sidelines for your side to win the conservative civil war. You’ll have a huge leg up if you guys get good at math. Have you read Commentary much recently? I’ll lay odds for anybody willing to bet that Jennifer Rubin can process calculations of numbers an order of magnitude larger than quantities she can use parts of her body to count.

#10 Comment By Daniel Koffler On July 17, 2008 @ 4:01 pm

Oh one other thing, I rarely give myself any practice writing a bunch of jokes into my posts, which means the risk of telling a dud or being misunderstood is amplified. Just to be clear, the target of my snark is primarily Morris and the kind of journalism typified by CNN and the AP

#11 Comment By Daniel Larison On July 17, 2008 @ 4:03 pm

I was not in the best mood when I was reading comments earlier today, and so I responded intemperately. Generally, I try not to take myself seriously, but I also have a short temper, which was even worse earlier, so I apologise for my curt remarks. I took a couple statistics classes in college, but have rarely made use of them since then. In fact, I do understand how I went wrong here and would have avoided the blunders had I not been posting in haste.

Thanks, Mr. Koffler. I’ll keep that in mind.

#12 Comment By Daniel Larison On July 17, 2008 @ 4:13 pm

I understood what you were trying to do, and I agree with your critique of this sort of journalism. A fairly good example of this sort of thing was when Newsweek changed the composition of their sample, came up with dramatically different results and then tried to explain–in a confused voice–how the race could have changed so quickly.

#13 Comment By c-joe On July 17, 2008 @ 4:16 pm

The Gallop and Rass. polls are weighted by party ID, so their extreme stability in always showing the race somewhere between tied and Obama by 7 is only reflected changes within parties, not shifts to one party or another.

An interesting point I learned at 538 is that Obama has a serious chance of winning the electoral vote but not the popular vote because of population shifts to Red States from 2000-2008 are not yet reflected in the electoral college. In other words, if redistricting were dynamic, Bush’s 2004 states would have gained several electoral votes.

Also, a perfect tie would go to the House and result in Obama winning.

So even if national polls are perfectly accurate, McCain would need to be at about +1.5 to actually win.

#14 Comment By c-joe On July 17, 2008 @ 4:22 pm

I also have a pretty strong gut feeling that black turnout this election is not being accounted for.

Typically blacks vote at the white rate in the deep south, but well below the white rate in several swing states such as Ohio. I think it will be higher than the white rate virtually everywhere in 2008.

How many of these black voters don’t get polled at all because they only have cell phones, or fail to pass the “likely-voter” screen because they have never voted before? A lot will not even pass the registered voter screen because they aren’t registered yet but will be by the election, or don’t remember (“not sure” answerers to “are you registered at your current address” get booted.)

#15 Comment By conradg On July 18, 2008 @ 8:42 am


No big deal about your anger. It’s endearing, really.