Dave Kleinschmidt has some commentary on the Nate Silver fangirl/boy-ing that many of us quantitative types have been engaging in for the last week.
My tribe—the data nerds—is feeling pretty smug right now, after Nate Silver’s smart poll aggregation totally nailed the election results. But we’re also a little puzzled by the cavalier way in which what Nate Silver does is described as just “math”, or “simple statistics”. There is a huge amount of judgement, and hence subjectivity, required in designing the kind of statistical models that 538 uses. I hesitate to bring this up because it’s one of the clubs idiots use to beat up on Nate Silver, but 538 does not weight all polls equally, and (correct me if I’m wrong) the weights are actually set
by handusing a complex series of formulae.
The point is that the kind of model-building that Nate Silver et al. do is not just “math”, but science. This is why I don’t really likethat XKCD comic that everyone has seen by now. Well I like the smug tone, because that is how I, a data scientist, feel about 538′s success. That is right on. But we’ve known that numbers work for a long time. Nate Silver and 538 is not just about numbers, about quantifying things. Pollsters have been doing that for a long time. It is about understanding the structured uncertainty in those numbers, the underlying statistical structure, the interesting relationships between the obvious data (polling numbers) and the less obvious data (economic activity, barometric pressure, etc.) and using that understanding to combine lots of little pieces of data into one, honkin’, solid piece of data.
When I teach stats, or talk about stats in my other classes, I try to hammer on this point about uncertainty. As scientists, we're dealing with noise in our data from all kinds of places. Is the sample under study "weird" in some way? Is our measure noisy? How noisy? How variable are people? Why? Does the time of day/day of week/week of year when people are tested matter? We can estimate how much uncertainty (what statisticians call "error") comes from each of these sources, and try to figure out if there's a structure/pattern underneath the noise, but in order to do that successfully you have to really think about the sources of the error. I think every time I've been really screwed over by an experiment, it's been because there was a source of variability or a kind of variability that I just didn't expect.