When Data Will Make You Reconsider–And They Don’t Have to Be ‘Big’

When it comes to policy, it’s not clear to me that ‘big data’ will be any more effective in convincing people to change their minds, as I’ve noted regarding educational data:

One of the themes that I continually flog like a rented mule that I often return to is the misuse of basic educational data. For example, the common claim that U.S. students have shown no improvement is both incorrect and widespread, to the point where even a smart, datamonkey like Bill Gates made this mistake (though he has since backed off this claim). In fact, this misperception was so widely held that several years ago, the only prominent bloggers (and print commentators too) that were making this point were Kevin Drum, Bob Somerby, and yours truly. And the only reason I consider including myself on the ‘prominent’ list is because there was no one else making this point; I win by default. If there were others, they were so few and far between that I, not being a professional education expert, never encountered them. So why were so few making this argument? Well, there were a lot of people ideologically, politically and economically invested in the idea that the U.S. educational system, across the board, was poor and making no gains*.

The reason I raise this example is that the revelatory data were not ‘big data.’ In fact, any NAEP dataset, by my day job’s standards (genomics) is quite puny (BIG DATAZ! I HAZ IT!).

Before I actually looked at the data, I, like many believed this misperception, and the data did change my mind–though I’ll admit that changing my mind didn’t require me to overturn deeply held and integral beliefs (like the belief that teachers unions are evil). But I find it interesting that the two people I referred to also changed their minds. Bob Somerby:

When we read that passage by Brooks, we wondered if we had ever switched sides on some topic based upon data. We thought of how amazed we were when, if we might borrow from Keats, we first star’d at the nation’s NAEP scores.

We’ll guess this was roughly ten years ago. For about the millionth time, we had seen the federally-run National Assessment of Educational Progress (NAEP) described as the “gold standard” of educational testing.

We decided to go to the sprawling NAEP site to see what it contained.

We were stunned by what we saw there. We saw that, according to this gold standard program, black kids had been making large progress in reading and math down through the years. (Hispanic kids too.)

Like everyone else who reads newspapers, we had never heard this. Like everyone else who reads newspapers, we had endlessly heard that our schools were awful and getting worse—that absolutely nothing was working.

And Kevin Drum:

Like Bob, I was also surprised the first time I really started to dig into test score data: it showed pretty clearly that we’ve made consistent progress over the past three decades, especially at the elementary school level. It turns out that American schools aren’t in terminal decline.

But the common feature that the three of us share (I think) is that we weren’t emotionally invested in the narrative of educational failure. In other words, data will change minds when it doesn’t require people to tear down cherished ideological or theological constructs. Whenever I mention that Massachusetts has one of the best educational systems in the world, there is a tremendous amount of pushback, in part because it undercuts many different narratives, from the general failure story to the importance of curriculum.

We do like our stories, and that seems impervious to the size of the data.

This entry was posted in Education, Statistics. Bookmark the permalink.

1 Response to When Data Will Make You Reconsider–And They Don’t Have to Be ‘Big’

  1. dr2chase says:

    Sure, I used to think that universal health care would be the right thing to do, even though it might cost more — but the cost made me think it could be a hard sell to some people. Then I looked at what other countries were spending (per capita, % of GDP) and at their life expectancy. Holy fucking-fuck — so why are all these other people opposed to living longer on the cheap?

    And I used to think that riding a bike was a good way to get around, but there was a crash risk, so there’s tradeoffs, and I’m not sure I can recommend it to everyone. Then I heard Mayer Hillman’s back-of-the-envelope 10:1 estimates for the exercise:crash mortality benefit, and then I saw this study ( http://archinte.jamanetwork.com/article.aspx?articleid=485349 ) indicating a 39% higher mortality for the non-cycle-commuters. Another related study suggests 2-5 years of extra life.

Comments are closed.