About That Cato Institute Education Chart

In the education bloggysphere, there has been a pissing match between the Cato Institute and the Shanker Institute over the legitimacy of claiming that, even as education costs have risen dramatically, NAEP test scores have essentially remained unchanged. I was going to let it slide until I read this in a Cato response:

Fortunately, there is a tried-and-true metric that researchers use to quantify effect sizes: they express them in terms of standard deviations, and those measures can in turn be converted to percentile scores. For example, the earliest available std deviation for the mean reading score of 17-year-olds was 46 points. Dividing 2 by 46, we get an effect size of 0.0435 SDs. That would take you from being in the middle of the pack in the early 1970s (that is, the 50th percentile), to being at the 51.7th percentile. So instead of outscoring half your peers, you’d outscore 51.7 percent of them. That’s not a huge difference is it? That’s not a spike-the-football, endzone dance, “In. Your. Face!” kind of improvement. It’s really pretty small.

In math, the story is similar. The earliest SD available is for the 1978 admin of the test, and it was 35. A two-point gain would be an effect size of 0.057 SDs, which would raise you from median performer to the 52.3rd percentile. Again, this is not winning the lottery. This is not an “I’d like to thank the Academy” kind of moment.

So the fact that the reading and math scores look essentially flat in the chart at the top of this post is an accurate representation of the trend in raw NAEP scores. They are essentially flat.

Ugh. Even if you take twelfth grade scores seriously–we’ll return to that in a bit–this misrepresents the case. Using the Long-Term Trend Data, if break out the mathematics data by parental educational status and race, we see a different picture (the picture is similar for reading). Scores for white students essentially are unchanged (in reading, they drop; my fellow honkeysoids, what’s going on?). But for black students, they typically increase around half a standard deviation, meaning that a student today would be at about the seventieth percentile in 1978. That’s not nothing. Hispanics move to around the sixty-fifth percentile give or take.

But here’s the thing: we probably shouldn’t take the twelfth grade data very seriously. The NAEP has always realized that high school seniors at that point know the test doesn’t mean anything, and they don’t take it very seriously (‘ridiculous’ answers such as blank pages, or all A’s are much higher). When we look at fourth and eighth grade scores, there are dramatic gains in performance over the last three decades. Within a socioeconomic group, the median student would typically fall between the 63rd and 85th percentile in the halcyon days of yore. That seems like a significant gain to me.

Of course, there are subtexts to this debate revolving around school funding (CUT ALL TEH BUJJITZ!!!), but the picture isn’t as bad as Cato paints it.

This entry was posted in Education. Bookmark the permalink.

2 Responses to About That Cato Institute Education Chart

  1. harrync says:

    I would have liked to have seen the chart break down total costs into teachers compensation and “other” – buildings, equipment, supplies, administration, etc. I think you would probably find student performance fairly closely tracked teacher pay, and that all the “waste” would be in “other.” (Of course, not all the increased expense would be actual waste; as conditions change, sometimes it is necessary to spend more just to stay where you are.) If what I suspect is correct, then by Cato logic, the way to get higher test scores is to increase teacher pay. Actually, I think I saw a study which showed that was the case in a state by state comparison – higher pay, higher scores. But like any such comparison, it may just be correlation, not cause.

  2. dr2chase says:

    Are they looking only at inflation-adjusted dollars for pay? Have they never heard of Baumol’s cost disease? Teachers are people who are smart enough and go to school long enough that they could have chosen some other profession if teaching had looked unattractive enough — if their pay does not keep pace with those alternatives, we’ll end up with fewer teachers or less-skilled teachers. Cato can whinge all they want about their mythical productivity measures not keeping pace with pay, but That Is Fucking Irrelevant. Baumol’s Cost Disease, tough shit, you’ll pay the wages or settle for less.

    Morons. As if I expected anything else from Cato.

Comments are closed.