In the education bloggysphere, there has been a pissing match between the Cato Institute and the Shanker Institute over the legitimacy of claiming that, even as education costs have risen dramatically, NAEP test scores have essentially remained unchanged. I was going to let it slide until I read this in a Cato response:
Fortunately, there is a tried-and-true metric that researchers use to quantify effect sizes: they express them in terms of standard deviations, and those measures can in turn be converted to percentile scores. For example, the earliest available std deviation for the mean reading score of 17-year-olds was 46 points. Dividing 2 by 46, we get an effect size of 0.0435 SDs. That would take you from being in the middle of the pack in the early 1970s (that is, the 50th percentile), to being at the 51.7th percentile. So instead of outscoring half your peers, you’d outscore 51.7 percent of them. That’s not a huge difference is it? That’s not a spike-the-football, endzone dance, “In. Your. Face!” kind of improvement. It’s really pretty small.
In math, the story is similar. The earliest SD available is for the 1978 admin of the test, and it was 35. A two-point gain would be an effect size of 0.057 SDs, which would raise you from median performer to the 52.3rd percentile. Again, this is not winning the lottery. This is not an “I’d like to thank the Academy” kind of moment.
So the fact that the reading and math scores look essentially flat in the chart at the top of this post is an accurate representation of the trend in raw NAEP scores. They are essentially flat.
Ugh. Even if you take twelfth grade scores seriously–we’ll return to that in a bit–this misrepresents the case. Using the Long-Term Trend Data, if break out the mathematics data by parental educational status and race, we see a different picture (the picture is similar for reading). Scores for white students essentially are unchanged (in reading, they drop; my fellow honkeysoids, what’s going on?). But for black students, they typically increase around half a standard deviation, meaning that a student today would be at about the seventieth percentile in 1978. That’s not nothing. Hispanics move to around the sixty-fifth percentile give or take.
But here’s the thing: we probably shouldn’t take the twelfth grade data very seriously. The NAEP has always realized that high school seniors at that point know the test doesn’t mean anything, and they don’t take it very seriously (‘ridiculous’ answers such as blank pages, or all A’s are much higher). When we look at fourth and eighth grade scores, there are dramatic gains in performance over the last three decades. Within a socioeconomic group, the median student would typically fall between the 63rd and 85th percentile in the halcyon days of yore. That seems like a significant gain to me.
Of course, there are subtexts to this debate revolving around school funding (CUT ALL TEH BUJJITZ!!!), but the picture isn’t as bad as Cato paints it.