Back when I worked on the microbiome, I noted several times that you have to understand the limitations of your data. You are always measuring something, but you might be pushing the interpretation of those measurements too far.
Recently, I criticized psychologist Laurence Steinberg for claiming that twelfth grade NAEP scores have flat-lined (ZOMG! TEH STUDENTZ ARE STUPID!). Of course, as is usually the case with such claims, once you disaggregate the scores (i.e., break them down by demographic factors), all students have made gains. But here’s something I didn’t know about the twelfth grade NAEP scores (boldface mine):
I responded to Larry Steinberg, whom I knew years ago, and pointed out that the NAEP scores for seniors are meaningless. When I was on the NAEP board in the early 2000s, we devoted a full meeting to discussing the fact that high school seniors don’t even try on NAEP tests. They know the tests don’t count towards high school graduation or college admission; they don’t count for anything, and the kids don’t care about them. They doodle on the answer page, they answer in patterns (like checking off every A), or they leave pages blank. They aren’t dumb. They know what they are doing. They are asked to jump over a meaningless hurdle, and they treat it as a joke. But the adults take their tomfoolery as evidence that they are unmotivated, possibly stupid. I don’t think the kids are stupid. I imagine how I feel when someone calls me on the phone and starts asking questions; usually I hang up, or I say something uncooperative because I don’t like to be interrupted for no reason to fit into someone else’s plan. I expect that the seniors feel the same way.
Interesting. We might want to reconsider the claims that the upper grades in the U.S. are doing poorly. Maybe, instead, they’re doing fine: wisdom might be too strong a word for a seventeen year-old, but they seem to have their heads screwed on better than some would like to credit.
Of course, I’m sure I failed to persuade you if you’re convinced there’s a problem no matter what.