Kevin Drum, on yet another education reform ‘success’ that turns out to be a mirage (boldface mine):
My cynicism about the ed reform community grows by leaps and bounds every time I read a story like this. And that’s pretty often. Here’s my advice for what you should do whenever you read an article about a school that’s shown miraculous results by applying some reform or another (or by hiring a miracle worker of some stripe or another):
1. Don’t believe it if it’s based on a single school or other small sample.
2. Don’t believe it if most of the evidence comes from the school itself.
3. Don’t believe it if the reform in question was put in place only a few years ago.
4. Don’t believe it if it hasn’t been replicated elsewhere.
5. Don’t believe it unless it’s been rigorously tested by academics who didn’t already support the idea in the first place.
6. And even if it passes all those tests, don’t believe it anyway.
The number of ed reforms that hold up when the evidence is looked at critically seems to be tiny. The number that continue to work when they’re scaled up seems to be tiny. The number that continue to show results all the way through high school seems to be tiny. The number that can withstand critical scrutiny seems to be tiny. And of the ones that are left, the cost to keep them up usually appears to be prohibitive.
The biggest problem–and one that probably unites some conservatives with a wild-eyed liberal like me–is that much, if not most, of what determines academic success comes from outside of the classroom. Much of the remainder is systemic (e.g., curriculum). Those things do not lend themselves to quick, cheap fixes.