Eight years ago, I noted the following about NAEP scores (considered to be the gold standard for educational testing):
One of the things often heard is that someone is leaving the city for the burbs because the schools are better (I use the generic city, since, in my experience this attitude doesn’t appear to be limited to any particular city). But what if parents aren’t choosing better schools, but better student bodies? What if parents are paying exorbitant housing costs, not because the schools perform better, but because those high housing costs are able to exclude students who perform poorly?
…This isn’t to say that individual students can’t over- or underperform. And populations (states or schools) can defy expectations: at the state level, Massachusetts scores twelve points higher than it should. Demography may not be destiny, but it is a heavy burden.
In the above example, there is virtually no way the poor population will ever appear as ‘good’ as the wealthy school, even if the poor school does a better job of educating its students–that is, getting them to achieve more than they should based on their socioeconomic status. Even if you compare a school with a ten percent poverty rate (half the national average) to a school with a two percent poverty rate, only a quarter of the time will the ‘poorer’ school perform better.
The point is when parents are choosing schools based on test scores, they are not necessarily assessing school quality, but child poverty. The educational system that they’re leaving might stink too, but there is a massive conflation going on here. Even if they don’t think they’re doing so, families who are moving in order to secure a better education are, to a considerable extent, fleeing ‘undesirable’ student bodies.
Well, eight years later, it’s seems this argument has become more broadly accepted (boldface mine):
Harvard students have remarkable post-collegiate outcomes, academically and professionally. But then, Harvard invests millions of dollars carefully managing their incoming student bodies. The truth is most Harvard students are going to be fine wherever they go, and so our assumptions about the quality of Harvard’s education itself are called into question. Or consider exclusive public high schools like New York’s Stuyvesant, a remarkably competitive institution where the city’s best and brightest students compete to enroll, thanks to the great educational benefits of attending. After all, the alumni of high schools such as Stuyvesant are a veritable Who’s Who of high achievers and success stories; those schools must be of unusually high quality. Except that attending those high schools simply doesn’t matter in terms of conventional educational outcomes. When you look at the edge cases – when you restrict your analysis to those students who are among the last let into such schools and those who are among the last left out – you find no statistically meaningful differences between them. Of course, when you have a mechanism in place to screen out all of the students with the biggest disadvantages, you end up with an impressive-looking set of alumni. The admissions procedures at these schools don’t determine which students get the benefit of a better education; the perception of a better education is itself an artifact of the admissions procedure. The screening mechanism is the educational mechanism.
Thinking about selection bias compels us to consider our perceptions of educational cause and effect in general. A common complaint of liberal education reformers is that students who face consistent achievement gaps, such as poor minority students, suffer because they are systematically excluded from the best schools, screened out by high housing prices in these affluent, white districts. But what if this confuses cause and effect? Isn’t it more likely that we perceive those districts to be the best precisely because they effectively exclude students who suffer under the burdens of racial discrimination and poverty? Of course schools look good when, through geography and policy, they are responsible for educating only those students who receive the greatest socioeconomic advantages our society provides. But this reversal of perceived cause and effect is almost entirely absent from education talk, in either liberal or conservative media.
While I think that post is too pessimistic about selection bias, it’s pretty clear that those dependent variables do matter, as I noted about charter schools in D.C.:
Anyway, when the percentage of students who are low-income is plotted against the average percentage of students who were proficient or advanced, there is a significant charter school effect: a charter school, on average, has the same effect as a fifteen point reduced in the percentage of low-income students (for every additional percentage point of low-income students, proficiency drops by 0.52%)…
When the same analysis is performed except using At-Risk students instead of low-income, the charter school effect essentially vanishes (charter schools increase the average percentage of proficient or advanced students by less than three percentage points, but this variable is not statistically significant, with p = 0.154). The model has an R-squared of 0.638, and the effect of At-Risk students is much greater than before, with every additional percentage point of low-income students decreasing proficiency by 0.76% percentage points.
…what this does mean, however, is that all of the ‘pre-processing’ steps can dramatically influence analysis outcomes. Simply by changing what we mean by ‘low-income’ (that is, kids whose lives are pretty miserable), we can get a fundamentally different result: we can move from charters have no significant effect to have a large one. It also means that maybe we need to revisit some of the charter school analyses, using better dependent variables. A child in a home with earnings of $35,000/yr isn’t demographically equivalent to one in a household with annual earnings of $10,000/yr–and shouldn’t be treated as such.
As I wrote at the beginning, this shouldn’t be viewed as anything close to definitive. But I think there’s something to the criticism that how we classify students might affect our conclusions regarding various interventions, including charters (i.e., the CREDO study).
Understanding how your data ‘pre-processing’ affects your analysis is critical.
Selection bias and dependent variables do matter.