Teachers Matter, But We’re Not Measuring How They Really Matter

As we like to say here, you have to understand the limitations of your data. One of the fads that education ‘reformers’ really like is value-added measurement, also known as value-added testing: the basic idea is to use a statistical model to determine how much of a student’s improvement in test scores is due to his or her teacher, then this effect is assessed across all of the students belonging to a given teacher, and presto chango, we can discriminate ‘good’ from ‘bad’ teachers. I’ve discussed the problems with value-added testing before, so I won’t rehash them here.

One assumption underlying this approach is that test scores of subject materials are the outcomes we want to measure. But one could make a compelling argument that eighth grade math scores are a proxy for more important measures, such as finishing high school, staying out of trouble, and adult earnings. Many parents would ultimately be more worried about the latter than a specific test score in junior high. If most reformers are to be believed, our education ‘crisis’ is hurting our economic competitiveness (squishy librul concerns like being a well-rounded citizen don’t really seem to enter into the discussion…). So, if nothing else, education reformers should be interested in determining how teachers affect these long-term outcomes.

C. Kirabo Jackson gets at this in a recent paper (pdf is of non-paywall version; boldface mine):

Using administrative data with students linked to individual teachers, 9th grade English and Algebra teachers have economically meaningful effects on test scores, absences, suspensions, on time 10th grade enrollment, and grades. The results indicate that teachers have larger effects on non-cognitive ability than they do on cognitive ability. A variety of additional tests suggest that these effects can be interpreted causally. Teacher effects on test scores and teacher effects on non-cognitive ability are weakly correlated such that many teachers in the top of test score value-added will also be in the bottom of teachers at improving non-cognitive skills. This means that a large share of teachers thought to be highly effective based on test score performance will be no better than the average teacher at improving college-going or wages. Under reasonable assumptions about the importance of non-cognitive skills for long run outcomes, calculations indicate that test score based measures may understate the importance of teachers by between 50 and 80 percent.

This study highlights that a failure to account for the effect of educational interventions on non-cognitive skills can lead to biased estimates of the effect of such interventions on important long-run outcomes. The two-sample factor analytic framework put forth in this paper can be used in other settings to estimate the effects of educational interventions on both cognitive and non-cognitive ability. Finally, the results illustrate the large potential gains that may be associated with making policy decisions about educational interventions based on estimated effects on both cognitive and non-cognitive outcomes rather than just test scores alone.

Admittedly, this is one study, but this is really important–and probably why it hasn’t been discussed at all in the mainstream media. The idea that value-added testing focused on subject material is telling us very little about long-term outcomes, such as income and high school graduation, isn’t, for obvious reasons, being promoted by education ‘reformers.’ Pretty certain that Pearson, which makes lots of Common Core-associated tests and materials, won’t be discussing this.

But they really are concerned about the children.

This entry was posted in Education. Bookmark the permalink.

4 Responses to Teachers Matter, But We’re Not Measuring How They Really Matter

  1. 1) Mainstream media doesn’t talk about value-added teaching all that much.
    2) I think that’s because it’s a deep-research issue. A lot of work and not necessarily a lot of interest. (Compare Gawker)
    3) I did see some figures on effectiveness of various interventions at https://speakerdeck.com/pycon2014/software-carpentry-lessons-learned-by-greg-wilson slide 21.

    I think you’re right that the test scores aren’t measuring what’s really interesting. And what gets measured, gets managed.

  2. crprod says:

    In North Carolina, we get hit with this sort of thing all the time from John Hood, the public voice of the Art Pope think tank complex:
    Now that Tillis has won the Senatorial primacy, it will be interesting to see how well complaints about this and other actions of the Republicans in Raleigh manage to stick to him.

  3. Pingback: There Is a Male Education Gap, But It’s Very Specific | Mike the Mad Biologist

  4. Pingback: One Quick Thought About “Teacher Quality”: What Do Want Teachers to Do? | Mike the Mad Biologist

Comments are closed.