One of the flaws–a bad frame if you must–in the communication debate flareups that happen every so often is that the debate (such as it is) is phrased as “why can’t scientists communicate?” This is actually an imprecise and incorrect way of stating the problem. It should be stated as “How do scientists convincingly explain that misinformation is actually misinformation?” Once we phrase it like that, we realize that the problem isn’t that we need to ‘stop being such a scientist’, because no one has a particularly good strategy for dealing with misinformation.
To use an example that won’t (maybe) cause too much unnecessary flaming, here’s some coverage of a recent review of the KIPP charter schools:
At about half the KIPP schools, the study found that the gains in math for students after three years in the schools were equivalent of 1.2 years of extra instruction, and .9 years of additional instruction in reading…
On average, the report says, KIPP middle schools have students who are more likely to be living in poverty and are more likely to be black or Hispanic than are students from the schools around them. Back when they were in 4th grade, the study also found, a majority of the KIPP middle school students had lower test scores on average than did students in their local school districts.
Sounds great! Is our children learning? Yes! But there’s a problem–and it is very technical, but, nonetheless, critical:
However, an initial analysis of the report by Professor Gary Miron of Western Michigan University concludes that this initial study report misrepresents the attrition data. According to Miron, “While it may be true that attrition rates for KIPP schools and surrounding districts are similar, there is a big difference: KIPP does not generally fill empty places with the weaker students who are moving from school to school. Traditional public schools must receive all students who wish to attend, so the lower-performing students leaving KIPP schools receive a place in those schools.”
In contrast, Miron explains, “The lower performing, transient students coming from traditional public schools are not given a place in KIPP, since those schools generally only take students in during the initial intake grade, whether this be 5th or 6th grade.”
The KIPP study’s description of attrition only considers half the equation, when comparing KIPP schools to matched traditional public schools. The researchers looked at the attrition rates, which they found to be similar – in the sense of the number of students departing from schools. But they never considered the receiving or intake rate. Even though the researchers agree that the students who are mobile are lower performing, they do not take into account the reality that KIPP schools do not generally receive these students.
This is highly technical, but really critical*. It’s pretty boring, actually. While a few people will have the time and inclination to follow this, most won’t. So what’s the effective pushback? This is really outside the boundaries of framing, since this truly is a matter of analysis. Once we reach the ‘dueling experts’ stage, all that’s left are ad hominem attacks: my expert is better than yours.
*KIPP schools do perform better, but if they had to retain poorly performing students, it’s not clear how KIPP schools would hold up.