Study reenactment: Evelyn Rose, 4 of Brighton, N.Y., participates in a reenactment of the marshmallow experiment used in a University of Rochester study published in the journal Cognition. The study was conducted at the University of Rochester Baby Lab. By J. Adam Fenster / University of Rochester

Something that bothers me (among many, many things that do…) is the use of the phrase ‘get technical’ as in “I don’t want to get technical.” I won’t claim I’ve never used the phrase, but it’s really inappropriate when science bumps up against public policy, since the quality and validity of the science depends on the technique. Good science, be it physical or social science, requires proper technique. That includes design and analysis, as well as the scope of claims one draws from the results. So technical details aren’t something to be glossed over–they have to be rigorously assessed. Put another way: if your methods are crap, then your results are garbage. If you are thinking of discussing those results, you need to shut your piehole.

That’s why the ‘education reform’ program of assessing teacher quality using value-added testing is idiotic. The methods are too imprecise and too variable to assess teacher performance. In terms of the scope of claims, there is no recognition that a testing program that determines if a teacher will lose her job will have the effect of altering what those tests are actually measuring. So it’s crappy technique, even if some of the explanations of why it’s crappy technique, well, get technical.

Which brings me to a classic behavorial study, often referred to as the Marshmallow study. Children were placed in a room and told that if they waited to eat a marshmallow, they would get two marshmallows. Years later, a significant difference in, well, lots of things was found to be strongly correlated with the ability to delay gratification (boldface mine):

The research builds on a long series of marshmallow-related studies that began at Stanford University in the late 1960s. Walter Mischel and other researchers famously showed that individual differences in the ability to delay gratification on this simple task correlated strongly with success in later life. Longer wait times as a child were linked years later to higher SAT scores, less substance abuse, and parental reports of better social skills.

Because of the surprising correlation, the landmark marshmallow studies have been cited as evidence that qualities like self-control or emotional intelligence in general may be more important to navigating life successfully than more traditional measures of intelligence, such as IQ.

This study was often interpreted as children with the innate ability of delay of self-gratification would do better in life. Needless to say, the corrollary–poor kids lack this self-control–was used in popular and policy treatments to argue that poverty was, to a considerable extent, innate (or at least formed very early in childhood). If so, there might not be that much we could or should do to ameliorate the effects of poverty and deprivation.

Decades later, researchers revisited the study, but this time they added a twist:

The Rochester team wanted to explore more closely why some preschoolers are able to resist the marshmallow while others succumb to licking, nibbling, and eventually swallowing the sugary treat. The researchers assigned 28 three- to five-year-olds to two contrasting environments: unreliable and reliable. The study results were so strong that a larger sample group was not required to ensure statistical accuracy and other factors, like the influence of hunger, were accounted for by randomly assigning participants to the two groups, according to the researchers. In both groups the children were given a create-your-own-cup kit and asked to decorate the blank paper that would be inserted in the cup.

In the unreliable condition, the children were provided a container of used crayons and told that if they could wait, the researcher would return shortly with a bigger and better set of new art supplies for their project. After two and a half minutes, the research returned with this explanation: “I’m sorry, but I made a mistake. We don’t have any other art supplies after all. But why don’t you use these instead?” She then helped to open the crayon container.

Next a quarter-inch sticker was placed on the table and the child was told that if he or she could wait, the researcher would return with a large selection of better stickers to use. After the same wait, the researcher again returned empty handed.

The reliable group experienced the same set up, but the researcher returned with the promised materials: first with a rotating tray full of art supplies and the next time with five to seven large, die-cut stickers.

What the researchers found is that kids in the unreliable environment were far more likely to eat the marshmallow than kids in the reliable group:

Children who experienced unreliable interactions with an experimenter waited for a mean time of three minutes and two seconds on the subsequent marshmallow task, while youngsters who experienced reliable interactions held out for 12 minutes and two seconds. Only one of the 14 children in the unreliable group waited the full 15 minutes, compared to nine children in the reliable condition….

In prior research, children’s wait time averaged between 6.08 and 5.71 minutes, the authors report. By comparison, manipulating the environment doubled wait times in the reliable condition and halved the time in the unreliable scenario. Previous studies that explored the effect of teaching children waiting strategies showed smaller effects, the authors report. Hiding the treat from view boosted wait times by 3.75 minutes, while encouraging children to think about the larger reward added 2.53 minutes.

The short-term cognitive environment–that is, what happened briefly before the test–had a massive effect on the results (the reliable environment also seemed to lead to ‘clever’ solutions: one kid sat on the marshmallow, which if you don’t care about your clothes, is pretty smart). Impulsiveness seems to be more variable than we thought. However, I’m not sure I agree with this:

The robust effect of manipulating the environment, conclude the authors, provides strong evidence that children’s wait times reflect rational decision making about the probability of reward. The results are consistent with other research showing that children are sensitive to uncertainly in future rewards and with population studies showing children with absent fathers prefer more immediate rewards over larger but delayed ones.

This might not have anything to do with probability of reward or high versus low-trust environments. It might simply be ‘crankiness’: I didn’t get a toy, I’m frustrated, I’ll eat the marshmallow. Hard to say, but adults think this way (consider how hard it is to eat right when you’re having a bad day).

This is why I’m skeptical of a lot of the behavioral genetics work: these are complex behaviors that can be strongly influenced by seemingly small changes in environment. The confounding variables can really throw a spanner into the works. Things like home environment during childhood do matter. Animal and plant breeders are far more rigorous (and are able to be more rigorous given their experimental system) than much of the behavioral genetics work. Technique matters.

Tangentially related: Razib runs into similar problems.