Many moons ago, I used to book and promote concerts, so I don’t pose that question with disdain towards people who put shows together. Scientists, on the other hand…. (heh)

In large part due to the media blitz surrounding Amanda Ripley’s book on education, *The Smartest Kids in the World*, in which she dwells almost exclusively on PISA scores for international comparisons, the results of the PISA international comparisons in math and reading are back in the news–tests on which the U.S. is middling and which are also very problematic. It also doesn’t help that Bill Keller of the *NY Times* is pimping this book, given his basic lack of understanding of educational statistics–he could worse than read Mikoto Rich’s excellent reporting in his very own newspaper (and let’s not forget Bill Keller’s cheerleading for the Iraq War, so I have complete faith in his ability to get the policy right…).

Anyway, back to the PISA evaluations. Ripley refers to PISA as “the most respected test of teenagers in the world.” It’s essentially a constructivist test. In a recent response to critics, Ripley described PISA thusly:

Unlike TIMSS, PISA was designed to test students’ abilities to apply knowledge to solve real-world problems and think for themselves. (TIMSS is a test of school curriculum.) I was most interested in those higher-order thinking skills, since they are increasingly valuable in the modern economy. To see if the hype on PISA was true, I took the test myself, and I found it to be a remarkably sophisticated test.

Here’s a sample question–I’m going to put the answer below the fold (no cheating!):

For a rock concert a rectangular field of size 100 m by 50 m was reserved for the audience. The concert was completely sold out and the field was full with all the fans standing. Which one of the following is likely to be the best estimate of the total number of people attending the concert?

A. 2000

B. 5000

C. 20000

D. 50000

E. 100000

The answer?

20,000. I, like a lot of people, got this wrong because, given a choice of one or four people per square meter (the field is 5,000 square meters), I thought one person was the better answer–because there are things called fire codes (promoted and booked concerts, remember?). Had it been fill-in-the-blank, I would have probably gone with 10,000 or 15,000 thousand.

This type of question highlights two problems with supposedly common sense, real world questions. One problem is that this is a typical culturally variant question. People used to crowds would probably answer 20,000, while those from Big Sky country would probably feel claustrophobic at one person per square meter. As we’ve discussed before, this sort of cultural variance is a massive problem when comparing countries.

But there is a more significant problem, and it gets back to what the fundamental goal of a mathematics education should be. If the goal is to enable students to tackle real-world problems, PISA isn’t bad. But the problem is that college mathematics–and the sciences–require formalism and abstraction. You need to be able think about the question posed above as “*4xy*“. This affects many fields. I could, if I choose to do so, explain the very basics of the neutral theory of evolution without recourse to math. But to do anything even slightly more difficult, you need to be familiar with formal math. That’s before you get to the harder stuff. Thinking creatively is great, but, at some point, you have to *know* things and have a facile use of abstraction. How you test–that is, what is important for students to learn–matters. Do we want a testing regime that would lead to smart, but mathematically untrained, college students. Will we offer them remedial mathematics in college if they suddenly decide to be STEM majors? Given the cost of college, is that an ethical thing to do?

Lest you think I’m overstating the problem, it’s worth noting that two hundred Finnish college math professors don’t like Finland’s curriculum for similar reasons (pdf, p. 11; boldface mine)):

When PISA results showed Finland to be the top country in the world in math, a group of more than two hundred university mathematicians in Finland petitioned the Finnish education ministry to complain that, regardless of what PISA was indicating,

students increasingly were arriving in their classrooms unprepared in mathematics. Knowledge of fractions and algebra were singled out as particularly weak areas. Two signers of the petition posed the question, “[A]re the Finnish basic schools stressing too much numerical problems of the type emphasized in the PISA study, and are other countries, instead, stressing algebra, thus guaranteeing a better foundation for mathematical studies in upper secondary schools and in universities and polytechnics.” One Finnish researcher, analyzing national data, compared the math skills of 15- and 16-yearolds on tests given in 1981 and 2003.Sharp declines were registered on calculations involving whole numbers, fractions, and exponents. The explanation: “‘Problem Solving’ and putting emphasis on calculators have taken time from explaining the basic principles and ideas in mathematics.”

I dunno, but fractions and exponents are kinda important. Algebra is pretty key too. But I’m a scientist so what do I know?

Obviously, most people aren’t going to use high level mathematics (though many skilled trades do). But high school should enable most, if not all, students to take college mathematics. Because concert planners are cool (even if I was one!), but we also need people who use formal math.

Cuz science.

I went with 1 per square meter. And where are the 100 porta-potties in all this? It’s like all those “city of the future” ideas with hardly any people dotting the landscape.

Like you, I would have gone for 10000 if a choice. I was about to go for 5000, when I noticed this was equal to 50 times 100 – the given size of the area. Figuring that this meant that there was a good chance the test writer was trying to fake me into picking 5000, I went with 20000. Thinking like this has got me test results in the 99.9 percentile on most of the SAT type tests I took back in the 1960’s. Unfortunately, there is no professional test taking circuit, so this skill doesn’t seem to have been of much use, other than getting me into better colleges than I otherwise would have been admitted to.

Let me defend PISA and the Finland math curriculum. (Let me say that this is not because of any problems I have with math. When I was 13 I was allowed to take the Mathematical Association of America test, which was normally given to high school seniors in our school, and not only beat the seniors but placed third in our multi-state region.) The US is an innumerate culture. Many, if not most adults cannot handle fractions, much less algebra. Not a few literate people seem to take a perverse pride in not doing math. A curriculum that prepared students to use math in everyday life would be a boon.

The key to the test question, aside from recognizing that 5,000 is likely to be the honeypot answer, is the phrase, “with all fans standing”. One square meter is plenty of room to sit down. Surely at least some of the fans would be sitting, if not most of them. At the same time, 20,000 is not really a real world figure, either. I suppose that the test makers thought that that number would ensure that nobody could sit down. (Unfortunately, this question, like many test questions, requires you to psyche out the test makers. They do not state that there are too many people for anyone to sit don. The fans could be standing because of excitement.)

At the same time, you have to provide formal math in the college prep courses. The people who would take those courses should have no problem applying everyday math.

“Surely at least some of the fans would be sitting…” Nope. If a concert is stand-up (actually went to one once, decades after school), there is probably no seating available. Your common sense is wrong. You are doomed to failure. Dooooooooomed, I tells ya.

***

If you follow the “sample question” link you will find a comment by Researcher2 who asserts “…common sense, attending concerts/football games, could have told you the best estimate)”

What the hell is a concert/football game?

I will assert this: at football games everyone is seated. There is no place to stand. I’ve never been to a football game or seen more than moments of one on TV, but I know my assertion is true. Or is R2 talking about soccer where the rabble like to press up as close to the restraining barrier as they can get, in which case density would seem to be about 8 per square metre, making the “common sense” answer off by a full order of magnitude (base 2).

This is a question with extremely simple arithmetic. To apply the arithmetic, it has to be recognized that it is a question not quite 4xy, as Mike states, but kxy. The “k” is the part that a person who uses “common sense” will realize they may well not know and for which they have to seek additional input. The make-it-up-on-the-fly types will pull numbers from thin air. Like the NITBAFS lying idiot. In the original question, any value of k between 1 and 8 might be “reasonable” (though I would leave for k > 2).

I would say this to those who set the question or maintain it is valid (like R2): prove to me that it provides any reasonable assessment of math ability. Do not rely on “common sense.” Do not make up the answer. Do not extract k through a terminal sphincter. Oh, and hold up your hands one metre apart.

A field is 500 metres by 1200 metres. All of the pumpkins grown there are pre-sold and will be used for Jack-o’-Lanterns. What is the best estimate of how many pumpkins the field will yield?

I can just picture the people who set some of the questions (like the those in Mike’s post of a few days back about a test for 11 year olds) positively quivering with the giddiness at how very clever they are.

As I was going to Saint Ives, I met a man with …

“Hello. Which way are you headed?”

My thoughts were pretty much the same as yours, I was unwilling to pick the higher number because it didn’t feel safe. But in hindsight I notice that I got the question wrong: they wanted to know the “best estimate”, and yes, 20k is a better guess than 5k.

An excellent post — many thanks.

Like most of us, I went for the 20,000 figure, on the grounds that 1 person per metre^2 seemed ‘way too low, but would have preferred a 10,000 or (at a push) 15,000 option. I think this reveals another basic problem of modern maths tests: that the multiple-choice format can mislead — not only the people taking the test but the interpreters of the results; I very, very much doubt, for example, that I’m a better mathematician than you, but I got the right answer (partly through psyching out the examining mentality) and you the wrong one — which have had the examiner recording the reality of our maths abilities exactly the wrong way round.

As an aside, at least it’s nice that they’re using metric units.

/sub

I went with 5000 – having been to many of those in the field type rock festivals when I lived and went to school in Finland (!) and Denmark when I was in my teens and early 20s. We always had room to stretch out our arms and sit with legs out – 1 per meter squared sounds about right. I ran circles around the Finnish students in Math and Science back even though I went to a specialized math and science high school in Finland (even with limited language skills because – math) then, and my daughter who is exactly the same age (12 hours difference in birth time) runs circles around my host sister’s daughter in math and science now , college age, even though mine went to a run of the mill MA public high school and my host’s sister went to a Math and Science High school. However, the skill of the Finns and other Scandinavian’s at speaking multiple languages puts Americans to shame. It is all about priorities in high school level schooling.