Keeping the week’s unofficial education theme, Kevin Drum posts about the latest “kids these days” study, namely the just-released NAEP Geography results. Kevin makes a decent point about the 12th grade questions being fairly sophisticated, but includes one comment that struck me as off base:
I gotta tell you: I went through the five sample questions for 12th graders, and they were pretty damn hard. This was not “identify France on a map” stuff. I ended up getting them all right, but I was half guessing on some of them.
He says that as if it’s a bad thing, but it seems to me that it’s really just about exactly right. If you want to test whether somebody really knows a subject, you should give them a test where they’re half guessing. Half guessing is what education is all about.
There’s a persistent image of education as a process of learning discrete facts and disgorging them later, but that’s really not a good mode to operate in. Unless your goal is to be a Jeopardy! champion, and even there, I doubt it’s really what you want.
Meaningful education doesn’t mean memorizing lists of facts, it means knowing what those facts mean, and how to combine them to do useful things. A big part of real education is knowing how to deal with incomplete information, and make reasonable judgments and guesses about things you’ve never directly learned. If you’ve just memorized facts, you’ll be paralyzed by those sorts of questions, but if you’ve actually learned the subject, you should be able to deduce the correct answer.
To take an example from the 12th grade sample questions, there’s this “identify a continent from its cross-section” question:
I don’t have height profiles for the various continents memorized, but I was able to get this right from reasoning through what I do know: The continent in question needs to have lots of mountains on its western side. North America would fit that, but it’s not one of the choices. Africa’s mountains are in the east, and drawing an east-west profile through Antarctica doesn’t make a great deal of sense. Which leaves Europe or South America. The Andes are on the west side of South America, with the Amazon to the east, so it’s a good fit; the exact profile of Europe is a little harder for me to picture, and would dpend a lot on where you draw the line between Europe and Asia. So South America was my guess, and it’s the right answer.
(The 23,000 feet thing is another clue, telling you that the mountains in question are really big. That requires a bit of additional trivia knowledge– Mt. Everest is about 29,000 ft, which means that the mountains of the mystery continent are really big, and rules out Europe.)
That kind of reasoning is exactly what we ought to be testing if we want to know whether students have learned anything, or just memorized facts. We want them to be half guessing based on what they know, rather than robotically reciting memorized answers.
This is why every exam I write has at least one problem on it with a part that’s different from any examples done in class. These questions use the same physics principles, but require students to put them together in a new way, rather than just following a familiar cookbook procedure. That’s the way you test whether somebody really knows the subject, as opposed to just committing a couple of example problems to memory.
(Of course, I also get plenty of grumbles that it’s “not fair” to ask anything on the exam that hasn’t been done multiple times in class. I have basically zero sympathy for this line, though.)
So, on the whole, I’m glad to see that the NAEP is asking these sorts of questions. I basically agree with Kevin’s larger point, too– if 70% of students are able to work their way through this kind of material, which is unlikely to come from a specific Geography class that would’ve let them memorize stuff– that’s not all that bad. But I wanted to speak against the idea that requiring guessing is unreasonable– if your test doesn’t contain at least some items where your students are at least half guessing, you’re doing something wrong.
Very interesting.
Along those lines, it is interesting to note that there is evidence that girls are less willing to guess on multiple choice format exams than boys:
http://www.jstor.org/pss/1434682
This is consistent with a general consensus that females are more risk averse than males.
There is also a stylized fact from social science research that when boys do well on a contest, they tend to attribute it to their ability; when they do poorly, they tend to attribute it to luck. For girls, the stylized fact goes in exactly the opposite direction: when they do well, they attribute it to luck, and when they do poorly, they attribute it to ability.
FWIW, I think a great big part of being a Jeopardy! champion is educated guessing, too. Knowing a huge pile of facts is very important, but it’s also important to know the relationships between these facts, and how to add them to the information provided in a question to come up with the answer. I suspect most of the other Jeopardy! champs of my acquaintance would agree.
I actually knew the answers to all of the questions EXCEPT the one above….and I guessed incorrectly.
i thought that was the “easiest” question – you can figure it out by thinking about the topography of the continents – as such, it seemed more like a “reason” than “rote” question
As far as the grumbling goes for the ‘unfair testing’, it would all depend on the teacher. Professor Orzel’s method is a wonderful way for students for learn, and is much more meaningful than memorizing lists of facts. However, if a professor did use that method, that kind of question would not be fair. As far as the method of teaching described above, I wish my education had been as profitable.