{"id":684,"date":"2006-10-07T10:04:41","date_gmt":"2006-10-07T10:04:41","guid":{"rendered":"http:\/\/scienceblogs.com\/principles\/2006\/10\/07\/how-to-lie-with-test-scores\/"},"modified":"2006-10-07T10:04:41","modified_gmt":"2006-10-07T10:04:41","slug":"how-to-lie-with-test-scores","status":"publish","type":"post","link":"http:\/\/chadorzel.com\/principles\/2006\/10\/07\/how-to-lie-with-test-scores\/","title":{"rendered":"How to Lie With Test Scores"},"content":{"rendered":"

Sean Carroll comments on<\/a> an item in the Atlantic Monthly<\/a> on test scores compared across nations. There are two things that really bug me about this item, the most important of which is the deeply dishonest graphic the Atlantic did to illustrate the item.<\/p>\n

Here’s the honest version of the graph, redone using data from this table<\/a> (the relevant figures don’t appear in the report cited in the original piece<\/a>). (Click on the graph for a larger version.) <\/p>\n

\"i-ba518544db2aa101dc9208f14031d61f-med_TIMMS.jpg\"<\/a><\/p>\n

I’ve plotted the normalized test score (the score for each country divided by the reported maximum score, because I’m a physicist and like numbers between 0 and 1) for the same seven countries cited in the original. I made the simplfying assumption that the minimum score is zero, because I can’t seem to find a clear statement of what the actual score range is. It may very well be something like 200-800, because the people who design standardized tests like to make the results as confusing as possible, but that doesn’t really change the point.<\/p>\n

If you look at that graph, you wouldn’t say that there’s a major crisis in American science education. Yeah, we’re at the bottom of the range, with a score of 66% compared to Singapore’s 72%, but the difference isn’t all that huge.<\/p>\n

The dishonest version of the graphic is below the fold:<\/p>\n

<\/p>\n

\"i-ad8b90f6277d51c9a3a589d5dcfac69d-eighthgrademath.jpg\"<\/p>\n

The top part of the graphic is the same set of scores, rescaled to make everything look more dramatic. If you zoom way in, suddenly the US bar is one-third the height of Singapore’s. Aiieeee! Crisis in science education!<\/p>\n

Now, look, I’m not going to argue that we couldn’t be doing a better job with science education– obviously, we could. But this sort of cheap graphical manipulation is straight out of How to Lie with Statistcs<\/cite>. The sky isn’t actually falling, and we’ve actually closed the gap somewhat since the 1995 test– the US score went from 513 to 527, while the top-scoreing countries remained the same (Singapore dropped from 580 to 578, and Japan went from 552 to 554). Granted, our score increase was nothing compared to Hong Kong, which jumped from 510 to 556, but it’s not completely trivial, either.<\/p>\n

Even if the point is to scare people into taking positive actions, though, there’s just no excuse for this level of graphical deception.<\/p>\n","protected":false},"excerpt":{"rendered":"

Sean Carroll comments on an item in the Atlantic Monthly on test scores compared across nations. There are two things that really bug me about this item, the most important of which is the deeply dishonest graphic the Atlantic did to illustrate the item. Here’s the honest version of the graph, redone using data from… Continue reading How to Lie With Test Scores<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"1","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8,13,33,7,11],"tags":[],"class_list":["post-684","post","type-post","status-publish","format-standard","hentry","category-academia","category-education","category-in_the_news","category-physics","category-science","entry"],"_links":{"self":[{"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/posts\/684","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/comments?post=684"}],"version-history":[{"count":0,"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/posts\/684\/revisions"}],"wp:attachment":[{"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/media?parent=684"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/categories?post=684"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/tags?post=684"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}