As long as I’m picking on education research papers in Science, I might as well call out the one immediately after the paper I wrote up in the previous post. This one, titled Graduate Students’ Teaching Experiences Improve Their Methodological Research Skills, is another paper whose basic premise I generally agree with– they found that graduate students who had teaching responsibilities as well as research responsibilities did a better job of writing research proposals than graduate students who only did research. From all appearances, it’s a good study, and makes a valuable point.
And yet, I have a hard time not mocking a paper that includes the following data graph:
I mean, really? You have two data points with error bars. does this really need to be presented as a graph? Let alone one playing “How to Lie With Statistics” games with the axis limits?
Wouldn’t a table be just as effective? In the space taken up by the two basically identical graphs of this form, you could’ve put a single table with all of your data, not just data on two selected subcategories of your evaluation. You could even have included a table with a detailed statistical breakdown in a convenient easy-to-read format, rather than scattering the results through the text.
But, I guess it isn’t Science-worthy if it doesn’t have a graph. Even an incredibly silly one.
I disagree with you on this one. A graphical display can be valuable for communication as long as it gives a visual impression of data that makes some things more readily apparent than the corresponding table does. So, for example, inferring that these error bars are almost non-overlapping (assuming they are calculated correctly, etc.) is much easier with a graph than with a table. With a table you would have to mentally add two numbers, then subtract two numbers, and compare the two resulting numbers. That requires a lot of working memory and arithmetic ability. You could argue that people should be skilled at this, and happy to do it, but unfortunately they are neither, so the graph can do a much better job of communicating the main point. Ideally the graphical display and the precise numbers should both be included in a scientific paper, in my opinion.
If you really want to convey that the uncertainty regions (those are 95% confidence limits, by the way, which I should’ve said in the post) just barelt overlap, there are lots of other ways to do that that don’t take up nearly so much space. You can quote maximum and minimum ranges for all points, for example, or put columns in the table for the difference and its statistical significance (which is the number that really matters). They could’ve left out both figures, and replaced them with Table S5 from the Supporting Online Material, which contains all the figures you would want to know.
For that matter, if you really want to have a graph, you could do a graph with all of the different evaluation categories in it. That would require the axes limits to be spread out a little more, as the two graphs in the paper have similar differences between scores, but very different average values), but that’s a more honest presentation of the data, anyway. And it would convey a greater amount of information in the actual paper, rather than burying it in the SOM.
I actually like this graph – It brings back fond memories of that old video game, Pong.
I don’t like presenting data in two dimensions, unless both axes are numerical. If a laundry list is presented as an axis, there is always the hidden message that their order is self-evident.
There are cases when it does make sense, but I judge them one by one.
Somewhere Edward Tufte just shot another kitten.