I’ve been grading lab reports in two different classes, and I’ve been struck once again by the way that students attach mystical properties to anything with a digital readout. The uncertainty used in calculations is invariably put down as half of whatever the least significant digit displayed was, even in cases where the readout visibly fluctuated during the measurement.
Even better, any experiment involving length measurements will inevitably produce lab reports suggesting that the uncertainty could’ve been improved if they had had a caliper with a digital readout, rather than a vernier caliper.
I think that the next time I do my usual introduction to uncertainty analysis, I’m going to borrow the digital-readout caliper from the shop, and have everybody in the class measure the length of some rough block of metal. Maybe seeing the variation in the results will drive home that digital metering only creates the illusion of absolute precision…
I’ve had the same issues when marking my student’s lab reports.
The problem seems to lie in their understanding of uncertainties. In class, they’re taught the rules of manipulating uncertainties, propagating uncertainties through complex equations. This makes them hold on tight to hard rules about uncertainties, and half a place in the last significant figure is one of them.
The problem is that estimating uncertainties is more of an art than a science, unless you’re willing to do full statistical analyses on hundreds of samples. I try to get my students to take a more relaxed attitude towards uncertainties – if one student estimates a 3mm error and their partner estimates 5mm, they will still both get roughly the same answer, and both will be reasonable.
I’ve also been trying to emphasize the importance of relative uncertainties. In a diffraction lab students measure the location of dots on a screen. They can measure the locations of the dots to an accuracy of a couple mm, but the distance from the slits to the screen with an accuracy of a couple cm. They usually panic about the second measurement, since their uncertainty is an order of magnitude larger. But a couple cm uncertainty in a 2 meter distance is tiny compared to a couple mm in a 2 cm measurement.
If we could get students to stop worrying about getting the “correct” answer, I think uncertainties would be a lot easier to teach. People are naturally pretty good guessers, and most of uncertainty estimation is just educated guessing.
Of course, bad estimates of uncertainties are the least of their problems. I can’t count how many lab reports I’ve received in which all the measured values have perfectly reasonable errors, but the calculated value is reported to 10 significant figures.
Ha. You should see how some people think about the results of computer simulations. When we do ground test measurements and compare them to simulations, we are usually elated when the simulations are within 10 percent of the measurement. Then when someone takes the models and does predictions, the results have somehow become gospel.
One more uncertainty pet-peeve:
Students who take good measurements, make a calculation error and arrive at a value that is 100x larger than the expected value, then claim in the conclusion that this is could be improved with more accurate measurements.
I’ve taken to starting every lab with the reminder that the experiment actually does work. If their calculated value is within 10 or 20% of the expected value, it’s probably ok. If they’re orders of magnitude out, it’s probably their fault, and they need to redo their calculations.
Loosely related, in my software engineering $job I keep running into people who program two sequential events, A then B, and assume the elapsed time after A but before B is somehow bounded; i.e., either will be at least so much, or at most so time, or both. That could be Ok if the assumed bounds had a rationale (i.e., a reason); or to put it differently, if they can prove the bound exists with the stated value (and, typically, no error bars!).
But trying to get the point across can be maddeningly difficult.
What I tend to do now is assert the algorithm must pass what I call the “17-42 Rule”: Prove to me the algorithm works if the elapsed time is less than 17ns (nanoseconds), and prove to me the algorithm also works if the elapsed time is more than 42 years. That seems to usually get the point across, probably (I’m guessing now) because the two values, 17ns and 42yr, and clearlyâ arbitrary (and in essentially all cases, for the work I do, absurd).
â Actually, not quite: 17 is for yellow pigs, and 42 is the answer.
Another source of spurious accuracy is in scale conversion; convert from ‘somewhere close to 37 degrees C’ to ‘98.6 degrees F’ and then worry if the child’s temp hits 99 :).
You should turn it into a bet; when someone writes down 4 significant figures of their calipers measurement, put $2 down on the table and bet the student that an independent measurement will get a different answer.
Then erase the appropriate number of significant figures and offer the bet again. Withdraw the bet, or decrease the payout, if they overestimate the errors.
Either they’ll get the point, or you’ll rake in a small pile of cash (for the sake of good will, you could spend it on a box of donuts before the next lecture.)
The question for Aler (#1) is “How many points do you take off?” One for each extra digit? Exponentially increasing with the silliness of the exaggeration? Don’t complain if you simply reinforce bad behavior by giving a “warning” without any grade penalty. My penalty increases each week of the semester. Ditto for absurd calculations like in #3, but with a reduced penalty if the student tells me s/he knows it is wrong but can’t find the mistake and was too damn lazy to do the calculation during lab so they could ask for help with it.
Body temp is a classic. The original data were something like (37 +/- 0.5) deg C, which should be converted to (99 +/- 1) deg F. A decade or so ago someone did a population study and found normal is not 98.6 deg F to 3 sig figs.
A friend of mine does something like #6 proposes during a ballistic pendulum lab. After calculating the initial v and its standard deviation, he has them predict where it will land when fired off the table … with an error band. OK, now how much of your grade (or whatever) will you bet that 3 of 5 shots will land on the band? Heh heh.
Related: You haven’t lived until you’ve listened to someone twenty years your senior tell you why measuring power output with a spectrum analyzer is better than with a microwave power meter.
It seems like this reliance on or belief in the accuracy of digital over analogue has been going on for about 20 years or so. I suppose that at least one generation of students has now gone on into their fields and now constitute what is commonly called “the overwhelming consensus” regarding some very critical concepts we are now addressing in society via policy and ideology.
I’m reminded that in a number of old SciFi novels of the “hard science” school, the engineering and science world had super-human computers, of course, but the genuinely proficient practicioners in the Heinlein-esque worlds still understood the value of the slide-rule and carried them, sometime surreptitiously, everywhere, being fully aware of how quickly the numbers could get away from reality. I know it’s old fashioned but I’d feel somehow safer if I knew major technological desicions were still being back checked by some old Gomer with a slip stick and a volt meter.
Ewan: Another source of spurious accuracy is in scale conversion
I have seen lots of this in science articles written for the general American public. Frequently, they will give a distance in miles to several significant figures, which turns out to be a round number in kilometers.
MarkP: You should see how some people think about the results of computer simulations.
This is a chronic problem in my field. The people who write global simulation codes understand their limitations; in some cases they explicitly state that a certain phenomenon only appears (or disappears) in their model when they crank up the resolution. Rumor has it that one global simulator changed sides in a major scientific debate in the field after upping the resolution in his model.
A related problem is trust in spreadsheets. Many people assume that whatever the spreadsheet spits out must be the right answer. That happens only if both of the following are true: (1) the input data are accurate, and (2) the formulas calculate the quantity they are intended to calculate. For example, I do my own proposal budgets, and our internal forms break out subtotals in a different way from how the agencies break them out. We have internal checks to be sure that the salary data are accurate, but the checkers tend to assume that everything else is done right. For one proposal I did a couple of years ago, the budget spreadsheet had a bug which went undetected locally; I noticed it only when I went to enter the numbers on line and the totals came up different.
Actual lab report: “In this measurement there is no error bar to be given. It was a digital readout.”
Luckily, it weren’t my students. The students were also no physicists, thank God, only doing their mandatory physics lab course. But my colleague had a nice talk with them, afterwards 🙂
Comes from advertising copywriters pushing the “computer-designed” meme.
Maybe it’s because I’m an oldster, though I put it down to discovering Isaac Asimov’s book “Mass, Length, and Time” during my high school years, but I think there is also an issue with explicating the difference between Accuracy and Precision. Digital measuring tools offer great precision – the 1/2 LSD being its measure – but make (almost*) no warranties as to the accuracy of the system in which they are embedded. It sounds like these students are confusing the precision of their meters with the accuracy of their measurements. Error bars have to do with accuracy, not precision (to a first order).
* accuracy: yes, the instruments make warranty about the accuracy of measuring the value at their terminals, or the pulse count, not that one can discern it without RTFM, but it is up to the experimenter to properly connect it, and to properly arrange for the representation (V, R, pulses, ..) to be measured to be properly associated with the phenomenon under study. Perhaps a small unit in instrumentation in lab courses? Already there? oh. well.
Gaffer,
Do you mean the book the internets keep saying is written by Norman Feather? Cause I couldn’t find one by Asimov, though I agree that would rock — that gent could write.
Gaffer may mean “Realm of Measure: from the yardstick to the Theory of Relativity” by Asimov.
they need to know difference between precision and accuracy!
Like the old man who ran tour caves in New Mexico and told people that the caves were one million and thirty seven years old. When asked how he knew that to such an accuracy, he said “well thirty seven years ago I got here and they said the caves were a million years old”…..
But you’re gonna have to fess up to them and tell them why digital signals are better (TV, etc….). The current rush for phones and such to use digital rather than analog makes digital sound godlike. Which for signal processing it can be if done right.
If you show students contact measurements lie they will start whining for optical comparators then interferometers. Lesson fom NASA and the Hubble primary mirror: Precision without accuracy can be made accurate… unless you fire the old fart who went ballistic when the data were remarkably precise and entirely impossible.
Tolerences in circuit boards average. Computers exist. Tolerences in magnetohydrodynamics multiply. Controlled thermonuclear fusion does not exist, desktop or star.
Another example of the distinction between precision and accuracy:
Archbishop Ussher claimed that the world was created on the 26th of October, 4004 BC, at 9 AM Garden of Eden local time. This is an uncertainty of about an hour (we don’t know exactly where in the Middle East the Garden of Eden supposedly was) in ~6000 years, or roughly 20 parts per billion. That’s very precise, but not accurate: astronomical and geochemical evidence suggests something more like 4.5 to 4.8 billion years. Also, the Jewish calendar, which makes the same assumptions Abp. Ussher made, claims that this is the 5768th year since creation, not the 6011th (remember, no year zero), so Ussher may not even have been accurate under his own assumptions.
Half of the LSD? Only if you are on LSD or don’t RTFM!
One way to get their attention is to change scales on the multimeter while keeping everything else the same. Once they see that the numbers are not being rounded (in any way shape or form), some other results (checking currents at a node) make a lot more sense.
I need to further develop some exercises they can do to discover that digital meters also have limitations. Explaining how they work is a bit beyond the scope of my course, but it is definitely a case of fake precision.
It’s also fun to show them the guard digits being used inside their calculator, and that their calculator can be smarter than they are. None of them know why ln(-1) says something about the answer not being real rather than just being an error. [I think it was the TI-85 that would automatically switch to complex mode if an inverse function was valid for the analytic continuation but not valid on the real numbers. It would give an imaginary answer for the ln of a negative number and even for the inverse sine of 1.1.]
This is important for scientific work and laboratories but it has an even greater importance to the wider society.
Too many people go through life thinking in absolute terms. As a tradesman I once watched a young tile setter getting very frustrated trying to lay out level lines around a bathroom. I was on lunch and the bathrooms in the half-built school building were one of the few places that were out of the sun and away from the usual construction noises.
Anyway the guy starts in the least seen corner, smart, he then proceeds to use his level and a pencil to draw his level line. Must have been better than 50′ of wall. Every time he went round the room his lines would fail to line up. An eighth or quarter of an inch would have been good enough. But when he would complete the circuit he was off by well over an inch. So he starts over. Still off. So he checks his level, which is pretty much as good as they get.
I’m in there eating my PB&J and apple and half paying attention. He is getting frustrated. he can’t get it right and doesn’t know why. Before I leave to get back to work I let him in on a little secret, all levels are inaccurate. He protests that his is a high-dollar, guaranteed, certified to standard unit that he treats with nothing but loving care. It is a fine tool. But not perfect.
I suggest that if he draws his line and flips the level end-for-end every time he will get better results. Doesn’t make sense to him but he tries it. His line ends almost match. Flipping the level each time the inherent error of the tool is automatically compensated for.
In society we have developed the idea that expensive tools are inherently more accurate. Sometimes they are. We also think that if the tool tells us a number it is accurate.
This gets into the whole right-wrong, truth-lie thing. which isn’t how much of life, or science (what little of it I understand of it) or reality works.
Life is less about absolutes than relative values and measurements. It isn’t about which version of a story is (capital ‘T’) True than which version is more, or less, true. And exactly how true, accurate, the result has to be to get useful results. When I studies electronics one of the concepts people had trouble with was understanding that when dealing with transistors it is less about which line is positive or negative than which is more or less negative, or positive. Elementary to you I’m sure but lessons that can be applied more widely.
In fact I see this issue playing out in the ID vs evolution debate. The IDers make great hay about their source being ‘absolutely’ correct, word of God don’t you know, but so much of science is estimates. And how every few years a ‘new version’, their words, comes out. They overlook the fact that the ‘new version’ is not so much entirely new as a refinement of the older study. That although the numbers change the accuracy and reliability of those presentations is,in general, always improving. That new science doesn’t supersede the old version but is built upon and advances what was known previously. That while we may never get an absolute, final and perfect, answer we keep getting closer.
That while we want to reach perfection and finality on so many questions we are forced to deal with a considerable and unavoidable level of doubt and uncertainty. Doubt and uncertainty that this culture is uncomfortable with. Doubt and uncertainty that is anathema to most western religions. For they claim to ‘know’. And knowing, independent of evidence, is pretty much what faith is.
Blind faith in the absolute accuracy of his tools, not knowing the limits of them and how to compensate for their weaknesses, is why my tile setter friend was having so much trouble.
I guess something fixed Isaac in my mind as the author early on. I do not have Feather’s name in my mental image of the cover at all. But we are talking over 40 years ago after all. But try as I might I could not find a picture of the cover to verify we are in fact talking about the right book.
I bought it as a Penguin edition in ~1963. Blue-Green cover, pretty thick, cover illustration included a large gear wheel. Of course, none of my book collection made to USA with me. Oh well.
Available from multiple sources as ‘old/rare’ though for only ~$10. Penguin Co. apparently do not keep records of old books on-line. That Feather chappie was almost as prolific as Isaac.
While on the subject of measurements, this is, I swear, a true story that happened to me: I went into Builders Emporium to get some 2″ x 1″ lengths for a project. The young lad in the wood section was very apologetic when he told me “sorry, don’t have any of that”. He thought for a moment or two, then said – perfectly straight face – “but we do have some 1″ x 2″ – would that do instead?”
This is a tangent, comparing guesstimation with error measurement analysis. There are a few quizes out on the interwebs, such as http://roughly.beasts.org/ that ask you to a question with a numerical answer – such as `how many plastic bags were used by Australians last year’ – and asks you to enter an estimate plus an error bar on your estimate, such that you are >80% confident your answer lies in this range. We tried it at a lab lunch one day and from 30 questions with 10 participants less than 50% of the real answers lay in our guesstimate ranges.