{"id":9652,"date":"2014-10-23T10:34:37","date_gmt":"2014-10-23T14:34:37","guid":{"rendered":"http:\/\/scienceblogs.com\/principles\/?p=9652"},"modified":"2014-10-23T10:34:37","modified_gmt":"2014-10-23T14:34:37","slug":"on-not-talking-for-the-right-reasons","status":"publish","type":"post","link":"http:\/\/chadorzel.com\/principles\/2014\/10\/23\/on-not-talking-for-the-right-reasons\/","title":{"rendered":"On Not Talking, for the Right Reasons"},"content":{"rendered":"

Over at Backreaction, Bee has a nice piece on our current age of virality<\/a>. Toward the end, she discusses some of the ways this applies to science, specifically a quote from this Nature article about collaborative efforts to measure “big G”<\/a>, and a story about a Chinese initiative to encourage collaboration<\/a>. She writes of the latter, “Essentially, it seems, they\u2019re giving out salary increases for scientists to think the same as their colleagues.”<\/p>\n

And I agree that this can be a problem– there’s a famous paper I can never find looking at the evolution of the accepted value of some physical constant over time, showing that it changed by many standard deviations from the initial measurement, but never all that much in a single step. This strongly suggests that experimenters measuring the value had an idea of the “right” answer, and may have (probably unconsciously) biased their own results in that direction. This sort of thing is why the practice of “blinded” data analysis has become more widespread in recent years (though the steady increase in computing power over that same period has no doubt helped, by making this much easier to arrange). Too much collaboration between groups can have a similar effect– if there are only one or two measurements out there, it’s hard to have confidence that there isn’t some subtle systematic bias that’s confusing things.<\/p>\n

At the same time, though, there are major advantages to pooling resources for difficult questions. One of the big stories in AMO physics the last few years has been the dramatic improvement in the search for an electric dipole moment of the electron driven by the ACME collaboration between Yale and Harvard<\/a>. They haven’t found an EDM, but they’ve found that there’s nothing to find<\/a> with precision an order of magnitude better than the previous experiments, within the span of about five years. This came about because the principals of the collaboration– Dave DeMille at Yale, and John Doyle and Jerry Gabrielse at Harvard– realized that they were all independently working on a really hard problem, and could do a lot better if they combined their efforts.<\/p>\n

Now, would it be a good thing for science if they also brought in Ed Hinds and Norval Fortson and Eric Cornell? Probably not– there might be some additional efficiency gains, but you would lose out on the sanity check of having some competing measurements out there using different systems. But I think that the net effect of their collaboration to this point has been very positive.<\/p>\n

So, there’s a balance to be struck: it’s important to have enough collaboration to smooth the path to difficult measurements, but enough competition that you can trust the results. You don’t want everybody to be talking, but at the same time, you want people to be not-talking for the right reasons.<\/p>\n

And I think the big-G collaboration described by Nature is trying to do this the right way:<\/p>\n

\nA big item on the agenda at the meeting will be debating how to choose a small number of these experiments to be conducted by members of a consortium, this time with an unprecedented level of oversight. Each experiment will be repeated by two independent groups, using identical sets of equipment created and tested at a third institution. While the experiments are going on \u2014 and there is still time to fix them \u2014 experts from outside those two groups will hunt for errors. In the past, says [Terry] Quinn [former director of the International Bureau of Weights and Measures (BIPM) in Paris], who is a driving force behind the NIST meeting, scientists have picked holes in each other\u2019s experiments only after they were published, making it difficult to verify whether those problems were really the cause of an error.\n<\/p><\/blockquote>\n

I think this sounds like an appropriate balance. These big-G measurements are extraordinarily difficult to do, and because each experiment is its own self-contained thing, they rely very much on local tacit knowledge of how their systems work, which can allow subtle errors to creep in. More consistent techniques, and integrated data analysis and error checking is a good step toward fixing the disagreement in measurements, which is a bit of an embarrassment for people who care about such things.<\/p>\n

And, of course, some more talking might help avoid the occasional disaster. After all, the root cause of the recent BICEP2 controversy is that the BICEP2 analysis of their data required them to figure out the contribution of dust to the signal they were seeing, and the information they needed to do that properly was held by the Planck collaboration, their direct competitors. BICEP2 did their analysis with incomplete information– apparently including estimates from values presented at a conference by Planck scientists– and ended up thinking their values were a lot more solid than they apparently are, once all the data from Planck are properly incorporated. Which is kind of embarrassing, and could’ve been headed off by a little more communication. And, in fact, the plan calls for the two experiments to collaborate on the next measurement, for just this sort of reason.<\/p>\n

So, you know, there’s a fine line between talking too much, and not talking enough. Figuring out exactly where that falls is a problem much too hard for mere physicists, and probably can only be done in retrospect, many years down the road.<\/p>\n","protected":false},"excerpt":{"rendered":"

Over at Backreaction, Bee has a nice piece on our current age of virality. Toward the end, she discusses some of the ways this applies to science, specifically a quote from this Nature article about collaborative efforts to measure “big G”, and a story about a Chinese initiative to encourage collaboration. She writes of the… Continue reading On Not Talking, for the Right Reasons<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8,146,5,19,265,7,565,11],"tags":[],"class_list":["post-9652","post","type-post","status-publish","format-standard","hentry","category-academia","category-atoms_and_molecules","category-blogs","category-experiment","category-in_the_media","category-physics","category-precision_measurement","category-science","entry"],"_links":{"self":[{"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/posts\/9652","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/comments?post=9652"}],"version-history":[{"count":0,"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/posts\/9652\/revisions"}],"wp:attachment":[{"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/media?parent=9652"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/categories?post=9652"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/tags?post=9652"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}