A history prof from Catholic University named Jerry Z. Muller is flogging a new book titled The Tyranny of Metrics, most recently via an interview at Inside Higher Ed (there’s also a version at the Chronicle of Higher Education, but it’s paywalled, so screw them). This is being hailed in many parts of my social-media universe as being a righteous takedown of the mania for “assessment” in all things.

And, you know, I have some sympathy for this position. A lot of the push for greater “assessment” of everything in academia is pretty useless, just resulting in a lot of silly knees-bent running about. All too often the people calling for more “assessment” of things don’t have any coherent idea of something to do with the quantitative data they generate, they just want to be seen to be generating quantitative data because that’s a Thing that accrediting bodies have decided is important for colleges and universities to be seen to be doing. As someone from a quantitative science, that kind of pointless number-accumulation drives me nuts (and I’ve said so frequently enough that colleagues sigh heavily when the subject comes up in a meeting I’m at, even before I raise my hand to speak…).

That said, I’m not quite ready to sign on with Muller and his anti-metrics campaign, largely because of comments like this (from the IHE interview):

The key components of metric fixation are the belief that it is possible and desirable to replace judgment, acquired by personal experience and talent, with numerical indicators of comparative performance based upon standardized data (metrics)

At a very superficial level, that sounds great, and is very flattering to the faculty ego. We have spent years acquiring subject-matter expertise, after all, and are surely qualified to judge students and others.

On another level, though, this creeps me out. The problem is that an exclusive reliance on “judgment acquired by personal experience and talent” unchecked by reference to standardized data is a wonderful way for decision-making processes to become hopelessly corrupted by individual biases. Many of the most pernicious features of academia have gotten there through the operation of biased personal judgement, stretching back decades, and a lot of efforts to improve academic culture are at their core efforts to unwind those years of biased judgments.

The inclusion of “talent” in that especially gives me pause, because it’s uncomfortably close to the idea that some people are special and succeed for that reason. This is one of the most harmful ideas in all of academia, the source of endless social-media angst, and I’m not at all comfortable with the idea of enshrining it as one of the criteria for who gets to exercise judgment.

This is not to say that any and all quantitative metrics are automatically superior to the judgment of a professional– many of the easy numerical measures we have are, as Muller correctly notes, basically garbage. At the same time, though, we know that they’re garbage because they’re quantitative. People objecting to the use of student course evaluations as a metric for faculty performance rely heavily on the argument that they’re biased against faculty from underrepresented groups. We know because that bias is something that can be quantified, and has in numerous studies.

As crazy as it is to claim that quantitative metrics are inherently objective because they’re quantitative, it’s even crazier to say that relying on individual judgment is the fix for that. And yet, that happens an awful lot when this subject comes up. On a few occasions, I’ve heard this pointed out, and the counter-argument basically amounts to “It’s OK because unlike the quantitative systems we object to, we have the right biases…” I don’t find this to be a great testament to the quality of judgment being employed.

So, on the topic of metrics and assessment and how they’re debated in academia, I find myself deeply conflicted. I’m happy to agree with the claim that many of the “assessment” exercises we currently do are useless, and many of the metrics used in higher education are bad. All too often, though, the anti-metric argument slides directly from “We’re doing a bad job of quantifying this particular thing” to “We should stop trying to quantify anything,” and that’s not something I can sign on to.

There’s a thread on everybody’s favorite low-stakes conflict, scientists vs. science journalists, going around on social media. It’s from a scientist who moved toward journalism (blogging for the Guardian), and lays out a bunch of reasons why it’s unreasonable for scientists to expect to see a news article article before it gets published. These are mostly pretty good, and founded in research into the content of news stories and institutional press releases.

Thread alert: Lots of heated discussion going on about this. It’s not the 1st time – this issue comes up repeatedly. I used to be firmly of the “I’m a scientist. Yes” camp, but I changed my views quite profoundly & here’s why. Gather round fellow scientists. 1/x https://t.co/Lec0DkcHOA

— Chris Chambers (@chrisdc77) February 5, 2018

While the thread as a whole is pretty good, it ends on a sour note for me, with a “get over yourselves” directed at scientists. I find that a puzzling decision, because a slightly different read on the same material would lay the blame for misrepresentation of science on the fact that many journalists are lazy and just lightly re-write press releases. There are plenty of good reasons for more humility on both sides.

The fundamental problem in this conflict, as in so many other stupid petty status fights, is that both sides have an overinflated view of their own work, and an overly cynical view of the other’s. Both sides seem themselves as noble seekers of TRVTH, thwarted by the nefarious actions of the miscreants on the other side.

Scientists in this argument like to portray themselves as ferreting out the secrets of nature, and prize accurate reporting of their work above all else. Any attempt to summarize or simplify their work will inevitably introduce some technical inaccuracies. This is seen as a horrible slight, misrepresenting the purity and elegance of what they do. Thus, the demand to fact-check things before they’re published, and the belief that refusal to allow this indicates an active desire to make them look foolish by inserting errors.

In reality, of course, most of the research carried out by scientists is not remotely important, and only a minuscule fraction of readers will even notice the bits that have been oversimplified. And the value of absolute accuracy tends to be very selective– scientists who fume about tiny errors in stories about their own work are often perfectly fine with gross oversimplifications of whole other fields of science.

But the journalists are not without fault, here. They have a tendency to see themselves as similarly noble in the pursuit of the public interest, ferreting out hidden details and penetrating layers of obfuscation to get to the real, uncomfortable truth. In that view, a scientist or other source asking to check the final story for accuracy is demanding an unconscionable breach of ethics, probably because they have some awful ulterior motive.

In reality, of course, only a tiny fraction of news stories are actually penetrating investigative capital-J Journalism. Most of them are small-r reporting, passing along a minor story that happens to be interesting. The fact that, as shown in the studies described in that Twitter thread, most distortion of science creeps into the news via press materials shows that the field isn’t exactly neck-deep in Edward R. Murrows.

So, yeah, scientists need to get over themselves, and accept that summarization and simplification are a necessary part of communication to a wider audience. Journalists also need to unclench a bit, and recognize that for the vast majority of stories there is not, in fact, any serious ethical concern raised by running a later draft past someone who can check its accuracy.

(There may be logistical issues regarding publication schedules and the like, which is a different conversation. I don’t think that’s an insurmountable obstacle either, but it’s not really the issue here.)

The fundamental problem, here, is a sort of crippling cynicism. Neither side really trusts the motives of the other, and that poisons interactions between them. Each side assumes the other is trying to put something over on them, and the end result is a kind of paralysis.

This sort of thing is extremely widespread in academia, and I find it utterly exhausting. When the starting point for analysis of any proposal or request is “How is this an attempt to screw me?” even trivial matters become enormous stupid fights. This kind of crippling cynicism is pervasive, and it’s responsible for about 2/3rds of the days when I hate my job and fantasize about doing something else for a living (the remaining third comes from grading…).

——

A couple of important caveats: First and foremost, Not All Collective Nouns. While there are too many pseudo-journalists out there lazily re-writing press releases, the best people in the field are very much not doing that. I’ve had very positive interactions with most of the journalists I’ve spoken to about stories they were working on, and I don’t recall feeling badly misrepresented by any of the stories that have quoted me. Similarly, there are a lot of scientists out there who are pleasant and reasonable and easy to work with on this sort of thing (I try to be one– I’ve never asked to review a final draft, and I hope my worst sin in interacting with the media has been losing track of the email asking me for a response).

Both science and journalism are, by and large, well-stocked with good people, and if you give them the benefit of the doubt you likely won’t be disappointed. The conversation tends to be dominated by the worst of the worst, both because they’re personally very obnoxious and because they make wonderfully flammable straw men for the other side to bring up.

I should also note that I write about this from the perspective of a physicist, which is arguably the field of science for which the generally low stakes are at their absolute lowest (our chief competitor being pure math). Most of physics, and especially the stuff I care about, has essentially no partisan political content, so there’s no political reason to misrepresent anything– it’s not like Trump voters are hugely invested in the Copenhagen interpretation while Clinton voters are big Many-Worlds proponents.

(Bernie Sanders is a Bohmian, though. No doubt about it.)

The stakes are higher for fields with partisan political relevance– anything touching on evolution or climate change, basically. Things may feel different in those fields, but even there I suspect that there’s a lot less actual incentive to distort than people think.

But then, I’m a sunny optimist at heart…

As part of my “blog more regularly” push, I’m going to start moving some things that I’ve been posting as Twitter threads over here, where they’ll maybe be more pleasant to read for the tiny number of people who bother to read them. Starting with this Ed Yong piece at The Atlantic on a couple of life-science journals making it easier to publish papers whose main results were “scooped” by another group within a few months.

This is very much a Good Thing, but also a bit of a head-scratcher for me, particularly the quote from journal editor Eve Marder saying “We are seeing a trend toward the co-submission of papers from labs that choose not to compete but, rather, to jointly announce new findings.” Thee are both things that have been reasonably common in my corner of physics, going back twenty-plus years.

I mean, two of the papers I published in grad school had technically been “scooped” in this way– our paper on optical control of collisions came out a couple of months after a similar paper from a Japanese group, as was the paper on collisions in optical lattices. In both cases, the Japanese group sent us advance notice of their work, and offered to delay publication slightly so the papers could come out together. We didn’t take them up on that, but in both cases, our paper came out later in the same journal, Physical Review Letters, which was about as high in the prestige hierarchy as it could plausibly get.

It’s not just us, either– when I was doing the research for the ebook on quantum simulation that I wrote for Physics World (which is a free download, you should read it), I saw a bunch of this. Papers announcing key advances in the techniques used for these experiments tend to come in groups: there are 2-3 papers demonstrating single-atom resolution in optical lattices with bosons, then 2-3 papers doing the same thing with fermions, etc. These come from different groups, but they’re all in glamour journals (granted, not exactly the same journal– typically one is in Science and the next in Nature, or vice versa). There doesn’t seem to be any career-destroying penalty to being second in these sort of races in the world of cold-atom physics.

Now, granted, in these groupings, the two papers tend to have slightly different angles on the same thing, but that’s a natural part of doing science– different labs think about the same problem in slightly different ways, so the results from two groups are never perfectly identical. But my experience within my home research community has not been that second-place papers are totally blocked from publication, or even blocked from publication in the top journals. There are plenty of examples of pairs or triplets of papers that report essentially the same physics result within a few months of each other, all in top-tier journals.

So, I find the story from biology slightly surprising. It’s a positive development, but also a bit of a “thank God I don’t work with these people…” kind of thing. I’m making another tally mark in the column headed “Life Scientists Are Assholes,” and then partially erasing it.