I didn’t pay that much attention to the mini-controversy over the NSF’s proposed revision of its grant evaluation criteria when they were first released, because I was working on the book. I was asked to say something about it yesterday, though, and having gone to the trouble, I might as well say something on the blog, too.
The main source of complaint is the “Broader Impacts” section of the grant, a category that has always been sort of nebulous, but which the new standards attempt to clarify:
Collectively, NSF projects should help to advance a broad set of important national goals, including:
- Increased economic competitiveness of the United States.
- Development of a globally competitive STEM workforce.
- Increased participation of women, persons with disabilities, and underrepresented minorities in STEM.
- Increased partnerships between academia and industry.
- Improved pre-K-12 STEM education and teacher development.
- Improved undergraduate STEM education.
- Increased public scientific literacy and public engagement with science and technology.
- Increased national security.
- Enhanced infrastructure for research and education, including facilities, instrumentation, networks and partnerships.
This list has generated a lot of complaints, including a letter to Science and an editorial in Nature that are reasonably summarized in non-paywalled form by this Physics Today story. The biggest issues are, quoting from the letter in Science:
First, the list focuses on economics and national security, but excludes protecting the environment and addressing other social problems. Aside from the consequences of neglecting these areas, this new focus may undermine the attractiveness of STEM disciplines to more idealistic students who are interested in meeting human needs rather than fostering economic competitiveness. Second, under the proposed new criteria, applicants and reviewers are restricted to the provided list of national needs, which will complicate efforts to respond to new challenges as they develop.
(the Nature op-ed has a similar set of concerns, but isn’t quoted as conveniently in a place I can get it for free).
My reaction to the whole thing is pretty much: “Meh.”
First of all, the quoted list of categories is a proposed change that was posted for comments. It might very well be revised before it gets to the final rule stage (the comment period is now closed, but was open when those pieces ran).
Even if the new rules went into effect unchanged, though, I don’t think there’s really anything to worry about, because, well, look at that list. It’s hard to imagine a worthy proposal that doesn’t meet at least one of these. If funding your grant wouldn’t result in “Enhanced infrastructure for research and education, including facilities, instrumentation, networks and partnerships,” you’re not asking for the right stuff. And if you’re working on energy or environmental issues and can’t make a solid argument that your work will increase national security or economic competitiveness, then you clearly have no idea what you’re doing. There’s even the “including” loophole, allowing you to make a case for the importance of something else that isn’t on the list.
Most importantly, I suspect that the practical impact of this is basically nil. I haven’t been on the final review panels, but every year I get sent 4-5 NSF grants for preliminary review, and based on the quality of the “Broader Impact” statements I keep seeing, I can’t imagine that this category is actually deciding who gets funded and who doesn’t. I could be wrong– I haven’t looked to see if any of the proposals with remarkably half-assed “broader impact” sections got denied funding, but if you were going to cut people off for doing a lousy job on this criterion, NSF would have a hard time finding people to fund.
And none of those proposals would have any trouble checking off boxes on the proposed list of “important national goals.” The only thing this will change is the formatting of the proposals– instead of wittering on vaguely about how training grad students supports science education, they’ll directly quote one of the relevant items from the list, and get it over with faster. It’s not going to “produce more hype and hypocrisy,” as Daniel Sarewitz claims in Nature, it’s just going to slightly reconfigure the hype and hypocrisy we already have.
The actual question I was asked about this wasn’t about the impact of the new rules, though, but was more about the “broader impacts” criterion itself. And regarding that, my opinion is basically that it probably doesn’t help all that much, but can hardly hurt.
The point of the criterion, as far as I can tell, is to force people seeking research funding to devote a minimal amount of thought and effort to public dissemination of their results. Which produces a lot of empty blather, and a large number of very traditional outreach efforts, but may also provide some people with the justification they need to do some good and innovative outreach programs.
It would undoubtedly be more effective to find those people and directly give them money to do innovative outreach programs, but academia isn’t set up to do that right now. There aren’t that many channels to get funding directly for outreach activities, and even if it brought in money, there’s a sort of stigma attached to it that would deter a lot of people anyway.
The NSF rule is about as much as you can do in a “top-down” sort of way– you can require that people who are getting research funding also do a small amount of public outreach, which at least gets you something. It’s not going to be optimal, but it’s almost certain to be better than nothing (it’s possible, to put on a public science program that actually makes matters worse, but it’s not easy), and if you didn’t have the requirement at all, “nothing” is what you would get.
What we need is a shift in the attitudes of scientists in general, to recognize public outreach and communication of science as a valuable thing in and of itself. At which point you could probably get adequate resources to those people who have the talent and inclination to do a really good job of bringing science to the general public, and improve the standing of science in general. That can be done– we are science, after all– but it involves changing a lot of minds, and a good deal of work.
“What we need is a shift in the attitudes of scientists in general, to recognize public outreach and communication of science as a valuable thing in and of itself.”
What we need is a shift in the attitudes of the government, to recognize that scientists are not to do two jobs for
one pay. In my opinion, “public outreach and communication of science as a valuable thing” should be done separately,
not for the same pay and not by the same people.
In my opinion, “public outreach and communication of science as a valuable thing” should be done separately,
not for the same pay and not by the same people
That would be fine, but it needs two additional things:
1) scientists to recognize that it is valuable, and not piss and moan about some funding being diverted to pay for public communication, and
2) scientists to recognize that there’s an art to doing public communication, and not piss and moan endlessly about non-scientists “dumbing down” their work and treating every simplification as a personal affront.
I’d be thrilled to see some recognition of the need for dedicated public communication of scientific results, along with the money to pay people to do it right, and some respect accorded to those people. We’re a long, long way from that, though.
It would undoubtedly be more effective to find those people and directly give them money to do innovative outreach programs, but academia isn’t set up to do that right now.
People get outreach grants and do stuff in existing institutions. You might say that the academic system isn’t really set up to properly evaluate and reward this sort of work, but I submit that grants are grants, overhead is overhead, and if a person brings in overhead $ for the Dean then they will still get tenure, irrespective of the type of grant.
Of course, I’m a cynic.
I have to disagree with your “meh”. As a matter of fact, NSF panels are looking more carefully, not less, at the BI parts of proposals, and yes, people do fail to get funded because of that criterion.
Moreover, it sounds like you think that the question is to argue that your research will contribute to these goals. That’s not the point at all — you have to include (and are expected to budget for) actual activities that fall in this category. You are right that the vast majority of funded research contributes to these goals in the long run, but NSF is looking for something very different from that.
I think that the most important thing missing from the list is a nice, big, bold faced “but not limited to” clause right after the “… national goals, including:” piece. That would encourage people to get creative about how they might do something toward a broader set of goals.
As listed, the goals leave out: (1) STEM education at the graduate or postdoctoral level, (2) partnerships between academia and non-profit and public service organizations (as written, it just says “industry”), (3) contributions to public health, environmental quality, or public safety (as written, it just says economic competitiveness).
I really hope that NSF does a better job of these criteria than this. I’m glad they are revising it, but it needs to be better.
Broader impacts play a deciding role in the funding of NSF grants decided at panel reviews. Of course, these are usually larger ones that can afford to support outreach specialists.
The major problem with the letter reviews is that NSF does not properly train reviewers (or if you prefer guidance) as to what to look for outside of the panel reviews.