Over at the Cocktail Party, Diandra Leslie-Pelecky has a post about the image of scientists that spins off this Nature article on the NSF’s “broader impact” requirement (which I think is freely readable, but it’s hard to tell with Nature). Leslie-Pelecky’s post is well worth reading, and provides a good deal more detail on the anecdote reported in the article.
While Leslie-Pelecky’s concern is about whether the outreach programs falling under the “broader impact” section of grants are having the desired effect, I’d like to comment on a different aspect of the article, namely the whole implementation of the “broader impact” thing. As Corie Lok writes in the article,
Many NSF-funded researchers find the foundation’s definition of broader impacts to be, perhaps unsurprisingly, broad, and frustratingly vague. Among the examples of activities listed in the foundation’s proposal guide are: developing educational materials for elementary, high-school and undergraduate students; involving these students in the research where appropriate; creating mentoring programmes; maintaining and operating shared research infrastructure; presenting research results to non-scientific audiences such as policy-makers; establishing international, industrial or government collaborations; developing exhibits in partnership with museums; forming start-up companies; and giving presentations to the public.
Because it lacks conceptual clarity, the broader-impacts requirement often leaves researchers unsure about what to include in their proposals, and leads to inconsistencies in how reviewers evaluate applications. “Broader impacts were designed to be open, but openness confuses a lot of people,” says Luis Echegoyen, the division director for NSF chemistry.
That’s definitely a problem. The official NSF criteria are incredibly broad, and just in case they weren’t broad enough, they’re prefaced with the disclaimer:
the following list is neither prescriptive nor exhaustive and should not be read in ways that constrain the creativity of researchers in proposing activities with broader impact
That certainly clears things up. Given that, it’s hardly a surprise that:
Researchers often end up repackaging what they’re already doing. “Overwhelmingly,” says Echegoyen, “the number one broader impact that most people in the chemistry division are using is ‘training graduate students and postdocs.'”
After all, if the criteria are broad enough to encompass anything, they probably encompass whatever you’re already doing, so why not just use that?
I have written one NSF grant, which received funding in 2004, and for the last several years I’ve been asked to read and report on several proposals. My experience fits with Echegoyen’s. While NSF has supposedly “cracked down” on the Broader Impacts criterion, most of the “Broader Impact” sections I’ve seen are pretty half-assed. People talk about how they train students, or mention that they have female or minority post-docs (“Some of my best friends…”), or just talk about how their research will lead to future technologies that will benefit society. The overall impression is of a box being checked off on a list. While there are proposals that talk about innovative and effective outreach programs, the majority of the PI’s whose proposals I have read seem to be getting the requirement out of the way with the minimum possible effort.
I’m as guilty as anyone else. Looking back at my own NSF proposal, I’m mildly surprised to find that I didn’t even have a separate “Broader Impacts” section in the main proposal, instead shuffling it off to the required RUI (“Research at Undergraduate Institutions”) statement. Where I pretty much talked about how the money would be used to train future graduate students.
I don’t think this is necessarily a major problem, though. While I would like to see more done to engage the general public’s interest in science (enough so that I’m now part of the committee overseeing the APS’s outreach program), I’d rather have that happen by funding people who really want to do outreach programs than by making people who have no interest in outreach do some perfunctory outreach activities as a condition of research funding.
This is similar to a point I try to make in my talk about blogging as a tool for communicating with the public. I don’t think every scientist needs to have a blog in order to communicate results to the public– many scientists would be really bad as bloggers, and would resent having to make the effort. But those who do have an interest in blogging and public communication should be encouraged to do so. I think the benefit to science from a smallish number of bloggers doing it really well is about the same as the benefit of lots and lots of bloggers doing it badly.
To circle back to the point made at the Cocktail Party, in many cases, we don’t really know whether the outreach programs people do actually have the intended effect. In many cases, programs that seem well designed to achieve some goal may not reach it, and may even create more confusion. I think this problem is magnified when the original efforts are kind of perfunctory, as many of the proposals I see seem to be.
The NSF “Broader Impact” requirement is a nice idea, but as it’s currently implemented, I’m not sure it’s working all that well. They need some better way to measure the effectiveness of the programs people do put on, and they need to think about whether this is really the best way to accomplish the goal.
I think the benefit to science from a smallish number of bloggers doing it really well is about the same as the benefit of lots and lots of bloggers doing it badly.
Really? I would think it would be much higher.
I agree that the requirement is very poorly defined. In my first NSF I proposed to work with two high school teachers to develop a microbiology curriculum. We would go to their classrooms and work with students and during the summers the teachers and some of their students would work in my lab to learn experimental techniques and further develop the curriculum. I also had the usual stuff about training students and in addition had a plan to enhance my course through web-based approaches through a collaboration with our graduate school of education (I had letters of collaboration for all of these ideas). You know what the reviews said? “The broader impacts are not sufficiently beyond what would normally be expected from an Assistant Professor.” I haven’t applied to NSF since.
On a panel I recently served, micromagnets plan would have been highly rated. But, fundamentally, the broader impact could only hurt you. Some programs were downgraded because of it, but only one program got higher marks for their broader impact.
At least the panel was pretty good about saying that mentoring students, postdocs or teaching classes is not broader impact, that is just doing your job.
Seems to me that a much better system is to drop general requirement for outreach in research grants: no such requirement exists in Canada, where a research proposal is evaluated based on the research proposed, period. I doubt there is much difference in amount and quality of outreach done in NSF funded research because you have to check that box. Better way to support outreach is allocating certain portion of the overall budget specifically for separate grants devoted entirely to outreach.
@3. Maybe I’ll try again.
That may be a better system Moshe. But I have to say in writing the grant and working on a plan with the high school teachers I got very excited about the possibility of the collaboration. I doubt I would have put that much effort into it if it wasn’t part of a larger grant. I think more specific guidelines would be helpful.
I don’t think this is necessarily a major problem, though. While I would like to see more done to engage the general public’s interest in science (enough so that I’m now part of the committee overseeing the APS’s outreach program), I’d rather have that happen by funding people who really want to do outreach programs than by making people who have no interest in outreach do some perfunctory outreach activities as a condition of research funding.
I agree with this completely. Your analysis is spot-on. To many of the “broader impact” statements are half-assed. It’s an important part of the NSF mission to do the broader impacts, but it’s typical administrative stupidity to say that every grant must participate in that mission. Some people will be better at it; outreach and educational activities should be judged on their own merits, not tied to research activities. Likewise, research activities may be great, but we shouldn’t hold them back because the PI has no interest in writing a real broader impacts statement. And, of course, because we know who reads these proposals, generally (even if the NSF is “cracking down”) a weak broader impacts can slide if the panel isn’t looking for reasons to shore up a dismissal of a proposal.
I have similar rants about the tying together of research and teaching all in the same one-size-fits all package for professors at research-focused institutions….
On the surface, it seems unobjectionable that publicly-funded scientists should be doing something to promote scientific knowledge among the public. But, as everyone else says, the devil is in the details. It’s all well and good to be a good citizen and do the occasional public talk or visit a school now and then or participate in an outreach program designed by somebody who is an expert on this.
I have enough respect for outreach to realize that it is not easy, and I don’t see why the logic of division of labor and specialization shouldn’t apply to outreach when it applies to so many other things in science. We don’t expect organic chemists to also be oceanographers and particle theorists.
Yeah, yeah, we wouldn’t want outreach to turn into a little world of its own that is completely detached from the rest of the scientific community, but there’s a lot of ground between that scenario and a scenario where every publicly-funded scientist is expected to maintain an individually-designed outreach program. It would be a more efficient use of resources to fund outreach programs as outreach programs, and strongly encourage/incentivize NSF-funded scientists to contribute to these efforts.
Also, I’ve never gotten a consistent answer on exactly what they want. One program officer said to me quite explicitly that the ethnic composition of my research group was a sufficient broader impact. (That grant nonetheless did not get funded, for reasons related to the science.) Others have been more vague. For some programs focused on providing opportunities for students, colleagues have been told that if they are funded they must only involve students from under-represented ethnic backgrounds. For other programs focused on providing opportunities for students, colleagues have been told that it would be illegal under federal law to restrict participation to students from certain ethnic backgrounds. I’ve been told by some people that you need to reinvent your own wheel, and I’ve been told by others that that’s overboard.
You touched on a lot of topics that I avoided to try to keep on a single point in my blog. Thanks for the post, as I now don’t have to write that one, as you covered everything pretty darn well!
I should have mentioned in my Cocktail Party Physics blog that I manage a website called the Broader Impacts Toolbox at http://www.broaderimpacts.info that addresses some of the same questions. The best work that’s been done clarifying what NSF wants is primarily from the Chemistry directorate, but I agree wholeheartedly with the observations that often, all the BI in your proposal can do is hurt you. You won’t get funded because it is brilliant, but it is one more target for reviewers to hit.
The broader impacts toolbox website has some reports that were done about whether BI was even being taken into account – the same things mentioned in the comments here are contained in that report, which is at least five years old. Within NSF, different program managers, even within the same program, interpret it in very different ways. Some divisions let you include budget, some expect you to find funding from your institution or (my favorite), just do it in your voluminous spare time.
This seems like a problem that really should have been solved by now, doesn’t it.