{"id":4773,"date":"2010-06-17T11:56:48","date_gmt":"2010-06-17T11:56:48","guid":{"rendered":"http:\/\/scienceblogs.com\/principles\/2010\/06\/17\/the-problem-of-broader-impacts\/"},"modified":"2010-06-17T11:56:48","modified_gmt":"2010-06-17T11:56:48","slug":"the-problem-of-broader-impacts","status":"publish","type":"post","link":"http:\/\/chadorzel.com\/principles\/2010\/06\/17\/the-problem-of-broader-impacts\/","title":{"rendered":"The Problem of Broader Impacts"},"content":{"rendered":"

Over at the Cocktail Party, Diandra Leslie-Pelecky has a post about the image of scientists<\/a> that spins off this Nature<\/cite> article on the NSF’s “broader impact” requirement<\/a> (which I think is freely readable, but it’s hard to tell with Nature). Leslie-Pelecky’s post is well worth reading, and provides a good deal more detail on the anecdote reported in the article.<\/p>\n

While Leslie-Pelecky’s concern is about whether the outreach programs falling under the “broader impact” section of grants are having the desired effect, I’d like to comment on a different aspect of the article, namely the whole implementation of the “broader impact” thing. As Corie Lok writes in the article,<\/p>\n

\n

Many NSF-funded researchers find the foundation’s definition of broader impacts to be, perhaps unsurprisingly, broad, and frustratingly vague. Among the examples of activities listed in the foundation’s proposal guide are: developing educational materials for elementary, high-school and undergraduate students; involving these students in the research where appropriate; creating mentoring programmes; maintaining and operating shared research infrastructure; presenting research results to non-scientific audiences such as policy-makers; establishing international, industrial or government collaborations; developing exhibits in partnership with museums; forming start-up companies; and giving presentations to the public.<\/p>\n

Because it lacks conceptual clarity, the broader-impacts requirement often leaves researchers unsure about what to include in their proposals, and leads to inconsistencies in how reviewers evaluate applications. “Broader impacts were designed to be open, but openness confuses a lot of people,” says Luis Echegoyen, the division director for NSF chemistry.<\/p>\n<\/blockquote>\n

That’s definitely a problem. The official NSF criteria<\/a> are incredibly broad, and just in case they weren’t broad enough, they’re prefaced with the disclaimer:<\/p>\n

\n

the following list is neither prescriptive nor exhaustive and should not be read in ways that constrain the creativity of researchers in proposing activities with broader impact<\/p>\n<\/blockquote>\n

That certainly clears things up. Given that, it’s hardly a surprise that:<\/p>\n

<\/p>\n

\n

Researchers often end up repackaging what they’re already doing. “Overwhelmingly,” says Echegoyen, “the number one broader impact that most people in the chemistry division are using is ‘training graduate students and postdocs.'”<\/p>\n<\/blockquote>\n

After all, if the criteria are broad enough to encompass anything, they probably encompass whatever you’re already doing, so why not just use that?<\/p>\n

I have written one NSF grant, which received funding in 2004, and for the last several years I’ve been asked to read and report on several proposals. My experience fits with Echegoyen’s. While NSF has supposedly “cracked down” on the Broader Impacts criterion, most of the “Broader Impact” sections I’ve seen are pretty half-assed. People talk about how they train students, or mention that they have female or minority post-docs (“Some of my best friends…”), or just talk about how their research will lead to future technologies that will benefit society. The overall impression is of a box being checked off on a list. While there are proposals that talk about innovative and effective outreach programs, the majority of the PI’s whose proposals I have read seem to be getting the requirement out of the way with the minimum possible effort.<\/p>\n

I’m as guilty as anyone else. Looking back at my own NSF proposal, I’m mildly surprised to find that I didn’t even have a separate “Broader Impacts” section in the main proposal, instead shuffling it off to the required RUI (“Research at Undergraduate Institutions”) statement. Where I pretty much talked about how the money would be used to train future graduate students.<\/p>\n

I don’t think this is necessarily a major problem, though. While I would like to see more done to engage the general public’s interest in science (enough so that I’m now part of the committee overseeing the APS’s outreach program), I’d rather have that happen by funding people who really want to do outreach programs than by making people who have no interest in outreach do some perfunctory outreach activities as a condition of research funding.<\/p>\n

This is similar to a point I try to make in my talk about blogging as a tool for communicating with the public. I don’t think every scientist needs to have a blog in order to communicate results to the public– many scientists would be really bad as bloggers, and would resent having to make the effort. But those who do<\/em> have an interest in blogging and public communication should be encouraged to do so. I think the benefit to science from a smallish number of bloggers doing it really well is about the same as the benefit of lots and lots of bloggers doing it badly.<\/p>\n

To circle back to the point made at the Cocktail Party, in many cases, we don’t really know whether the outreach programs people do actually have the intended effect. In many cases, programs that seem well designed to achieve some goal may not reach it, and may even create more confusion. I think this problem is magnified when the original efforts are kind of perfunctory, as many of the proposals I see seem to be.<\/p>\n

The NSF “Broader Impact” requirement is a nice idea, but as it’s currently implemented, I’m not sure it’s working all that well. They need some better way to measure the effectiveness of the programs people do put on, and they need to think about whether this is really the best way to accomplish the goal.<\/p>\n","protected":false},"excerpt":{"rendered":"

Over at the Cocktail Party, Diandra Leslie-Pelecky has a post about the image of scientists that spins off this Nature article on the NSF’s “broader impact” requirement (which I think is freely readable, but it’s hard to tell with Nature). Leslie-Pelecky’s post is well worth reading, and provides a good deal more detail on the… Continue reading The Problem of Broader Impacts<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"1","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8,13,339,11,75],"tags":[335,123,336,337,338,98,340,88],"class_list":["post-4773","post","type-post","status-publish","format-standard","hentry","category-academia","category-education","category-outreach","category-science","category-society","tag-broader-impact","tag-education-2","tag-funding-2","tag-grants","tag-nsf","tag-outreach-2","tag-public-communication","tag-science-2","entry"],"_links":{"self":[{"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/posts\/4773","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/comments?post=4773"}],"version-history":[{"count":0,"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/posts\/4773\/revisions"}],"wp:attachment":[{"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/media?parent=4773"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/categories?post=4773"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/chadorzel.com\/principles\/wp-json\/wp\/v2\/tags?post=4773"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}