« Cooperation gets complex | Main | Almost a no-brainer »

Defining "transformative research" for NSF

According to a National Science Foundation survey, 30% of grant panel members say that they often recommended "transformative research" projects for funding, but only 10% of other panel members (who also, presumably, answered the same question) did. This seems like a mathematical impossibility, like men having had more girlfriends, on average, than women have had boyfriends. But, just as the definition of "girlfriend" (or "sex", for that matter) may vary, so may the definition of "transformative research."

Can we do better than "the projects I like are transformative; those you like aren't"?

I think most scientists woud agree that transformative research is that which overturns existing ideas or opens up a new area. Predicting, in advance, which projects will do that may be difficult, but can we at least come up with an objective measure of "transformativeness", after the work has been published long enough to see its impact?

A variation on citation analysis might work. The total number of times a paper is cited (in effect, linked to by other researchers in their publications) is a widely-used measure of impact. OK, but many of the papers cited confirm earlier results, perhaps more thoroughly, rather than challenging old ideas or opening up new areas. I suggest that a paper should get one "transformation point" every time it is the oldest paper in a cluster of citations. So

Mitochondria are descended from bacteria (Margulis 1975; Smith 1985; Jones 1995).
would earn one transformation point for Margulis but none for Smith or Jones. On the other hand
Margulis (1975) proposed that mitochondria were descended from bacteria, which has subsequently been confirmed empirically (Smith 1985, Jones 1995)
would earn transformation points for both Margulis and Smith.

Maybe citations during the first year or two after publication should be excluded. Many early citations could be either unfairly negative (from those whose ideas are being challenged) or uncritical acceptance of an interesting idea prior to confirmation. If a paper is still being cited years later, however, and it's the earliest of the citations on some question, doesn't that meet an objective definition of "transformative?"

If NSF rewarded reviewers who correctly predict which projects will be most transformative, perhaps with small travel grants, that would probably lead to the right balance between openness to new ideas and skepticism of unsupported assertions.

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

Type the characters you see in the picture above.