You are here

Research Assessment and Impact in the Digital Age

Cameron Neylon of the Public Library of Science visits Harvard.


March 21, 2013—At a recent lecture discussing research assessment and impact, Cameron Neylon, advocacy director for the Public Library of Science (PLOS), focused on how online tools can revolutionize scientific practice and open access to its results, broadening participation in the scientific process across diverse audiences. But from the outset, Neylon stressed that he had no answers, only questions.

“As scholars, as an institution, as a community, why do you value research, and what do you value it for?” he asked his audience. “The unfortunate truth is that in many cases, it turns out that most of the things we care about are very difficult to measure. So we measure a small set of things, and then by default, the things that we measure become the things that we actually care about—what is the journal in which we publish? What is the publisher we choose to send our monograph to? That becomes more important than who reads it, and what they do with it, and how it affects the outside world.”

Neylon, whose talk was sponsored by Harvard Library’s Office for Scholarly Communication, observed that “we live in a space where we can measure things better,” because so much scholarship and research, including critiques and citations of that research, now appears online. As a result, there are larger traces of the use of research available to determine the impact of that research. He asserted that the purpose of scholarship is to be used—in education, for cultural enrichment, for policy development and so on—but also to be re-used and re-usable.

“When we talk about impact, it all involves some form of re-use,” he said, adding that with online social networking systems such as Facebook and Twitter, it’s no longer really about where scholarship is published, but more about reaching the right people and what they do with that research.

Neylon also said that narrative, and discourse, were vital to understanding scholarship and the dissemination of that research in the modern age. Web-scale networks “don’t have a top and a bottom,” he said. “Rankings tend to not be very useful in this space. There is no ‘better’ or ‘best’—only whether something is more useful in a specific case.

“If we understand what we’re trying to achieve—when we can articulate that to each other and to ourselves—then we can measure that better,” Neylon said. “We can make subjective decisions about what data we’re going to bring to bear to help us to understand how well we’re doing. If we get that right, that assessment will enhance the impact of the scholarship we’re doing.”

Neylon added that instrumental use of ranking and numbers are actually of little help in a networked world. “The diversity of impact, or the portfolio of what you’re creating or doing, is actually more important than any given ranking,” he said, emphasizing that scholars should reframe their view from traditional publishing venues to maximize their exposure and potential discovery from “the unexpected—those exciting re-uses you haven’t thought of, that come from the random collisions of people.

“At the end of the day, we can’t build a system on yesterday’s truth,” he concluded. “That’s not how the world operates anymore. In the end, we have to look five or ten years out to where the world is going to be, not where it is today. At the end of the day, the innovators, the ones who really make a difference, aren’t the ones who are following markets—they’re the ones who are building them.”