Ripping yarns

"Impact" has had a bad press from many working in higher education. To some, the intention to assess the impact of research is a crude infringement of researchers' freedom to pursue truth, beauty and uncertainty, and as such is likely to corrupt and diminish the value of research. To others, it is yet another example of the sinister marketisation of higher education, where the "public" good is being sidelined in the headlong pursuit of "private" benefit.

However, now that the final panel guidance on the research excellence framework has been published, resistance seems to have melted away. An impact army has been mobilised in universities across the UK as people get to grips with the impact framework and begin to identify and draft their case studies ready for the submission deadline in 2013. In the process, do we risk moving seamlessly from a period of spirited resistance to one of slavish compliance with the new assessment regime?

It seems like a good point at which to take stock.

As director of a centre that supports researchers to develop the quality and impact of their engagement with the public, I confess to being well disposed towards the notion of impact and the task of assessing it - as you might expect. I accept that "impact" can be a crude formulation of what is in practice a sophisticated process of interaction between researchers and wider society. It risks reducing research to a "magic bullet" that delivers effects to the wider world. But what I like about it is that it challenges academic researchers to answer the question: "To whom does our research have value, beyond our academic peers? And what professional responsibility have we taken for trying to realise that value?"

I am also encouraged by the process undertaken by the funding bodies to turn the initial policy commitment into a meaningful framework that challenges researchers but also rewards excellence. Someone close to the REF process told me recently that one of the most significant factors in the creation of the final guidance was the quality of dialogue between the academic and non-academic members of the panels. Perhaps as a result, the guidance presents a very rich "impact palette" for researchers to choose from - articulating the different domains in which that research might generate value, while encouraging academics not to be limited by that.

Far from prioritising financial or commercial returns, all the panels invite research units to explore the reach and significance of their work in the public sphere. How has their work enriched public discourse, civil society and cultural life? How has it informed professional practices in the public services and the third sector as well as in business? What influence has it had on the development of policy - including how it might have helped to prevent bad policy or practices being implemented? What contribution has it made to health and well-being, or to the environment? I struggle to imagine how anyone could question the value of asking and attempting to answer such questions, particularly when researchers are in the privileged position of receiving significant amounts of public money to pursue their work, and it is only a relatively small proportion of their output that needs to be submitted to this scrutiny.

But it's all very well to ask difficult, probing questions: these questions deserve equally good responses, the formulation and assessment of which is challenging. Many worry that asking about impact might be a worthy intention, but that in practice it will be impossible to assess the responses fairly or meaningfully. I have a pragmatic response to this. The keys to success will be the clarity and precision of the responses, supported by reliable and valid evidence, and a realistic response from the panels to the challenge of assessing the subtle and complex relationship between research and its potential impact.

I think that the biggest challenge - at which many people seem to stumble - is to find a way to articulate with clarity and simplicity the nature of the underpinning research insights that are at the root of the claimed impact.

Stating the obvious is something that academic training encourages us to avoid, but far too many of the draft case studies that I have seen are vague and imprecise about the specifics of the underpinning research.

If we are to attempt to trace an "impact" then we need to know what the idea, insight or discovery was and define it precisely. Sometimes this might be one very specific discovery, or it might be a cluster of related insights that have been crystallised through the research. It seems to be surprisingly difficult to give these definition. Once you have, there should be both intense pleasure and intellectual challenge in tracing how these have influenced and been influenced by the world outside the university - a combination of detective work and archaeology and, one would hope, far from a box-ticking, bureaucratic exercise.

Conversation and dialogue, involving all the stakeholders in the research, is going to be critical in beginning to finesse these impact stories. Let me give an example. A colleague - an eminent scholar in American literature - was working on an impact case study linked to her work on American popular culture. This work was wide-ranging, including high-end, excellent research in gender and cultural studies; the publication of a popular biography of an American film star; regular appearances on broadcast television and radio; and popular journalism, in which she was asked to comment on contemporary culture.

At first, when she was preparing a draft impact case study and was asked to give "evidence", all she could give were numbers of appearances and the audience size, which, she ruefully said, made it look as if she thought that was what the value of her work was, or should be. Numbers were the only kind of evidence she had to hand, so she shared them, but no one, including her, thought it very valuable or meaningful. However, when invited to explain the meaning of her work and her motivation for undertaking the public engagement activities she had listed, she formulated a much more compelling account of the difference her work had made. What she had actually managed to achieve was to nurture some powerful new insights about femininity and gender identity in the late 20th century and then introduce these ideas into the public sphere. In the process, she had significantly enriched the vocabulary and conceptual framing of media discourse on these topics, as well as feeding back new insights into her own work. She could point to some precise examples of where these radical ideas had crept out of the academy and emerged blinking into the real world, giving context and meaning to the impressive audience reach she could also quote.

I suppose the lesson is that when academics are asked to give reductive accounts of their own work or to shove it into pre-shaped boxes, their work ends up looking reductive and unimaginative. However, when afforded the space to explain the value of what they do, many academics are inspired and engaging about the meaning of their work. Impact becomes all the more meaningful the more meaningful the questions that are asked. This is of particular importance to work in the humanities, because it is not easily quantified.

The final guidance from the assessment panels, published in January, encouragingly seems sensitive to the subtle and complex relationship between research and its impact - and to be open to the potential richness my colleague was able to distil.

Another example, which initially had nothing to do with impact, proves relevant to this discourse. I was recently catching up with an old friend and colleague from The Open University, with whom I worked on a child development course 15 years ago. He was reflecting on what he had been doing since we last worked together. His original research had explored the inter-relations of social and cognitive development in early childhood, and effective means of assessing these, and revealed the critical importance of supporting parents and carers in developing their skills in such assessments. This work had been under way when we'd worked together all those years ago. Since then, he has been working pretty tirelessly to introduce these ideas to those who could benefit from them. He has developed broadcast, web-based and printed support materials to engage with families, and created online experiments for them to feed back their experiences into the research. He has worked closely with policymakers on the development of new guidance and training materials for childcare professionals. Much of our conversation was taken up with anecdotes about the pleasures and pitfalls of working with broadcasters (and with high-profile egos) and the long, hard grind of influencing policy.

But by the end of the conversation it was plain that there was an exceptional impact case study begging to be written: about how the extensive public engagement work he had undertaken had helped his research insights to transform children's lives through influencing policy, professional practice and parenting skills. That there is an opportunity for such stories to be told and for them to be "valued" seems to me to be a very good thing.

My final example - and one to which I would urge anyone to turn if they are cynical about impact - is the Nobel prizewinning economist Daniel Kahneman's Thinking, Fast and Slow (2011). His "impact" is even celebrated in the publisher's blurb: "Daniel Kahneman, recipient of the Nobel Prize in Economic Sciences for his seminal work in psychology challenging the rational model of judgment and decision-making, is one of the world's most important thinkers. His ideas have had a profound impact on many fields - including business, medicine and politics - but until now he has not brought together his many years of research in one book."

Rather extraordinarily, the book contains the two original papers on which his subsequent work has built, as appendices. As such, then, it is perhaps an example of a new publishing genre: an impact epic. What it does compellingly is bring into focus the process by which a researcher can bring critical new insights, vocabulary and understanding into the public sphere - and how their own work can be enriched by that interaction. His introduction playfully identifies people's "gossip" as the sphere he particularly wants to influence: "Every author, I suppose, has in mind a setting in which readers of his or her work could benefit from having read it. Mine is the proverbial office water-cooler, where opinions are shared and gossip is exchanged. I hope to enrich the vocabulary that people use when they talk about the judgments and choices of others, the company's new policies, or a colleague's investment decisions."

Kahneman's research has revealed profound new insights into the faulty and lazy judgements we all fall back on. His goal is to encourage all of us to think more carefully - not just when we gossip but in our professional lives as well. He targets everyday language as his impact battleground - what our REF panels term "public discourse" - and is gifted at giving complex phenomena everyday labels that make them memorable without corrupting or "dumbing down" their meaning. He coins 20 such labels, for instance the "halo effect", which he defines as the way in which someone's charisma and presentation style can seduce us into believing what they say.

He explains: "Ultimately, a richer language is essential to the skill of constructive criticism. Much like medicine, the identification of judgment errors is a diagnostic task, which requires a precise vocabulary. The name of a disease is a hook to which all that is known about the disease is attached, including vulnerabilities, environmental factors, symptoms, prognosis and care. Similarly labels such as 'anchoring effects', 'narrow framing' or 'excessive coherence' bring together in memory everything we need to know about a bias, its causes, its effects and what can be done about it."

Kahneman serves as an example of what I might crudely and simplistically describe as an "impact hero": an academic unashamedly committed to developing groundbreaking research and to simultaneously ensuring those insights are animated by dialogue and interaction with the world outside academia. In his case, he chooses to define his impact narrative using the metaphor of "conversation" and to pinpoint the effects in terms of how his insights have penetrated personal and professional language.

It is an inspirational book, and I would urge anyone who is sinking under the bureaucratic weight of REF administration to take time to read it, and to enjoy a brilliant account of how research animates and is animated by engagement with the wider world.