Universities involved in a pilot exercise to establish how to best assess research impact have found it a difficult and time-consuming process, but academics appear to be slowly accepting the controversial measure.

At a meeting at King’s College London, UK, on 25 June a number of institutions that took part in the Higher Education Funding Council for England’s (Hefce) pilot scheme described their experiences, with many characterising it as a challenging task that needs further development before being rolled out as part of a new research appraisal approach in a few years’ time.

The pilot was launched following an outcry from the research community that under proposals for the Research Excellence Framework (REF), the new system for assessing research quality in UK universities, research ’impact’ could count for 25 per cent of the overall assessment.

The pilot scheme involved 29 institutions focusing on designated subject areas and submitting case studies that illustrate examples of impact along with an impact statement describing the positive effects achieved by the research and its wider relevance. 

’The impact statement was really very difficult to prepare,’ said Mary Fridlington, senior assistant registrar at the University of Oxford, UK, where the clinical medicine department and the earth systems and environmental science department took part in the pilot. ’We had a very difficult time trying to pull the information out of the researchers - it’s just such a very different way for them to think about things.’ 

This was also the experience at the University of York, UK, whose physicists appeared to struggle with the approach. ’It felt like they were very close to the research and very keen on describing the actual work,’ explained Rachel Curwen, research policy officer at the university. ’There was a language barrier about selling their research in a very particular way.’ 

Anna Grey, research policy manager at York who led the physics pilot at the university, says this is a skill scientists will need to develop as implementation of the REF draws nearer. ’Writing about impact is a qualitative narrative and scientists are just not used to writing in this way,’ she says. ’It’s something the physics department identified as a skill they haven’t got yet - I suspect it will come as they get used to it because the sector needs to start describing impact better.’ 

The definition of ’impact’ was also a particular obstacle. Hefce had left the definition relatively broad so as not to exclude potential impacts, but this proved difficult when it came to compiling impact statements and identifying research impact. 

’Academics had real difficulty using Hefce’s definition of impact,’ said Fridlington. ’No matter how many times we explained it they were still thinking it was all about money.’ David Sweeney, director of research, innovation and skills at Hefce, acknowledges that ’the "I" word’ has been unhelpful during development of the REF. ’Impact means different things to different parts of the community,’ he said at Friday’s meeting. ’We are trying to tease out how better to express these things in ways that some are more comfortable with.’ According to Hefce, while the meaning of impact may be better defined in future iterations, it is likely to remain a broad concept so that all impacts can be included. 

Embracing impact 

Despite the hostile reaction to the introduction of impact assessment, feedback from the pilot exercise seems to suggest that researchers are coming round to this new method of research appraisal. 

’We’ve noticed a real change in the attitude to the impact agenda over the last year,’ says Curwen. ’I think there was that initial anger about it, then there was panic, then there was acceptance this was something they were going to have to do. What we’ve seen over the last six months is genuine excitement.’  

Grey says the anxiety expressed by the research community was largely down to the concern that this was yet more work on their plates. ’Anyone experiencing change finds it difficult at the start. Once they get over that hurdle and appreciate what it is they have to do they start engaging with it more, and they find it actually isn’t that bad,’ she says. ’It was a really good opportunity for them to explain to non specialists outside their very specific field what their research was about and what impact it had.’ The one concern Grey does have, however, is the amount of effort that may have to go into submissions for the REF once it is implemented: ’It takes an awful lot of time to track the impact and to identify it, and that’s because blue skies research can take a long time to have an impact.’ 

The UK’s new science minister in the coalition government, Conservative MP David Willetts, has voiced his scepticism of the impact agenda within the REF, and the Conservative party has previously been in favour of delaying implementation of the REF. Hefce is due to meet with Willetts to discuss his concerns before the final conclusions of the impact pilot exercise are presented in the autumn, though a meeting between the two parties has yet to be arranged. 

Anna Lewcock