Reflections on the REF

After a formidable amount of work, the 2014 Research Excellence Framework (REF) exercise is complete and published. The results, which capture the UK chemistry research environment, and the quality and impact of that research, will no doubt prompt much discussion, analyses and comparison around the world. But what do the results mean for UK chemistry today and for the future of chemistry in general? 

Before we discuss the results, we must emphasise that the process is conducted with the utmost rigour, and praise the professionalism and dedication of the panel, the assessors and secretariat. And of course, we must thank researchers and their institutions for their time spent preparing submissions.

The assessment process took 10 months, starting early in January 2014, and involved seven meetings of the panel and a huge amount of work between the meetings. At the start, calibration exercises compared and moderated the scoring of different panellists. Every item in the submission was then read by at least two panellists and there was a large measure of agreement and convergence in the scores. Careful consideration was given to ‘borderlines’ between the quality ratings and, where appropriate, items were discussed by the full panel. 

We should also note that the process relies on panellists’ expertise to assess the quality of research; journal impact factors were not considered, nor were author contributions. Citation data were used where appropriate, but ultimately the assessment was based on expertise, not metrics. 

The chart gives the overall profile for chemistry and a comparison with the Research Assessment Exercise 2008.  

The first and most obvious observation is the growth in the top (4*) rating and the reduction in 2* ratings. Is this ‘grade inflation’? Emphatically not: it represents a real increase in the quality of the submissions. Several members of the panel (including ourselves) served on the RAE2008 assessment panel and it was quite clear to us that departments have upped their game in recent years. 

Chemistry is essential for a flourishing economy and a healthy society

Clearly, UK university chemistry departments produce high quality science. Almost a quarter of papers submitted are world leading (4*) while the great majority of the remainder exhibit international excellence (3*). In particular, UK chemistry departments show growing strength in many cutting-edge, emerging fields of chemistry: chemical biology, materials, catalysis, computational chemistry, nanoscience and supramolecular chemistry. But it is also clear that excellence in these themes is underpinned by excellence in core areas, including synthesis and spectroscopy.

Our results also reflect trends in chemistry research globally, as chemistry broadens its influence across borders and between sciences. Many outstanding papers were based on interdisciplinary research, but where chemistry is the core component, and a related development is the increased number of outputs involving large teams from several institutions.

The ‘environment’ component, an assessment of capability to undertake high quality research, was equally encouraging. Almost all the submissions, in their summaries of strategy, staffing, income, infrastructure and in their collaborations and contributions, demonstrated a strong and internationally competitive environment for contemporary research. Several showed real world class environments. Institutions have invested in both staff and infrastructure, and the investment has borne fruit. 

One further encouraging statistic is a 25% growth in PhD graduates over the assessment period. The only warning sign concerned research funding: many departments have not managed to diversify their sources of funding, and some are clearly struggling to retain adequate levels, for which the real terms decline in funding from research councils must be a major factor. If the health of the discipline is to be maintained, this must be addressed.

However, the most revealing component of the exercise is ‘impact’. The quality here is exceptionally high, with exciting case studies that really show how chemistry contributes to the economy and society. As might be expected, we see evidence of the crucial role our discipline plays in the manufacturing industry and in developing new pharmaceuticals. But they also showcase the truly impressive breadth of chemistry’s impact: for example, restoring and preserving key historical objects or developing air pollution policy. We have always known that chemistry is an essential discipline for both a flourishing economy and a healthy society; now we have the evidence.

Of course, there are downsides to the process. The workload – both preparation and evaluation – is a concern. But the only way to reduce this is to move towards proxies, for example bibliometrics, rather than expert review, and we must consider as a community if this is what we want. Also, as scientists we know that measuring a system perturbs it. There is no doubt that assessment exercises have a major influence on, for example, staffing policies. Some of these are unquestionably beneficial, but are they all? Institutions must use the results wisely. Excellence should be recognised and rewarded, but we must remember that assessment is not an end itself.

There are still many other issues and questions, and much debate to be had. But let us conclude on a positive note: the most important message from this exhaustive (and exhausting!) process is that our discipline is in robust health.

Richard Catlow and Graham Hutchings are the chair and deputy chair of  the REF subpanel for chemistry