Meaningless metrics limit researchers

I’m a sucker for gamification. Give me points or badges to reward my efforts and I’ll obsessively rearrange my life to get more – for a couple of weeks at least.

But numbers without context can be misleading. The Duolingo app recently congratulated me on spending 223 days in a row learning Japanese. Sounds impressive: time spent learning a language should be a rough proxy of linguistic ability. But after all that time the only sentence I can reliably translate is ‘I eat vegetables’. The problem is that only my activity within the app is rewarded; the hard work that I’d need to do outside the app to consolidate my learning is not.

There are parallels here with how we reward and assess academic researchers. Publication metrics – which are much easier to measure than it is to objectively assess the quality of a researcher’s work – have traditionally been used as a proxy for research ability. The number of papers a researcher produces supposedly indicates productivity; and citation counts apparently represent the quality of that work.

But, just like with my language-learning streak, high numbers do not necessarily indicate high quality.