Hexbyte Glen Cove Banishing bias across disciplines, genders and experience—new tool for fairer research metrics thumbnail

Hexbyte Glen Cove Banishing bias across disciplines, genders and experience—new tool for fairer research metrics

Hexbyte Glen Cove

Left panel: Distribution of within-discipline residuals of the relationship between Arel and loge years publishing by discipline (ARC = archaeology, CHM = chemistry, ECO = ecology, EVO = evolution and development, GEO = geology, MIC = microbiology, OPH = ophthalmology, PAL = palaeontology), each comprising 60 researchers (30 ♀, 30 ♂). Right panel: Distribution of among-discipline residuals. Credit: Professor Corey Bradshaw at Flinders University.

‘Where researchers are in their career, be they exciting newcomers or distinguished professors, can also have a strong effect on current metrics.

‘The Epsilon Index is a new way to reduce systemic biases in assessing researcher quality via citations by providing career-stage, gender, and opportunity corrections to citation-based performance metrics,’ he says.

The tool is freely available as a ready-made app—simply punch in a few data for a sample of researchers from open-source databases like Google Scholar, and it does the heavy lifting to produce the result, enabling comparison of researchers at any stage of their career and from any discipline on the same scale.

It’s a boon for anyone who wishes to use an objective metric to rank researchers, be it for grant applications, job interviews, promotions and awards, or even as a staff performance indicator.

Relationship between scaled citation mass and loge years publishing ( for 480 researchers in eight different disciplines (ARC = archaeology, CHM = chemistry, ECO = ecology, EVO = evolution and development, GEO = geology, MIC = microbiology, OPH = ophthalmology, PAL = palaeontology) comprising 60 researchers each. Credit: Professor Corey Bradshaw at Flinders University.

The approach to develop and test the ε-index was itself an exercise in multi-disciplinarity and specifically engineered to ensure gender balance.

The tool was tested and refined through collaboration on sample data with an archaeologist—Assistant Professor Dr. Stefani Crabtree (Utah State University/Santa Fe Institute), a geologist and vertebrate palaeontologist—Professor Kate Trinajstic (Curtin University), a chemist—Professor Justin Chalker, a microbiologist—Professor Bart Eijkelkamp, a palaeontologist—Professor John Long, an ophthalmologist—Professor Justine Smith, and an evolutionary biologist—Professor Vera Weisbecker (all Flinders University).

The sample comprised 480 researchers with Google Scholar profiles, stratified evenly into eight disciplines (archaeology, chemistry, ecology, evolution and development, geology, microbiology, ophthalmology, palaeontology), three career stages (early-, mid-, late-career), and two genders.

Professor Justine Smith, who’s been a ‘superstar of STEM’ actively promoting women’s engagement in science, technology, engineering and mathematics, hopes the ε-index will make a difference to how women in science are perceived, and in turn encourage more women to enter the sciences.

‘More accurately assessing the contribution of women to the sciences and celebrating their successes is pivotal to encouraging future generations of girls into the sciences; as the saying goes, “you can’t be what you can’t see”. The ε-index gives fairer and greater visibility to women’s achievements, and that it does so across a range of disciplines makes it especially beneficial,’ Professor Smith says.

Violin plots of scaled residuals and m-quotient across all eight disciplines relative to career stage (ECR = early career; MCR = mid-career; LCR = late career). Credit: Professor Corey Bradshaw

How it works

The new index is a ranking algorithm that can be standardised across disciplines, can be corrected for career breaks, and provides a sample-specific threshold that can determine whether individual performance is greater or less than expected relative to the other researchers in a sample.

Using either the R code or online app, it requires just four items of information from public databases such as Google Scholar or Scopus to calculate a researcher’s ε-index:

  • the number of citations acquired for the researcher’s top-cited paper
  • the i10-index (number of articles with at least 10 citations)
  • the h-index, and
  • the year in which the researcher’s first peer-reviewed paper was published.

The tool also provides a simple method to scale the index across disciplines with variable citation trends (ε′-index) to enable fairer comparison of researchers in different areas.

  • Median ranks among the eight disciplines examined (ARC = archaeology, CHM = chemistry, ECO = ecology, EVO = evolution and development, GEO = geology, MIC = microbiology, OPH = ophthalmology, PAL = palaeontology). Credit: Professor Corey Bradshaw
  • Ranks by gender and career stage (ECR = early career researcher, MCR = mid-career researcher, LCR = late-career researcher); (b) bootstrapped debiased (i.e., calculating the scaled residuals for each gender separately, and then ranking the combined dataset) ε′ ranks by gender and career stage. Credit: Professor Corey Bradshaw

The ε-index easily allows benchmarking of subsets of researchers into women-only or men-only to adjust the threshold such that the ranks are more comparable between these two genders. Alternatively, dividing the genders and benchmarking them separately followed by a combined re-ranking effectively removes the gender bias in the ε, which is difficult or impossible to do with other ranking metrics.

‘No ranking metric is perfect, but the ε-index is a big leap forward in overcoming bias and giving a clearer, fairer picture of research performance across the board,’ Professor Bradshaw says.



More information:
Corey J. A. Bradshaw et al, A fairer way to compare researchers at any career stage and in any discipline using open-access citation data, PLOS ONE (2021). DOI: 10.1371/journal.pone.0257141

Citation:
Banishing bias across disciplines, genders and experience—new tool for fairer research metrics (2021, September 10)
retrieved 11 September 2021
from https://phys.org/news/2021-09-banishing-bias-disciplines-genders-experiencenew.html

This document

Read More Hexbyte Glen Cove Educational Blog Repost With Backlinks —