In a recent opinion paper, [1] Julian Kirchherr complains about that “up to 50% of the articles that are now being published in many interdisciplinary sustainability and transitions journals may be categorized as “scholarly bullshit”. These articles, Julian states, “typically engage with the latest sustainability and transitions buzzword (e.g., circular economy), while contributing little to none to the scholarly body of knowledge on the topic.” Despite their poor quality, these articles “tend to accumulate significant citations and are thus welcomed by many journal editors”. System-wide incentives to focus on publication metrics drive “more and more authors into publishing on the very latest buzzword, e.g., ‘circular economy’, which creates a perpetuum mobile respectively vicious circle (depending on your perspective) regarding publications on such topics. […] All contributors (journal editors, authors) know they may be producing scholarly bullshit; however, publishing such works is advantageous for everyone involved in this contemporary academic system.”
Julian’s piece is a refreshing reading indeed, as it candidly expresses a frequent impression shared by many sustainability and transition researchers: That the content of some publications does not justify the impact they generate. The general fear is that a highly self-referential publication bubble emerges, with little actual knowledge and ultimately, little real-world decision support and impact. And that this state would be, despite the success on the surface, deeply unsatisfactory for the colleagues involved and fundamentally wrong by the core ethical standards of science.
We don’t have metrics for evaluating the actual knowledge generation, quality, relevance, and impact of research. Probably because these aspects are immeasurable. Instead, we use indicators, like citation counts or altmetric scores (based on internet activity around a publication) as proxies for impact and quality. History books, managers, and behavioral economists know that metric-based evaluation can create perverse incentives. Some funny and many tragic examples are listed on the Wikipedia. [2]
In other words, Julian’s impression is that about half of what is published in sustainability and transitions research only serves the proxy success indicators and has de-coupled from the actual, the true mission of sustainability science. The feedback he got on Twitter shows that many colleagues from the field tend to agree with him.
This problem deserves further investigation and discussion. In particular, it would be interesting to find out how the sustainability and transition research community ‘performs’ in comparison to other fields. Like psychology, for example, where there is also a wide debate on the value of much of the research published in the last decades due to wrongly applied statistical methods and strong publication bias towards interesting and ‘significant’ findings. (Check for the so-called replication crisis. [3,4] Interestingly, the share of research there that is not reproducible is also 50% and more… Maybe a 50% bullshit rate is the new norm …)
While I tend to agree with Julian, the sad part is that he hardly touches upon the solutions to reducing ‘bullshit’ that are already in place. So, let’s get to the core message here: No matter how big this problem is, we know how to fix it! While scientists, as all humans, respond to behavioral-economic incentives, we are constrained in our actions by the ethical principles of our fields. A core ethical principle is the adherence to the state of the art of our respective research field.
Science is a self-governed enterprise. This means that the standards for good scientific work are not set and enforced by external auditors, but by the respective scientific communities and their members, the scientists, themselves. This is to ensure scientific freedom and independence and because the community members are the experts and no one else. From this freedom a great responsibility arises. Each field has the responsibility to organize itself to define what constitutes sound research in this particular field and what doesn’t.
The standards of sound research are continuously enforced by the members of the community: During teaching, when advising doctoral students, during journal editing and peer review, during admission of submitted abstracts to conferences, when selecting which papers to cite, and in funding and promotion decisions.
The standards for conducting state of the art research are never static. They evolve over time to take into account new ethical standards, technical possibilities, and inter-disciplinary research approaches. For example, conducting double-blind psychological and medical experiments has only become the gold standard in the middle of the 20th century. [5]
The International Society for Industrial Ecology (ISIE), [6] for example, has worked for decades with its different organizational bodies to establish and raise the standards of sound systems research to explore how material and energy are used by society and to find solutions to complex environmental problems, including the challenge of establishing a more circular economy (CE). Examples of this work include:*
- Contribution of many industrial ecology (IE) community members to the ISO 14040 and 14044 standards for life cycle assessment and to the system of environmental-economic accounting (SEEA) via the guidelines for economy-wide material flow accounting.
- Development of data transparency and accessibility guidelines for publications in the Journal of Industrial Ecology [7] led by the journal’s founding editor Reid Lifset and the then-president of the ISIE Edgar Hertwich
- The work by the ISIE sections, e.g., the section on socio-economic metabolism to establish guidelines for transparent and reproducible research [7a]
- Development of teaching material with best-practice examples of how to tackle recurrent research problems [8]
In addition, I have tried to cast my own modelling and general research experience into recommendations, focusing on cumulative research for IE/CE modelling software [9] and sustainability research in general. [10]
Given that Julian’s piece targets circular economy research, in particular, it becomes clear that the circular economy research community members need to raise their standards of what constitutes sound research and enforce them in their different activities. In particular, the newly established circular economy research organization [11] needs to expand its activities to include the formulation of best research practice.
We all have a moral obligation to further develop the state of the art of research and to live up to the standards set by the community. Accepting and fulfilling this obligation is the High Road to sustainability research of high quality and relevance.
While we cannot measure the actual knowledge generation, quality, relevance, and impact of research, we do have a broad scientific community of sustainability and transition research that values sound science! And even if we can’t identify them by measurement, the scientific community will recognize and reward the authors of sound contributions to science, while it will tend to ignore and ultimately forget about those contributions that don’t meet its standards.
*) Please write a message to in4mation@indecol.uni-freiburg.de if you have other examples that should be listed here!
References (access date: June 2nd, 2022)
[1] https://doi.org/10.1007/s43615-022-00175-9
[2] https://en.wikipedia.org/wiki/Perverse_incentive
[3] https://www.nature.com/articles/nature.2015.18248
[4] https://en.wikipedia.org/wiki/Replication_crisis
[5] https://en.wikipedia.org/wiki/Blinded_experiment#History
[7] https://jie.yale.edu/data-openness-badges
[7a] https://doi.org/10.6094/UNIFR/217970
[8] http://www.teaching.industrialecology.uni-freiburg.de/
[9] https://onlinelibrary.wiley.com/doi/10.1111/jiec.12316
[10] https://doi.org/10.1038/s41893-019-0443-7 (also available via sci-hub.ren via DOI search)
