Do not try to evaluate research results in a hurry

Citation

Liu, S. and Yang, Y. (2019), "Do not try to evaluate research results in a hurry", Grey Systems: Theory and Application, Vol. 9 No. 1, pp. 2-4. https://doi.org/10.1108/GS-02-2019-082

1. Do not try to evaluate research results in a hurry

1.1 Scientific research is a business in nature

On July 27, 2017, Professor Michele Pagano published an article entitled “Don’t run biomedical science as a business” in Nature (Pagano, 2017). We agree with the statement of Professor Michele Pagano that the scientific enterprise has been betrayed by the mismanagement of its financial support. However, it is not possible simply to ask people not to run science as a business.

There has been 100 years of history since scientific research became a profession. An occupation is a means to make a living and is a business in its nature. It is inequitable if we ask scientists not to run their occupation as a business in a world where everybody run his/her occupation as a business.

2. Trying to evaluate research results in a hurry caused the mismanagement

The mismanagement of scientific enterprise and scientists are not caused by running science as a business, but results from people trying to evaluate research results in a hurry. This occurs because most stakeholders believe that researchers can be assessed by their publications of top journals and granted funds from famous funding agencies.

It is suggested that the rank of a journal can be used as a proxy for the quality of its publications and the rank of funding agency can be used as a proxy for the quality of its grants. It is generally accepted that an article published in a top journal will have higher quality and a project granted by famous funding agencies will have true innovation. In most institutions around the world, top journal publications have been considered equal to high-quality publications when they try to evaluate research results just after they were published, and grants from famous funding agencies have been considered equal to high innovation project when they attempt to evaluate a research project just after they were granted.

In China, a researcher has a much better chance to be recognized or selected as a named scholar such as “Outstanding youth scholar,” “Excellent youth scholar” by NSFC or “Distinguished professor of Yangtze River” by Ministry of Education, PR China, etc. if he/she has some international top journal publications and/or has received a major funding grant.

Dr Chunyu Han has obtained many honors and financial support after he published a paper, entitled “DNA-guided genome editing using the Natronobacterium gregoryi Argonaute” in Nature Biotechnology in 2016. However, many other scientists failed to repeat his achievement. Dr Han applied to withdraw the paper on August 3, 2017 (Xinhua, 2017). Some of famous bio-scientists in China suggested to inquire thoroughly into the matter (http://news.sciencenet.cn/htmlnews/2018/9/417149.shtm, 2018).

3. This distorted research assessment is harmful to research

As Professor Michele Pagano has previously identified, the scientific enterprise has been betrayed by the mismanagement of research evaluation. The utilitarian academic ethics encourage utility outputs. Guided by the current research assessment system, most researchers are directing their efforts toward the utility direction following the favored evaluation. As a result, significant resources and energy of researchers are diverted to activities other than academic innovation. Some researchers cannot resist the temptation of money, fame or position in seeking to gain as many publications as possible on so-called top journals and gaining further grants from famous funding agencies. This distorted research assessment is wasting taxpayers’ money and the researchers’ valuable time. It is easy to apply for funds for the authors with many top journal publications. Therefore, top journal or high rank grant seeking has become hugely popular among researchers, because most of the output of original innovation is difficult to obtain grants and cannot be published in so-called top journals.

In fact, seeking top journals may lead to a great loss of priority. For example, Professor Zhuo-Hua Pan and Professor Alexander Dizhoor are reputed to have been the first to invent optogenetics. However, after being rejected by three so-called top journals, somebody else published very similar work before them (Vlasits, 2016).

4. The best solution is to wait

Undoubtedly, most of the research outputs published in top journals with high quality or more likely with high quality. Therefore, the research outputs published in top journals can help to increase their influence. However, a paper published in a top journal should not automatically be counted as a high-quality output. Similarly, project funding helps research, but the funding itself should not be counted as a research output. The granted funding can be viewed as a kind of input for research rather than for outputs. A researcher that has gained significant funds without enough valuable outputs should be viewed as a problem rather than achievement in a research accountability system.

To solve these problems, the best solution is to wait. People can judge a research output more easily after 10, 30, 50 or more years of publication.

The true value of research output will show themselves if we can wait for another 10, 30, 50 years or sometimes even longer. It is a result of their real impact verified by the public through long term of practice. Unlike the verified impact, a paper published in a top journal or project granted by famous funding agencies needs to satisfy only a limited number of reviewers. Therefore, the impact of such a paper or project is at most a hypothesis rather than a fact. Indeed, Gangaram Singh et al. (2007) found that there were substantial classification errors from using journal ranking as a proxy for quality. A high impact factor does not necessarily mean a high-quality journal, and a major grant from famous funding agencies does not necessarily mean a true innovation. The recent article in Watch demonstrates how many low-quality papers had been published by so-called top journals, some of them with a high impact factor (IF=34).

5. The Nobel Prize which awarded according to impact only enjoy lofty prestige in the world

The true value of most research outputs show themselves after many years. Nobel prizes are awarded to research output which has produced the greatest impact. Most Nobel Prize winners have waited for several decades. Charles Kuen Kao received the Nobel Prize in Physics 2009 for his contribution to the development of low-loss optic fiber used in optical fiber communication systems though his creativity thinking published by an ordinary journal in 1966 (Kao and Hockham, 1966). Another Nobel Prize winner in Physiology or Medicine was Professor Youyou Tu; her research output had brought about a huge impact, though her output was never published by a so-called top journal. Professor Tu discovered artemisinin (also known as qinghaosu) and dihydroartemisinin, used to treat malaria, a significant breakthrough in twentieth century tropical medicine saving millions of lives in developing countries in South Asia, Africa, and South America (https://en.wikipedia.org/wiki/Tu_Youyou).

6. Recognized early is harmful to researcher

For the young scientists at the early stage of their careers, people can also evaluate their research outputs by impacts, or by real peer experts, but not simply judged by the quantity of top journal papers or grants from famous funding agencies.

A researcher that has been recognized at an early stage of their career maybe harmful to the life of his/her research. After an analysis of 47 Fields Medal winners and 86 other mathematicians who had not received Fields Medals, G.J. Borjas and K.B. Doran found that of those who received their Fields Medals, both the quantities and qualities of their outputs were lower than their competitors (https://en.wikipedia.org/wiki/Tu_Youyou). In China, many of the “Excellent youth scholar” by NSFC or “Distinguished professor of Yangtze River” by the Ministry of Education, PR China find that there is no time for research, as they become an academic leader such as university president or vice-president or even a government officer soon after they are recognized as a named scholar.

7. Conclusion

Research results should be evaluated after their impacts (academic or non-academic) are fully released, and not immediately after publication. Many of the problems associated with mismanagement in research could be eradicated if people did not try to evaluate research results immediately after publication.

Sifeng Liu and Yingjie Yang