Diagnosis Unreliability of ChatGPT for Journal Evaluation

Mehdi Dadkhah; Marilyn H Oermann; Mihály Hegedüs; Raghu Raman; Lóránt Dénes Dávid

doi:10.34172/apb.2024.020

Advanced pharmaceutical bulletin. 14(1):1-4. doi: 10.34172/apb.2024.020

Editorial

Diagnosis Unreliability of ChatGPT for Journal Evaluation

Mehdi Dadkhah ^{Conceptualization,}^{Data curation,}^{Formal analysis,}^{Investigation,}^Methodology,^{Project administration,}^Software,^Validation,^{Visualization,}^{Writing – original draft,}^1,^2,^*
Marilyn H Oermann ^{Conceptualization,}^{Investigation,}^Methodology,^Supervision,^Validation,^{Writing – original draft,}^{Writing – review & editing,}³
Mihály Hegedüs ^{Conceptualization,}^{Investigation,}^Methodology,^Validation,^{Writing – original draft,}⁴
Raghu Raman ^{Conceptualization,}^{Investigation,}^Methodology,^{Project administration,}^Validation,^{Writing – review & editing,}⁵
Lóránt Dénes Dávid ^{Conceptualization,}^{Investigation,}^Methodology,^{Project administration,}^Validation,^{Writing – review & editing,}^6,⁷

Author information:

¹Amrita School of Engineering, Amrita Vishwa Vidyapeetham, Amritapuri, Kerala, India.

²Technology Forecasting Department, SnowaTec Technology Center and Innovation Factory, Entekhab Industrial Group, Isfahan, Iran.

³School of Nursing, Duke University, Durham, North Carolina, USA.

⁴Tomori Pál College, Chamber of Hungarian Auditors, Budapest, Hungary.

⁵Amrita School of Business, Amrita Vishwa Vidyapeetham, Amritapuri, Kerala, India.

⁶Faculty of Economics and Business, John von Neumann University, Kecskemét, Hungary.

⁷Institute of Rural Development and Sustainable Economy, Hungarian University of Agriculture and Life Sciences, Gödöllő, Hungary.

*Corresponding Author: Mehdi Dadkhah, Email: d_mehdi@av.amrita.edu

Abstract

Purpose:

Academic and other researchers have limited tools with which to address the current proliferation of predatory and hijacked journals. These journals can have negative effects on science, research funding, and the dissemination of information. As most predatory and hijacked journals are not error free, this study used ChatGPT, an artificial intelligence (AI) technology tool, to conduct an evaluation of journal quality.

Methods:

Predatory and hijacked journals were analyzed for reliability using ChatGPT, and the reliability of result have been discussed.

Results:

It shows that ChatGPT is an unreliable tool for journal quality evaluation for both hijacked and predatory journals.

Conclusion:

To show how to address this gap, an early trial version of Journal Checker Chatbot has been developed and is discussed as an alternative chatbot that can assist researchers in detecting hijacked journals.

Keywords: Hijacked journals, Predatory journals, ChatGPT, Artificial intelligence, Language models, Research ethics

Copyright and License Information

©2024 The Authors.
This is an Open Access article distributed under the terms of the Creative Commons Attribution (CC BY), which permits unrestricted use, distribution, and reproduction in any medium, as long as the original authors and source are cited. No permission is required from the authors or the publishers.

Introduction

Artificial intelligence (AI) technology refers to the development of data systems that allow machines to perform tasks generally associated with human intelligence such as reasoning and decision making.¹ Its use is becoming ubiquitous, and along with the benefits it offers, there are risks. AI tools present a paradigm shift in academic writing in that they can be used by authors to improve the quality of their papers and by editors to speed up the paper inspection process (e.g., detect plagiarism)² but also by dishonest researchers for purposes of academic misconduct. A popular AI-based tool is a chatbot entitled ChatGPT (GPT stands for generative pre-trained transformer). ChatGPT is a language model developed by OpenAI which can answer questions based on an immense set of data collected from the internet. Users can chat with ChatGPT and receive answers to questions about various fields of knowledge^3-5; however, research shows that ChatGPT may provide invalid output and unreliable data.

Additionally, ChatGPT can be used by paper mill companies to generate fake research papers.^6-10 As ChatGPT becomes increasingly popular, more researchers may ask it to recommend journals or to evaluate the quality and credibility of particular journals to which they intend to send queries. It is imperative to understand whether ChatGPT can provide trustworthy answers to such requests.

Literature review

Historically, two types of questionable journals have plagued academic publishing: predatory and hijacked journals. Predatory journals publish papers without fair review and do not follow the essential publishing process to provide high quality literature.^11-15 There exists a rich body of knowledge as well as criteria and online lists to help researchers identify potential predatory journals and publishers.^16-19 Hijacked journals are fake websites that list reputable indexed journals but have no relation to authentic journals; the actual journals are mimicked in order to charge authors fees and publish papers without peer review.^20-22 The extent of journal hijacking is not limited to a particular domain, and indexed hijacked journals have appeared in the Clarivate (previously known as Thomson Reuters) and other citation databases.^23,24 Both predatory and hijacked journals can damage scientific integrity, propagate errors, decrease countries’ scientific ranking, waste university budgets, and hurt academic reputations.^25-29

Our literature search for ChatGPT use for journal quality evaluation did not provide rich information. Most of the available studies are focused mainly on the detection of ChatGPT-generated papers and the possibility that such papers could be published in predatory journals. There is only one research study about ChatGPT use for journal quality evaluation and concluded that ChatGPT provides accurate answer⁵ about the current status of art.³⁰ However, as most of the currently available methods and tools for detecting predatory and hijacked journals are complex or have critical weaknesses.^17,22,31 many researchers may tend to use ChatGPT to evaluate and recommend journals before submitting a paper.

Reliability of ChatGPT for journal quality evaluation

The use of ChatGPT is widespread, yet there exist no strong evidences regarding its reliability in evaluating journal authenticity. In order fill this gap, we shaped three datasets. The first dataset contained a list of all currently detected hijacked journals because existing lists of hijacked journals were not up-to-date.^32-34 We used an updated (to July 8, 2023) online list of hijacked journals published by Retraction Watch Blog entitled “Hijacked Journal Checker”³⁵ URL: https://retractionwatch.com/the-retraction-watch-hijacked-journal-checker/ for this dataset.

The second dataset contained a list of 50 potential predatory publishers gathered from the https://beallslist.net by random sampling. The third dataset contained a list of 100 potential predatory journals gathered from https://beallslist.net by random sampling. There has been some criticism about the use of potential lists of predatory publishers and journals; however, we used these lists solely to examine ChatGPT’s ability to provide accurate information. We presented ChatGPT with the hijacked journals’ URLs and requested its opinion of the quality and authenticity of each URL. Additionally, we asked ChatGPT to provide its opinion on each journal/publisher on the of list of predatory journals and publishers (by use of exact name of journals and publishers). Results are shown in Figures 1 and 2.

Figure 1.

Percentage of correct answers regarding hijacked journals

Figure 2.

Percentage of correct answers regarding predatory journals

ChatGPT recommended about 30% of hijacked journals as reputable publications. For example, https://www.hfsp-journal.org is a hijacked version of an authentic journal, Frontiers of Interdisciplinary Research in The Life Sciences (https://www.hfsp.org/hfsp-news/alert-fake-journal-fraudulent-use-hfsp-name).²¹ When ChatGPT was asked, “Is https://www.hfsp-journal.org an authentic journal?” it responded:

Yes, the HFSP Journal is an authentic journal. It is an open-access, peer-reviewed scientific journal that publishes original research articles, reviews, and perspectives in the fields of life sciences and related disciplines. The journal is published by the Human Frontier Science Program (HFSP) Organization, which is an international organization that supports frontier research in the life sciences. The HFSP Journal is indexed in several reputable databases, including PubMed Central, Web of Science, and Scopus, and its editorial board consists of experts in various fields of life sciences. The journal has a rigorous peer-review process in place to ensure the quality and validity of published research. Therefore, the HFSP Journal is a legitimate and reputable journal in the scientific community.

When asked “Is https://www.hfsp-journal.org hijacked journal?” ChatGPT responded, “As far as I am aware, there is no evidence to suggest that the HFSP Journal is a hijacked journal”.

Although ChatGPT demonstrated an ability to provide appropriate recommendations as well as accurate and rich information about potential predatory publishers, 28% of its responses were mistaken recommendations of predatory journals. Our findings were that ChatGPT is not error free in this respect and cannot be relied upon as a tool for journal quality evaluation. This means that ChatGPT did not use current literature well, because most of information about predatory and hijacked journals (plus their URLs) are freely available on internet from 2015 until now.

The solution that we need: Journal Checker Chatbot

Given the tendency of ChatGPT to mistakenly recommend questionable journals, we developed a trial limited “Journal Checker Chatbot,” using Botpress. It is a platform based on natural language recognition that provides a visual tool to set up a chatbot.^36,37 There is also online version of Botpress that support easy publishing of chatbot in the web. This chatbot also uses OpenAI API to provide next-generation chatbots and more intelligence answers based on the provided dataset.^38,39 For the purpose of this paper, only data pertaining to the identification and evaluation of authentic and hijacked journals were prepared to help the chatbot learn. The term “hijacked journals” was searched in Google Scholar (22 June 2022), and the full texts of all publications whose main theme pertained to hijacked journals were collected. In addition, an updated list of hijacked journals was collected and added to the dataset.

The developed chatbot is accessible online via this address: https://t.me/hijackedjournalcheckerbot. It can answer questions about hijacked journals with acceptable reliability; however, as this is a trial version and for test purpose not wide spread usage, researchers should be cautious with its use. The aim was to show it is possible to have a research specific chatbot. This chatbot has the ability to answer questions pertaining to the evaluation of particular journals, the nature of hijacked journals, and various methods of detection. Furthermore, it can be useful in teaching novice researchers about the prevalence and dangers of hijacked journals. Figure 3 illustrates screenshots from chatbot. Questions that ChatGPT answered inaccurately were answered correctly by the Journal Checker Chatbot.

Figure 3.

Screenshots from Journal Checker Chatbot

Conclusion

The current study analyzes the reliability of ChatGPT for journal quality evaluation. The results indicate that ChatGPT is not a reliable tool because it recommends questionable journals for publishing. As questionable journals are proliferating and their influence is spreading via a variety of means (e.g., emails, advertisements in search engines), a usable and accessible tool for both journal detection and related education is much needed. To show how it can meet this need, for test purpose, a chatbot entitled “Journal Checker Chatbot” was developed. This chatbot is a beta version and for test purpose that will undergo improvements to ensure an even higher level of accuracy. With future investment in its infrastructure, we will use predatory journals data to train this chatbot. The author will utilize advanced data technologies to gather lists of hijacked journals which the chatbot will use automatically to increase its accuracy and undermine the damage being caused by hijacked and predatory journals.

Competing Interests

No conflict of interest.

Ethical Approval

Not applicable.

Funding

There is no funding.

References

Cai Z, Liu L, Chen B, Wang Y. Artificial Intelligence: From Beginning to Date. China: World Scientific; 2021.
Golan R, Reddy R, Muthigi A, Ramasamy R. Artificial intelligence in academic writing: a paradigm-shifting technological advance. Nat Rev Urol 2023; 20(6):327-8. doi: 10.1038/s41585-023-00746-x [Crossref] [ Google Scholar]
Biswas SS. Role of ChatGPT in public health. Ann Biomed Eng 2023; 51(5):868-9. doi: 10.1007/s10439-023-03172-7 [Crossref] [ Google Scholar]
Liebrenz M, Schleifer R, Buadze A, Bhugra D, Smith A. Generating scholarly content with ChatGPT: ethical challenges for medical publishing. Lancet Digit Health 2023; 5(3):e105-e6. doi: 10.1016/s2589-7500(23)00019-5 [Crossref] [ Google Scholar]
Lund BD, Wang T. Chatting about ChatGPT: how may AI and GPT impact academia and libraries?. Libr Hi Tech News 2023; 40(3):26-9. doi: 10.1108/LHTN-01-2023-0009 [Crossref] [ Google Scholar]
Ali MJ, Djalilian A. Readership awareness series - paper 4: chatbots and ChatGPT - ethical considerations in scientific publications. Semin Ophthalmol 2023; 38(5):403-4. doi: 10.1080/08820538.2023.2193444 [Crossref] [ Google Scholar]
Cabanac G, Labbé C. Prevalence of nonsensical algorithmically generated papers in the scientific literature. J Assoc Inf Sci Technol 2021; 72(12):1461-76. doi: 10.1002/asi.24495 [Crossref] [ Google Scholar]
Gao CA, Howard FM, Markov NS, Dyer EC, Ramesh S, Luo Y, et al. Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. bioRxiv [Preprint]. December 27, 2022. Available from: https://www.biorxiv.org/content/10.1101/2022.12.23.521610v1.
Sun GH, Hoelscher SH. The ChatGPT storm and what faculty can do. Nurse Educ 2023; 48(3):119-24. doi: 10.1097/nne.0000000000001390 [Crossref] [ Google Scholar]
van Dis EAM, Bollen J, Zuidema W, van Rooij R, Bockting CL. ChatGPT: five priorities for research. Nature 2023; 614(7947):224-6. doi: 10.1038/d41586-023-00288-7 [Crossref] [ Google Scholar]
Beall J. Guaranteed hits: how to make your library’s web site stand out in web search engines. Coll Res Libr News 2019; 59(3):160-2. [ Google Scholar]
Beall J. Open access, research communities, and a defense against predatory journals. Cent Asian J Med Hypotheses Ethics 2021; 2(1):14-7. [ Google Scholar]
Frandsen TF. Authors publishing repeatedly in predatory journals: an analysis of Scopus articles. Learn Publ 2022; 35(4):598-604. doi: 10.1002/leap.1489 [Crossref] [ Google Scholar]
Kendall G, Linacre S. Predatory Journals: Revisiting Beall’s Research. Publ Res Q 2022; 38(3):530-43. doi: 10.1007/s12109-022-09888-z [Crossref] [ Google Scholar]
Oermann MH, Wrigley J, Nicoll LH, Ledbetter LS, Carter-Templeton H, Edie AH. Integrity of databases for literature searches in nursing: avoiding predatory journals. ANS Adv Nurs Sci 2021; 44(2):102-10. doi: 10.1097/ans.0000000000000349 [Crossref] [ Google Scholar]
Dadkhah M, Bianciardi G. Ranking predatory journals: solve the problem instead of removing it!. Adv Pharm Bull 2016; 6(1):1-4. doi: 10.15171/apb.2016.001 [Crossref] [ Google Scholar]
Kendall G. Beall’s legacy in the battle against predatory publishers. Learn Publ 2021; 34(3):379-88. doi: 10.1002/leap.1374 [Crossref] [ Google Scholar]
Kennedy MS. Predatory publishing is alive and well. Am J Nurs 2020; 120(4):7. doi: 10.1097/01.NAJ.0000659932.41029.4a [Crossref] [ Google Scholar]
Nejadghanbar H, Hu G. Where predatory and mainstream journals differ: a study of language and linguistics journals. Learn Publ 2022; 35(4):574-84. doi: 10.1002/leap.1485 [Crossref] [ Google Scholar]
Dadkhah M, Maliszewski T, Jazi MD. Characteristics of hijacked journals and predatory publishers: our observations in the academic world. Trends Pharmacol Sci 2016; 37(6):415-8. doi: 10.1016/j.tips.2016.04.002 [Crossref] [ Google Scholar]
Dadkhah M, Rahimnia F, Darbyshire P, Borchardt G. Ten (Bad) reasons researchers publish their papers in hijacked journals. J Clin Nurs 2021; 30(19-20):e60-e3. doi: 10.1111/jocn.15947 [Crossref] [ Google Scholar]
Dadkhah M, Rahimnia F, Rafati Niya S, Borchardt G. Jourchain: using blockchain to avoid questionable journals. Ir J Med Sci 2022; 191(3):1435-9. doi: 10.1007/s11845-021-02697-x [Crossref] [ Google Scholar]
Bohannon J. How to hijack a journal. Science 2015; 350(6263):903-5. doi: 10.1126/science.350.6263.903 [Crossref] [ Google Scholar]
Dadkhah M, Lagzian M, Borchardt G. Questionable papers in citation databases as an issue for literature review. J Cell Commun Signal 2017; 11(2):181-5. doi: 10.1007/s12079-016-0370-6 [Crossref] [ Google Scholar]
Beall J. Predatory journals and the breakdown of research cultures. Inf Dev 2015; 31(5):473-6. doi: 10.1177/0266666915601421 [Crossref] [ Google Scholar]
Dadkhah M, Maliszewski T. Hijacked journals - threats and challenges to countries’ scientific ranking. Int J Technol Enhanc Learn 2015; 7(3):281-8. doi: 10.1504/ijtel.2015.072819 [Crossref] [ Google Scholar]
Forero DA, Oermann MH, Manca A, Deriu F, Mendieta-Zerón H, Dadkhah M. Negative effects of “predatory” journals on global health research. Ann Glob Health 2018; 84(4):584-9. doi: 10.9204/aogh.2389 [Crossref] [ Google Scholar]
Musiał-Karg M, Zamęcki Ł, Rak J. Debate: publish or perish? How legal regulations affect scholars’ publishing strategies and the spending of public funds by universities. Public Money Manag 2023; 43(8):779-80. doi: 10.1080/09540962.2023.2213920 [Crossref] [ Google Scholar]
Shrestha J. Predatory journals as threats to the academic publishing: a review. J Agric Nat Resour 2021; 4(2):1-10. [ Google Scholar]
Tsigaris P, Kendall G, Teixeira da Silva JA. What does ChatGPT advise about predatory publishing? J Prof Nurs2023. 10.1016/j.profnurs.2023.08.002.
Dadkhah M, Raja AM, Memon AR, Borchardt G, Nedungadi P, Abu-Eteen K. A toolkit for detecting fallacious calls for papers from potential predatory journals. Adv Pharm Bull 2023; 13(4):627-34. doi: 10.34172/apb.2023.068 [Crossref] [ Google Scholar]
Abalkina A. Detecting a network of hijacked journals by its archive. Scientometrics 2021; 126(8):7123-48. doi: 10.1007/s11192-021-04056-0 [Crossref] [ Google Scholar]
Dadkhah M, Maliszewski T, Teixeira da Silva JA. Hijacked journals, hijacked web-sites, journal phishing, misleading metrics, and predatory publishing: actual and potential threats to academic integrity and publishing ethics. Forensic Sci Med Pathol 2016; 12(3):353-62. doi: 10.1007/s12024-016-9785-x [Crossref] [ Google Scholar]
Jalalian M, Dadkhah M. The full story of 90 hijacked journals from August 2011 to June 2015. GeogrPannonica 2015; 19(2):73-87. doi: 10.5937/GeoPan1502073J [Crossref] [ Google Scholar]
Else H. Hijacked-journal tracker helps researchers to spot scam websites. Nature 2022. 10.1038/d41586-022-01666-3.
Gorgias A. Implementation of a ChatBot for Helping Users Find the Nearest and Cheapest Gas Station. https://repository.ihu.edu.gr/xmlui/bitstream/handle/11544/29598/a.gorgias_mwc_01-12-2019.pdf?sequence=1. Published 2019. Accessed July 10, 2023.
Shirokov SA, Rypalov AJ. Dispatching method of incoming events for ERP system based on analysis of emails using artificial intelligent technique. J Phys Conf Ser 2020; 1694(1):012034. doi: 10.1088/1742-6596/1694/1/012034 [Crossref] [ Google Scholar]
Botpress Cloud. https://github.com/botpress/botpress. Accessed July 10, 2023.
Botpress. Tips & Tricks. https://botpress.com/docs/cloud/getting-started/tips-and-tricks/#use-chatgpt-api-in-execute-code. Published 2023. Accessed July 10, 2023.