To read this content please select one of the options below:

Can we trust LLMs to help us? An examination of the potential use of GPT-4 in generating quality literature reviews

Min Zhao (School of Economics and Management, Faculty of Agriculture, Shanxi Agricultural University, Taigu, China)
Fuan Li (Department of Management, Marketing and Professional Sales, William Paterson University of New Jersey, Wayne, New Jersey, USA)
Francis Cai (Department of Economics and Finance, William Paterson University of New Jersey, Wayne, New Jersey, USA)
Haiyang Chen (College of Business, Western Carolina University, Cullowhee, North Carolina, USA)
Zheng Li (School of Economics, Liaoning University, Shenyang, China)

Nankai Business Review International

ISSN: 2040-8749

Article publication date: 24 September 2024

113

Abstract

Purpose

This study aims to examine the ability of Generative Pre-trained Transformer 4 (GPT-4), one of the most powerful large language models, to generate a literature review for peer-reviewed journal publications. The objective is to determine whether business scholars can rely on GPT-4’s assistance with literature reviews and how the nature of human–artificial intelligence (AI) interaction may affect the quality of the reviews generated by GPT-4.

Design/methodology/approach

A survey of 30 experienced researchers was conducted to assess the quality of the literature reviews generated by GPT-4 in comparison with a human-authored literature review published in a Social Science Citation Index (SSCI) journal. The data collected were then analyzed with analysis of variance to ascertain whether we may trust GPT-4’s assistance in writing literature reviews.

Findings

The statistical analysis reveals that when a highly structured approach being used, GPT-4 can generate a high-quality review comparable to that found in an SSCI journal publication. However, when a less structured approach is used, the generated review lacks comprehensive understating and critical analysis, and is unable to identify literature gaps for future research, although it performed well in adequate synthesis and quality writing. The findings suggest that we may trust GPT-4 to generate literature reviews that align with the publication standards of a peer-reviewed journal when using a structured approach to human–AI interaction.

Research limitations/implications

The findings suggest that we may trust GPT-4 to generate literature reviews that align with the publication standards of a peer-reviewed journal when using a structured approach to human–AI interaction. Nonetheless, cautions should be taken due to the limitations of this study discussed in the text.

Originality/value

By breaking down the specific tasks of a literature review and using a quantitative rather than qualitative assessment method, this study provides robust and more objective findings about the ability of GPT-4 to assist us with a very important research task. The findings of this study should enhance our understanding of how GPT-4 may change our research endeavor and how we may take a full advantage of the advancement in AI technology in the future research.

Keywords

Acknowledgements

The authors express their deep gratitude to Professor Dahui Li (the editor) and the two anonymous reviewers for their constructive comments and invaluable suggestions, which have significantly improved the quality of this paper.

Citation

Zhao, M., Li, F., Cai, F., Chen, H. and Li, Z. (2024), "Can we trust LLMs to help us? An examination of the potential use of GPT-4 in generating quality literature reviews", Nankai Business Review International, Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/NBRI-12-2023-0115

Publisher

:

Emerald Publishing Limited

Copyright © 2024, Emerald Publishing Limited

Related articles