Can we trust LLMs to help us? An examination of the potential use of GPT-4 in generating quality literature reviews

Abstract

Purpose

This study aims to examine the ability of Generative Pre-trained Transformer 4 (GPT-4), one of the most powerful large language models, to generate a literature review for peer-reviewed journal publications. The objective is to determine whether business scholars can rely on GPT-4’s assistance with literature reviews and how the nature of human–artificial intelligence (AI) interaction may affect the quality of the reviews generated by GPT-4.

Design/methodology/approach

A survey of 30 experienced researchers was conducted to assess the quality of the literature reviews generated by GPT-4 in comparison with a human-authored literature review published in a Social Science Citation Index (SSCI) journal. The data collected were then analyzed with analysis of variance to ascertain whether we may trust GPT-4’s assistance in writing literature reviews.

Findings

The statistical analysis reveals that when a highly structured approach being used, GPT-4 can generate a high-quality review comparable to that found in an SSCI journal publication. However, when a less structured approach is used, the generated review lacks comprehensive understating and critical analysis, and is unable to identify literature gaps for future research, although it performed well in adequate synthesis and quality writing. The findings suggest that we may trust GPT-4 to generate literature reviews that align with the publication standards of a peer-reviewed journal when using a structured approach to human–AI interaction.

Research limitations/implications

The findings suggest that we may trust GPT-4 to generate literature reviews that align with the publication standards of a peer-reviewed journal when using a structured approach to human–AI interaction. Nonetheless, cautions should be taken due to the limitations of this study discussed in the text.

Originality/value

By breaking down the specific tasks of a literature review and using a quantitative rather than qualitative assessment method, this study provides robust and more objective findings about the ability of GPT-4 to assist us with a very important research task. The findings of this study should enhance our understanding of how GPT-4 may change our research endeavor and how we may take a full advantage of the advancement in AI technology in the future research.

Citation

Zhao, M., Li, F., Cai, F., Chen, H. and Li, Z. (2024), "Can we trust LLMs to help us? An examination of the potential use of GPT-4 in generating quality literature reviews", Nankai Business Review International, Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/NBRI-12-2023-0115