What if ChatGPT generates quantitative research data? A case study in tourism

Abstract

Purpose

This study aims to explore whether Chat Generative Pre-training Transformer (ChatGPT) can produce quantitative data sets for researchers who could behave unethically through data fabrication.

Design/methodology/approach

A two-stage case study related to the field of tourism was conducted, and ChatGPT (v.3.5.) was asked to respond to the first questionnaire on behalf of 400 participants and the second on behalf of 800 participants. The artificial intelligence (AI)-generated data sets’ quality was statistically tested via descriptive statistics, correlation analysis, exploratory factor analysis, confirmatory factor analysis and Harman's single-factor test.

Findings

The results revealed that ChatGPT could respond to the questionnaires as the number of participants at the desired sample size level and could present the generated data sets in a table format ready for analysis. It was also observed that ChatGPT's responses were systematical, and it created a statistically ideal data set. However, it was noted that the data produced high correlations among the observed variables, the measurement model did not achieve sufficient goodness of fit and the issue of common method bias emerged. The conclusion reached is that ChatGPT does not or cannot yet generate data of suitable quality for advanced-level statistical analyses.

Originality/value

This study shows that ChatGPT can provide quantitative data to researchers attempting to fabricate data sets unethically. Therefore, it offers a new and significant argument to the ongoing debates about the unethical use of ChatGPT. Besides, a quantitative data set generated by AI was statistically examined for the first time in this study. The results proved that the data produced by ChatGPT is problematic in certain aspects, shedding light on several points that journal editors should consider during the editorial processes.

研究目的

本研究旨在探讨ChatGPT是否能够为那些可能通过数据伪造行为不道德的研究人员生成定量数据集。

研究方法

本研究进行了与旅游领域相关的两阶段案例研究, 并要求ChatGPT（v.3.5.）代表400名参与者回答第一个问卷, 以及代表800名参与者回答第二个问卷。通过描述统计、相关分析、探索性因子分析、验证性因子分析和哈曼的单因素测试对人工智能生成的数据集的质量进行了统计测试。

研究发现

结果显示, ChatGPT能够按照所需的样本大小水平回答问卷, 并以表格格式呈现生成的数据集, 以便进行分析。还观察到ChatGPT的回答是系统性的, 并且它创建了一个在统计上理想的数据集。然而, 本研究注意到所产生的数据在观察变量之间存在较高的相关性, 测量模型未能达到足够的拟合度, 并出现了共同方法偏差的问题。本研究得出的结论是, ChatGPT目前不能生成适用于高级统计分析的数据, 或者说不适合这样做。

研究创新

本研究表明, ChatGPT可以为试图不道德地伪造数据集的研究人员提供定量数据。因此, 它为关于ChatGPT不道德使用的持续争论提供了一个新而重要的论点。此外, 在本研究中首次对由人工智能生成的定量数据集进行了统计检验。结果表明, ChatGPT生成的数据在某些方面存在问题, 为期刊编辑在编辑过程中考虑的几个要点提供了启示。

Citation

Sop, S.A. and Kurçer, D. (2024), "What if ChatGPT generates quantitative research data? A case study in tourism", Journal of Hospitality and Tourism Technology, Vol. 15 No. 2, pp. 329-343. https://doi.org/10.1108/JHTT-08-2023-0237