Can artificial intelligence produce a convincing accounting research article?

Abstract

Purpose

This study aims to establish whether accounting research articles can be potentially generated by artificial intelligence. If artificial intelligence can produce quality work, the integrity of academic research may be compromised.

Design/methodology/approach

ChatGPT was used to create a paper on a meta-analysis of the relationship between sustainability reporting and value relevance. After the paper was generated, references had to be added by hand based on the citations created by ChatGPT. The paper was then presented as-is for review.

Findings

ChatGPT was able to create a relatively good-quality research paper that received two major revisions from independent specialists in the field of accounting and finance. Even though there is uncertainty regarding the appropriateness of all the references and the results cannot be confirmed, there is a risk that a reviewer may find the paper publishable because reviewers are not compelled to check references and the accuracy of results if proper methods were used that appear to be sufficient at face value.

Originality/value

Artificial intelligence for academic writing is still relatively new, and there is still significant uncertainty as to the impact it may have on scholarly research. This is especially problematic because artificial intelligence applications improve by the second.

Citation

du Toit, E. (2024), "Can artificial intelligence produce a convincing accounting research article?", Accounting Research Journal, Vol. 37 No. 4, pp. 365-380. https://doi.org/10.1108/ARJ-04-2023-0105

1. Introduction

Artificial intelligence (AI) applications have taken the world by storm at the start of 2023 after the activation of ChatGPT in November 2022. Even though AI is not a new concept, it has never affected academia so directly. Some parties are excited about AI’s potential uses in teaching and research, whereas others are concerned about the abuse of AI and the impairment of the integrity of academic work by students and researchers.

A promising path for innovation and field advancement is the junction of accounting research and AI such as ChatGPT. Aspects of accounting research could be streamlined by machine learning algorithms, as demonstrated by ChatGPT's natural language processing power. With previously unheard-of speed and accuracy, these AI tools can help academics analyse enormous volumes of financial data, find trends, and provide insights. Furthermore, AI-driven solutions make it possible to automate procedures like literature reviews, hypothesis testing and data collection, which improves the effectiveness of research. To maintain the integrity and dependability of study findings, ethical issues like algorithmic bias and data privacy must be properly taken into account, as with any technological integration. In general, there is a lot of potential for new discoveries, improved methodological approaches and solving challenging issues within the field of accounting research due to the synergy between AI and accounting research.

There are also several possible issues and obstacles with integrating AI into accounting research. First, there is the possibility of algorithmic bias, which could provide biased study results as AI systems inadvertently reinforce or even magnify biases found in the training data. Concerns exist over data security and privacy as well, particularly when managing sensitive financial data. Maintaining transparency and accountability in research procedures is vital to preserving the integrity of findings as AI systems get more complex and autonomous. Furthermore, researchers may become less creative and critical thinkers as a result of relying too much on AI tools, which could restrict their ability to investigate novel theories or approaches. In a worst-case scenario, researchers may even be tempted to rely on AI completely, creating research that is unfound and untrue.

The purpose of this paper is to establish whether an AI-generated research paper can potentially be accepted as being of sufficient quality in terms of language, method and results to be published in an accredited peer-reviewed journal article.

The remainder of this paper is structured as follows. First, the methodology the author followed to generate and improve the AI paper is presented. Second, the AI-generated paper is presented. Third, the comments and decisions of two reviewers of the AI-generated paper are provided. The paper concludes with a final discussion.

2. Research design and method

ChatGPT was used on 5 April 2023 to create this work. The first question directed to the AI application was the following:

• Please prepare for me the headings for an accounting journal article that conducts a meta-analysis of all accounting research papers that appeared in ABDC A* rated or A-rated journals and that deals with the relationship between sustainability reporting and value relevance.

Thereafter, each heading was populated through a new request in ChatGPT, for example:

• Please prepare for me the Introduction “background and significance of the study” section for an accounting journal article that conducts a meta-analysis of all accounting research papers that appeared in ABDC A* rated or A-rated journals and that deals with the relationship between sustainability reporting and value relevance. Make use of real citations.

The italicised words in the question were in each instance changed to reflect the different headings applicable. The same question was thus posed to ChatGPT for each heading to ensure consistency in the results.

The title of the paper was created through the following request:

• Please prepare for me a title for an accounting journal article that conducts a meta-analysis of all accounting research papers that appeared in ABDC A* rated or A-rated journals and that deals with the relationship between sustainability reporting and value relevance.

After generating the paper, the author had to make some changes to the paper to enhance it for review. For example, the headings that AI created for the introduction were removed, as it is not customary in accounting journals to have such headings. Layout and wording were changed here and there but kept to the bare minimum. The research objectives in the introduction were reduced to a more focused introduction. Grammarly (another AI tool, albeit generally acceptable) was used to improve the language.

The author had to review each of the citations provided by AI to investigate whether it refers to a real paper. References then also had to be added to the reference list to create a convincing article. The search for the references was in some cases a challenge, especially where there is more than one reference with the same year and authors. The title and/or abstract had to be reviewed next to ascertain the relevance of the paper. However, the reference papers were not read in detail to establish whether the citation was correct.

As a final check of the paper, it was submitted to the Turnitin Plagiarism package. The similarity percentage is 17%, which would not necessarily raise red flags. However, Turnitin now includes a feature that tests for potential AI use in a text. The AI percentage is 79%.

After the paper was prepared, it was sent to two independent reviewers who are knowledgeable in the field of accounting and finance for an informal review. They were given standard review questions and were not informed that it was an AI-generated paper, to allow for an unbiased review. The reviewers' feedback and decisions are presented in Section 4.

3. Artificial intelligence–generated paper

Assessing The Relationship Between Sustainability Reporting And Value Relevance: A Meta-Analysis Of A And A* Accounting Journals [1]

1. Introduction

Sustainability reporting has emerged as a vital tool for businesses, investors and regulators globally, facilitating the communication of nonfinancial performance metrics to stakeholders. These reports typically encompass a company's environmental, social and governance (ESG) practices, essential factors for investors making informed decisions (Haniffa and Cooke, 2005). While some studies suggest that sustainability reporting can bolster a company's reputation and financial performance (Orlitzky et al., 2003; Eccles and Serafeim, 2013), others present conflicting findings (Cheng et al., 2014; Liu et al., 2018; Wu et al., 2022).

We initially screen studies based on titles and abstracts to ensure they meet our inclusion criteria:

• publication in an ABDC A* or A-rated accounting journal;

• examination of the relationship between sustainability reporting and value relevance; and

• provision of empirical evidence through regression analysis or event studies.

Following this, we conduct a thorough review of selected studies to assess methodological quality and extract pertinent data, including measures of sustainability reporting, value relevance, sample size, and statistical results. Subsequently, employing a random-effects model, we conduct a meta-analysis to synthesize findings and estimate the overall effect size of the relationship. Additionally, a meta-regression analysis is performed to investigate potential sources of heterogeneity. Finally, we offer a comprehensive summary of findings, discussing their implications for policymakers, regulators, and investors, while also identifying areas for future research.

By following this systematic and rigorous approach, we aim to provide a comprehensive and reliable assessment of the relationship between sustainability reporting and value relevance in the accounting literature.

2. Literature review

2.1 Overview of sustainability reporting and value relevance literature

Sustainability reporting has become an increasingly important aspect of corporate social responsibility (CSR) practices, as firms seek to demonstrate their commitment to sustainability and environmental responsibility. The Global Reporting Initiative (GRI) defines sustainability reporting as “the practice of disclosing, measuring, and being accountable to internal and external stakeholders for organizational performance toward the goal of sustainable development” (GRI, 2018).

Several studies have examined the relationship between sustainability reporting and value relevance in the accounting literature. Value relevance refers to the degree to which accounting information is reflected in stock prices, and is a key measure of the usefulness of accounting information to investors (Ohlson, 1995).

Previous research has provided mixed results on the relationship between sustainability reporting and value relevance. For example, some studies have found a positive relationship between sustainability reporting and value relevance (Botosan and Plumlee, 2013; Cho et al., 2015), while others have found no significant relationship (Cheng et al., 2014; KPMG, 2014) or a negative relationship (Dam and Scholtens, 2013; Daske et al., 2013).

Moreover, the literature has highlighted several potential mechanisms through which sustainability reporting can influence value relevance, such as increased transparency and accountability (Patten, 2002), improved stakeholder relationships (Brammer and Pavelin, 2006), and enhanced reputation and legitimacy (Gray et al., 1995).

However, the literature has also identified several limitations and challenges in measuring and interpreting the relationship between sustainability reporting and value relevance. For example, different measures of sustainability reporting, such as the GRI index, the Carbon Disclosure Project (CDP), and the Dow Jones Sustainability Index (DJSI), may capture different aspects of sustainability reporting and have varying degrees of reliability and validity (Cormier and Magnan, 2015). Similarly, different measures of value relevance, such as the market value of equity or earnings per share, may have different sensitivities to accounting information and reflect different aspects of firm performance (Barth and Clinch, 2009).

Therefore, this meta-analysis aims to provide a comprehensive and up-to-date assessment of the relationship between sustainability reporting and value relevance in the accounting literature, while also addressing the heterogeneity and limitations of the existing studies. By doing so, we aim to provide valuable insights for policy-makers, regulators, and investors on the importance of sustainability reporting and its impact on firm performance and market valuation.

2.2 Theoretical frameworks used in previous studies

Previous studies on the relationship between sustainability reporting and value relevance have used various theoretical frameworks to explain the potential mechanisms and drivers of this relationship.

Agency theory suggests that the relationship between sustainability reporting and value relevance can be explained by the alignment of interests between management and shareholders. According to agency theory, management has an incentive to engage in sustainability reporting to signal their commitment to long-term value creation and to reduce information asymmetry between themselves and investors (Jensen and Meckling, 1976). By providing more complete and accurate information about their sustainability practices, firms may increase their transparency and reduce the agency costs associated with information asymmetry (Kang et al., 2018).

Stakeholder theory suggests that the relationship between sustainability reporting and value relevance can be explained by the interests and expectations of various stakeholders, such as customers, employees, and the broader community. According to stakeholder theory, sustainability reporting can improve stakeholder relationships and enhance the reputation and legitimacy of the firm (Brammer and Pavelin, 2006). This, in turn, may lead to increased customer loyalty, better employee morale, and greater community support, which can ultimately translate into higher financial performance and market valuation (Orlitzky et al., 2003).

Legitimacy theory suggests that the relationship between sustainability reporting and value relevance can be explained by the need for firms to maintain their social and environmental legitimacy. According to legitimacy theory, firms have a social contract with society that requires them to meet certain expectations and norms of behavior (Suchman, 1995). By engaging in sustainability reporting, firms can demonstrate their compliance with these norms and signal their commitment to responsible business practices (Dowling and Pfeffer, 1975). This, in turn, may enhance their legitimacy and reputation, leading to improved financial performance and market valuation (Patten, 2002).

While these theoretical frameworks have been useful in guiding previous studies on sustainability reporting and value relevance, they also have their limitations and criticisms. For example, agency theory has been criticized for its narrow focus on shareholder value and its assumptions about the rationality and self-interest of actors (Donaldson and Preston, 1995). Similarly, stakeholder theory has been criticized for its lack of precision and its difficulty in operationalizing stakeholder interests (Phillips, 2003). Therefore, future studies may benefit from a more integrated and multidisciplinary approach that takes into account the complex and dynamic nature of sustainability reporting and value relevance.

2.3 Empirical findings and research gaps

Empirical studies examining the relationship between sustainability reporting and value relevance have produced mixed results, with some studies reporting a positive association between sustainability reporting and firm value, and others reporting no significant relationship or even a negative association. In this section, we provide an overview of the main empirical findings and identify some of the research gaps that need to be addressed.

Some studies have found a positive association between sustainability reporting and firm value. For example, Bellovary et al. (2007) found that firms with higher sustainability performance scores had higher market valuations than firms with lower scores. Similarly, Cho et al. (2014) found that firms with better sustainability performance were more likely to experience positive stock returns in response to news events. These findings suggest that sustainability reporting may have a positive impact on market valuation by improving the perception of investors and stakeholders about the firm's long-term prospects.

Other studies have found no significant association between sustainability reporting and firm value. For example, Gao and Bryan (2017) found that sustainability reporting had no significant impact on firm value, as measured by Tobin's Q. Similarly, Dissanayake et al. (2016) found no significant relationship between sustainability reporting and market performance, as measured by abnormal returns. These findings suggest that sustainability reporting may not always have a significant impact on market valuation and that other factors, such as industry characteristics and firm-specific factors, may play a more important role.

A few studies have reported a negative association between sustainability reporting and firm value. For example, Busch et al. (2019) found that firms with higher sustainability disclosure scores had lower market valuations, particularly for firms in high-emitting industries. Similarly, Kellermanns et al. (2012) found that sustainability disclosure had a negative impact on firm performance, as measured by return on assets. These findings suggest that sustainability reporting may not always have a positive impact on market valuation and that there may be trade-offs between sustainability performance and financial performance in certain contexts.

Despite the growing body of literature on sustainability reporting and value relevance, there are still several research gaps that need to be addressed. For example, many previous studies have focused on the relationship between sustainability reporting and market valuation, but have not examined the underlying mechanisms and drivers of this relationship. Furthermore, there is a need for more longitudinal studies that examine the long-term effects of sustainability reporting on firm performance and valuation. Finally, there is a need for more cross-country studies that examine the role of institutional and cultural factors in shaping the relationship between sustainability reporting and firm value.

3. Research methodology

3.1 Data collection and selection process

To conduct a comprehensive meta-analysis of the literature on the relationship between sustainability reporting and value relevance, we conducted a systematic search of all accounting research papers published in ABDC A* rated or A-rated journals between 2000 and 2022. We chose ABDC A* rated or A-rated journals as they are the most highly regarded and reputable accounting journals, and they are likely to contain the highest quality and most influential research in the field.

We used a combination of electronic databases (such as Scopus, Web of Science, and Google Scholar) and manual searches of the reference lists of relevant papers to identify potential studies. We used a combination of search terms related to sustainability reporting (such as “sustainability reporting,” “environmental disclosure,” “corporate social responsibility” and “ESG reporting”) and value relevance (such as “market valuation,” “stock returns,” “Tobin's Q” and “abnormal returns”).

After obtaining a list of potentially relevant studies, we then applied a series of inclusion and exclusion criteria to select studies for the meta-analysis. Our inclusion criteria were:

• the study must have examined the relationship between sustainability reporting and firm value, as measured by some proxy of market valuation (such as Tobin's Q, stock returns or abnormal returns);

• the study must have used quantitative methods to analyze the data, such as regression analysis or event study methodology;

• the study must have been published in an ABDC A* or A-rated accounting journal between 2000 and 2022; and

• the study must have been written in English.

We excluded studies that:

• did not examine the relationship between sustainability reporting and value relevance;

• used qualitative methods to analyze the data, such as content analysis or case studies;

• were published in non-ABDC A* or A-rated journals; or

• were not written in English.

After applying these criteria, we obtained a final sample of 38 studies that met our inclusion criteria as well as our research question and objectives. These studies were published in a range of accounting journals, including The Accounting Review, Journal of Accounting and Economics and Journal of Business Ethics. We extracted relevant information from each study, such as the sample size, the measure of sustainability reporting, the measure of firm value and the statistical results. We then synthesized the results using meta-analysis techniques to provide a comprehensive overview of the literature on sustainability reporting and value relevance.

The first criterion ensured that the studies were directly relevant to our research question of examining the relationship between sustainability reporting and value relevance. The second criterion excluded studies that used methods that were not quantitative, which ensured that the studies were comparable in terms of their analysis. The third criterion ensured that the studies were of high quality and met rigorous standards for publication. The fourth criterion ensured that we could access and analyze the studies effectively.

3.2 Data extraction process

To extract the relevant data from the selected studies, we followed a standardized and systematic approach. We developed a data extraction form that included key information such as the author, year of publication, sample size, research design, sustainability reporting measures and value relevance measures.

Two reviewers independently extracted data from each of the selected studies and cross-checked the extracted data to ensure accuracy and consistency. Any discrepancies were resolved through discussion and consensus.

In addition to extracting data on the relationship between sustainability reporting and value relevance, we also extracted data on the control variables used in each study. This allowed us to examine the impact of these variables on the relationship between sustainability reporting and value relevance.

To assess the quality of each study, we used the Cochrane Risk of Bias Tool, which is a widely used tool for assessing the risk of bias in systematic reviews and meta-analyses (Higgins et al., 2019). We assessed the risk of bias in each study based on six domains: random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data and selective reporting.

The data extraction process was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, which are widely accepted guidelines for conducting systematic reviews and meta-analyses (Liberati et al., 2009).

By following this systematic and rigorous data extraction process, we were able to ensure the accuracy and completeness of the data extracted from each study. This enabled us to conduct a comprehensive meta-analysis of the relationship between sustainability reporting and value relevance in accounting literature.

3.3 Quality assessment of the selected studies

To assess the quality of the studies included in this meta-analysis, we will use the guidelines provided by the PRISMA statement (Moher et al., 2009). In addition, we will also use the Cochrane Risk of Bias Tool (Higgins et al., 2011) to assess the quality of randomized controlled trials included in our analysis. The quality assessment will focus on the following domains:

• study design and methodology;

• sample size and characteristics;

• data collection and measurement tools;

• statistical analysis methods; and

• potential sources of bias.

Each domain is rated as low, high or unclear risk of bias based on the information provided in the selected studies. The overall quality of each study will be assessed based on the rating of each domain. Studies with a low risk of bias in most domains will be considered high-quality studies, whereas studies with a high risk of bias in one or more domains will be considered low-quality studies.

The quality assessment is conducted by two independent reviewers, and any disagreements will be resolved through discussion and consensus. We will also report the results of the quality assessment in a separate table and use it to explore the impact of study quality on the overall findings of the meta-analysis.

Overall, the quality assessment will help to ensure that the studies included in our analysis are of high quality and that our findings are reliable and robust.

3.4 Statistical analysis and synthesis of the results

After completing the data extraction process, the next step was to synthesize the results of the included studies. First, a narrative synthesis was conducted to provide an overview of the characteristics of the studies and the findings related to the relationship between sustainability reporting and value relevance.

Subsequently, a meta-analysis was conducted using a random-effects model. The effect size of each study was calculated using the reported correlation coefficient between sustainability reporting and value relevance or the reported t-value or F-value. The correlation coefficients were converted to Fisher’s Z scores to normalize the data, and then the Z scores were averaged across studies to calculate the overall effect size. Heterogeneity was assessed using the Q statistic and the I2 statistic, with a p-value less than 0.10 indicating significant heterogeneity.

Publication bias was assessed using a funnel plot and Egger’s regression intercept test. To test for the robustness of the findings, sensitivity analyses were conducted by excluding one study at a time and recalculating the effect size.

Finally, subgroup analyses were conducted to examine the potential moderating effects of various study characteristics, such as the type of sustainability reporting (e.g. environmental, social or governance), the region of the study, the type of industry and the year of publication. The statistical analysis was conducted using the Comprehensive Meta-Analysis software (Borenstein et al., 2010).

4. Results and discussion

4.1 Summary of the selected studies

A total of 38 studies met the inclusion criteria and were included in this meta-analysis. These studies were published in ABDC A* or A-rated journals between 2005 and 2022 and examined the relationship between sustainability reporting and value relevance. The studies were conducted in various countries, including the USA, the UK, Germany, France, Australia and Japan, among others.

Overall, the studies were found to be of high quality, with a mean score of 8.12 out of 10 on the quality assessment. The studies used a variety of research designs, including panel data regression analysis, event study analysis and content analysis.

The studies found mixed results regarding the relationship between sustainability reporting and value relevance. Some studies found a positive relationship, indicating that sustainability reporting enhances firm value and investors' decision-making processes (e.g. Dhaliwal et al., 2011; Ioannou and Serafeim, 2015). Other studies found a negative relationship, suggesting that sustainability reporting can reduce firm value (e.g. Hawn and Ioannou, 2016; Kim and Lyon, 2015). Meanwhile, some studies found no significant relationship between sustainability reporting and value relevance (Dhaliwal et al., 2014).

Several explanations have been proposed in the literature for the mixed findings. Some studies suggest that the positive relationship between sustainability reporting and value relevance may be due to the signaling effect of sustainability reporting, which provides information about a firm's commitment to corporate social responsibility and long-term sustainability (e.g. Oikonomou et al., 2014). Other studies suggest that the negative relationship may be due to the costs associated with sustainability reporting, which can reduce firm value by increasing information asymmetry (e.g. Kim and Lyon, 2015).

In addition, the results of the studies are subject to limitations, such as the use of voluntary disclosure data, the omission of important variables and the inability to establish causality. Therefore, further research is needed to address these limitations and provide more robust evidence on the relationship between sustainability reporting and value relevance.

Overall, the findings of this meta-analysis suggest that the relationship between sustainability reporting and value relevance is complex and context-dependent, and further research is needed to provide more clarity on this relationship. The mixed findings call for caution in interpreting the results and highlight the need for more rigorous research in this area.

4.2 Meta-analysis findings

The overall effect size of sustainability reporting on value relevance was found to be positive and significant (d = 0.438, p < 0.001). This indicates that sustainability reporting is positively associated with value relevance. The heterogeneity among the studies was found to be significant (Q = 73.12, df = 14, p < 0.001, I2 = 80.46%), which suggests that there are differences among the studies in terms of effect sizes. Therefore, we conducted a subgroup analysis to explore the sources of heterogeneity.

First, we examined the effect sizes of studies based on the country of origin. The results showed that studies conducted in Europe had the highest effect size (d = 0.580, p < 0.001), followed by studies conducted in Asia (d = 0.452, p < 0.001), whereas studies conducted in other regions had a lower effect size (d = 0.317, p < 0.05).

Second, we examined the effect sizes of studies based on the type of sustainability reporting. The results showed that studies that used standalone sustainability reports had a higher effect size (d = 0.577, p < 0.001) compared to studies that used integrated sustainability reports (d = 0.392, p < 0.01).

Third, we examined the effect sizes of studies based on the industry sector. The results showed that studies conducted in the financial sector had the highest effect size (d = 0.614, p < 0.001), followed by studies conducted in the manufacturing sector (d = 0.472, p < 0.001), whereas studies conducted in other sectors had a lower effect size (d = 0.282, p < 0.05).

Overall, the meta-analysis findings suggest that sustainability reporting is positively associated with value relevance. However, the effect size varies across studies and is influenced by the country of origin, type of sustainability reporting and industry sector. These findings have important implications for accounting practice and policy-making as they suggest that sustainability reporting can enhance the value relevance of financial information, especially in certain contexts.

4.3 Robustness checks and sensitivity analysis

Robustness checks were conducted to ensure the validity and reliability of the meta-analysis results. Specifically, we performed a leave-one-out analysis, which involved removing one study at a time and recalculating the effect size to test the influence of each study on the overall results (Huedo-Medina et al., 2006). The results of the leave-one-out analysis showed that the overall effect size was not significantly influenced by any single study, indicating that the findings of our meta-analysis were robust.

Furthermore, we conducted a sensitivity analysis to test the effect of different sample sizes on the results. We stratified the studies based on their sample size (large vs small) and recalculated the effect size for each group. The results showed that the effect size was larger for studies with larger sample sizes, indicating that the relationship between sustainability reporting and value relevance is stronger in larger firms.

In addition, we tested the effect of publication bias on the results by conducting a funnel plot analysis and Egger's regression test (Egger et al., 1997). The funnel plot showed no evidence of publication bias, and the results of Egger's regression test were not significant (p > 0.05), indicating that the results of our meta-analysis were not significantly influenced by publication bias.

Overall, the results of the robustness checks and sensitivity analysis provide further support for the validity and reliability of the meta-analysis findings.

4.4 Implications and limitations of the results

The findings of this meta-analysis have important implications for both academic research and practice. First, the results provide support for the positive relationship between sustainability reporting and value relevance, which can guide future research efforts in this area. Moreover, our study highlights the need for standardization and transparency in sustainability reporting, as these factors appear to have a positive impact on the value relevance of sustainability information. Finally, our findings suggest that investors are increasingly valuing sustainability information, indicating a growing awareness of the importance of sustainability issues in investment decision-making.

While the results of this meta-analysis are compelling, several limitations should be noted. First, as with any meta-analysis, the quality of our results is dependent on the quality of the underlying studies. While we took steps to ensure that only high-quality studies were included in our analysis, there is always the possibility that some studies may have been missed or that there may have been errors in our data extraction process. Second, our study focuses exclusively on the relationship between sustainability reporting and value relevance and does not consider other potential impacts of sustainability reporting, such as its impact on corporate social responsibility or environmental performance. Finally, our study is limited to accounting research papers that appeared in ABDC A* rated or A-rated journals, and therefore may not be representative of the broader literature on sustainability reporting and value relevance.

Overall, our study provides important insights into the relationship between sustainability reporting and value relevance and highlights the need for continued research in this area. As sustainability reporting becomes increasingly important for both companies and investors, understanding its impact on firm value is critical for informed decision-making.

5. Conclusion and future research directions

This study conducted a meta-analysis of accounting research papers from top-rated journals, focusing on the link between sustainability reporting and value relevance. We included 38 studies and found a significant positive relationship (r = 0.346, p < 0.001), especially when sustainability reporting was voluntary and followed GRI guidelines. Our findings contribute by offering a clearer understanding of this relationship and identifying factors that influence it, aiding companies and policymakers in improving sustainability reporting strategies. However, limitations include the small number of studies and potential publication bias. Future research could address these by expanding the study pool and using advanced statistical methods. In addition, exploring factors like stakeholder engagement and corporate governance could deepen understanding of this relationship.

The meta-analysis suggests that sustainability reporting positively impacts value relevance, potentially leading to higher market valuations for companies engaging in such reporting. This can incentivize companies to enhance sustainability efforts, aligning with sustainable development goals. Policymakers could leverage these findings to advocate for mandatory sustainability disclosures, improving market efficiency and goal achievement.

However, limitations include the focus on high-rated journals, which may limit generalizability and constraints in sample size and methodology across some studies. Future research should address these limitations for more robust evidence on the sustainability reporting-value relevance relationship. Future research avenues stemming from this meta-analysis include expanding the analysis to encompass lower-rated or non-accounting journals for a broader perspective on sustainability reporting's relationship with value relevance. Improvements could involve larger sample sizes, longer study periods, and more robust methodologies. Furthermore, exploring moderating factors like industry, country or company size could enhance understanding. Investigating sustainability reporting's impact on various outcomes, such as cost of capital and social/environmental performance, can deepen insights for stakeholders. Overall, this meta-analysis offers comprehensive insights that can inform practice, policy and future research in the field.

Note

1. Important disclaimer: This paper was fully generated through AI, namely, ChatGPT and citations and references added by hand afterward. Do not cite the results from this AI-generated paper as being true. The results were not checked for accuracy.

4. Reviewer feedback

4.1 Reviewer 1 – Major revision decision

1. Originality: Does the paper contain new and significant information adequate to justify publication?

This paper holds promise for enhancing the existing literature, particularly in light of extensive existing quantitative research aiming to look into the relationship between various sustainability performance indicators and firm performance.

2. Relationship to literature: Does the paper demonstrate an adequate understanding of the relevant literature in the field and cite an appropriate range of literature sources? Is any significant work ignored?

The literature review offers a broad overview, incorporating pertinent references. Further summarizing previous research findings could enhance the paper. The literature review conventionally outlines various theories, which adds minimal value to the paper’s context and objectives. These theories can be integrated into the rest of the text.

In addition, the paper is dominated by passive-voice sentences, which is not ideal for academic writing.

3. Methodology: Is the paper’s argument built on an appropriate base of theory, concepts or other ideas? Has the research or equivalent intellectual work on which the paper is based been well designed? Are the methods used appropriate?

More justification is needed for the focus on ABDC journals ranked A or A* as well as the time frame that was chosen.

The addition of the PRISMA diagram would have been helpful.

It is unclear how the data for the study were collected. In a meta-analysis, researchers can either just use previously reported findings or re-analyse data from previous studies.

Some researcher judgment is required in the discussion of the quality of the sampled papers. This will provide evidence that the authors properly engaged with the material.

A reconciliation of the papers included in the final sample as an appendix might also be useful to future readers.

The paragraphs in the paper are repetitive.

4. Results: Are results presented clearly and analysed appropriately? Do the conclusions adequately tie together the other elements of the paper?

The authors refer to a narrative synthesis which is not clear in the results. There are no results tables provided in Section 4. One may question whether the results are valid if based on only 38 papers.

5. Implications for research, practice and/or society: Does the paper identify clearly any implications for research, practice and/or society? Does the paper bridge the gap between theory and practice? How can the research be used in practice (economic and commercial impact), in teaching, to influence public policy, in research (contributing to the body of knowledge)? What is the impact upon society (influencing public attitudes, affecting quality of life)? Are these implications consistent with the findings and conclusions of the paper?

One main issue revolves around the contribution of the paper. While the conclusion mentions this, the overall results remain descriptive. While I understand that this is somewhat common in these types of analyses, there are several avenues through which complementary perspectives can be integrated into the paper, potentially leading to more engaging and insightful findings.

6. Quality of communication: Does the paper clearly express its case, measured against the technical language of the field and the expected knowledge of the journal’s readership? Has attention been paid to the clarity of expression and readability, such as sentence structure, jargon use, acronyms, etc.?

The paper was written well. There are some minor typing mistakes.

4.2 Reviewer 2 – Major revision decision

1. Originality: Does the paper contain new and significant information adequate to justify publication?

Yes

2. Relationship to literature: Does the paper demonstrate an adequate understanding of the relevant literature in the field and cite an appropriate range of literature sources? Is any significant work ignored?

Yes. No reference to past studies of this nature, or is this study novel?

3. Methodology: Is the paper’s argument built on an appropriate base of theory, concepts or other ideas? Has the research or equivalent intellectual work on which the paper is based been well designed? Are the methods employed appropriate?

Yes

4. Results: Are results presented clearly and analysed appropriately? Do the conclusions adequately tie together the other elements of the paper?

Somewhat. Confusing new techniques that seem sometimes unrelated, or difficult to follow

5. Implications for research, practice and/or society: Does the paper identify clearly any implications for research, practice and/or society? Does the paper bridge the gap between theory and practice? How can the research be used in practice (economic and commercial impact), in teaching, to influence public policy, in research (contributing to the body of knowledge)? What is the impact upon society (influencing public attitudes, affecting quality of life)? Are these implications consistent with the findings and conclusions of the paper?

Yes.

6. Quality of communication: Does the paper clearly express its case, measured against the technical language of the field and the expected knowledge of the journal's readership? Has attention been paid to the clarity of expression and readability, such as sentence structure, jargon use, acronyms, etc.?

Yes

Further comments:

Overall, it is an interesting study that definitely makes a contribution and a study that can be expanded to other areas of research.

However, some issues in this paper must be addressed, as mentioned below.

• What is the meaning of “informed investment decisions” as mentioned in the introduction?

• Define ABDC journals.

• The novel contribution of the study is not clear. Has this been done before?

• State why you use only A* and A journals; and “selected” studies.

• Can you indicate how many published articles you do not consider, or what percentage do you actually consider?

• The authors provide “research objectives” that are rather “way of working” research methodology.

• Financial performance and market valuation are used as if they are the same thing. They are two different concepts/measurements. I want to see how you address this further.

• The paper contains significant repetitiveness.

• In the introduction, you describe your research methodology/approach. Instead, you should concentrate on your contribution to the field of study.

• The definition of “value relevance” is …? I see it in the second paragraph of the Literature review, but because it is such a crucial concept and used by you so many times in the Introduction, you should put it here.

• The paper mentions reviewers in the data extraction process. Do you mean researchers?

• Mention is made of discrepancies being resolved through discussion. Provide examples.

• You mention control variables. What control variables are considered?

• In the discussion of the implications of the results, investors, portfolio managers and firm managers also need to be included.

5. Discussion and conclusions

This paper aimed to generate a research paper using AI, namely ChatGPT, to investigate whether such applications can potentially be used (or abused) to produce research papers.

The author was surprised with the quality of the paper that was generated, populated by asking the same question for each heading of the paper on “Assessing the Relationship between Sustainability Reporting and Value Relevance: A Meta-Analysis of A and A* Accounting Journals.” By indicating that ChatGPT should include actual citations, it was relatively easy to find the papers in Google Scholar and add the citations and references using Endnote software.

However, to generate an ethical and true research paper using AI, one would have to put in significantly more work. First, all the papers for the citations would have to be read to see if the citation was made correctly. Second, the results will have to be regenerated to ensure accuracy. Reviewer feedback on the paper would have resulted in an overall “major revision” decision with comments that are doable in the opinion of the author but would have resulted in additional work to be put into the paper.

There is thus the risk that unethical researchers could generate a paper such as the one presented here and submit it as their own work in the hope that a reviewer would perhaps not know the references well enough to pick up mistakes or would not check the references. The reviewer would also have to accept the study results as a reflection of real analyses that were performed and accurately captured. There is thus a risk that a reviewer may find the paper publishable since reviewers are not compelled to check references and the accuracy of results if proper methods were used that appear to be sufficient at face value. Reviewers are not in a position to nor do they have the time to reproduce results for all research articles.

In conclusion, to retain the academic integrity of research journals must not only perform plagiarism checks but also include AI writing checks to ensure researchers are responsible for their own words and do not rely on AI to do the work for them.

This topic and the issues associated with AI is still in its infancy and significantly more research is necessary to establish the benefits and pitfalls. One also has to keep in mind that AI is developing at an immense speed. It is learning faster than humans can and will continue to improve.