Editorial

Citation

Hayes, L. (2019), "Editorial", Managerial Auditing Journal, Vol. 34 No. 8, pp. 865-870. https://doi.org/10.1108/MAJ-09-2019-019

Data are more than numbers. It also includes textual communications, reports, social media posts, images, spoken word data and videos. Accounting and audit research increasingly analyzes textual data in many different ways to investigate a wide range of research questions (Fisher, 2018; Fisher et al., 2016; Loughran and McDonald, 2016; Teoh, 2018; Vasarhelyi et al., 2015; Zhang et al., 2019), as illustrated by the following four studies in this eBook:

• Chang and Stone (2019a) examine the association between auditor selection and audit proposal readability, measuring readability using a simple linear combination of counts of syllables, words and sentences in the proposals;

• Chang and Stone (2019b) study the association between audit firm size and report readability using natural language processing (NLP) measures that are new to audit literature;

• Boskou et al. (2019) use supervised machine learning to develop a proxy for internal audit quality based on textual disclosures extracted from management’s discussion and analysis (MD&A); and

• Zorio-Grima and Carmona (2019) use both manual content analysis and a variety of automated textual analysis approach to study transparency reports issued by Big 4 audit firms.

These studies demonstrate that quantitative approaches may be used to derive insights from textual data once the text has been appropriately transformed. Machine learning and data visualization tools commonly associated with “Big data” studies are frequently used for textual analysis, even when sample sizes are small. As discussed below, the four articles in this eBook lay bare both the opportunities for adopting text analytic techniques and the challenges of preparing textual data and understanding the models used.

Data transformation

Once textual data has been parsed, filtered and structured, it may be analyzed with techniques and software tools commonly used to analyze numeric data. For example, the machine learning algorithms used by Boskou et al. (2019) to classify internal control quality based on textual disclosures include algorithms used in financial restatement, going concern and financial statement fraud studies that use numerical data (Chye Koh and Kee Low, 2004; Dutta et al., 2017; Kirkos et al., 2007; Perols, 2011). All four articles in this issue detail the transformation (preprocessing) actions commonly used so that textual data may be analyzed quantitatively. These actions include extracting relevant portions of text (e.g. ignoring diagrams, charts and boilerplate language) and then applying additional preprocessing steps that depend upon the analytic methods and tools used. For example, in this issue, Boskou et al. (2019) and Zorio-Grima and Carmona (2019) describe transforming text into a matrix that can be analyzed using statistical software and machine learning techniques. Transformations used in other studies range from straight-forward keyword or dictionary-based word counts in disclosure studies (Campbell et al., 2014) to vector-based measures in studies that compare document similarity (Brown and Tucker, 2011).

The success of analytics depends on data quality. Both the integrity of the data and the extent of data cleaning required to deal with outliers and missing data depend on the data source. Sources of textual data such as auditors’ reports, MD&A, and news articles from reputable journals will require less cleaning effort than customer social media comments (Applebaum et al., 2017). Loughran and McDonald (2016, p. 1192) elaborate on the challenges of transforming textual data for quantitative analysis and caution that “we must be careful that the imprecision of the method does not overwhelm any hoped-for gains in identifying meaning.” Zorio-Grima and Carmona (2019) demonstrate that both automated textual analytics and manual content analysis may be used in the same paper to address this challenge, i.e. they adopt a “mixed, partially automated content analysis” approach that is used in many textual analysis papers (Zhang et al., 2019, p. 151).

Democratization of textual analytics

Many of the analytic methods used today were developed over half a century ago (CPA/AICPA, 2018; Loughran and McDonald, 2016). Today, the increase in computing power and reduced cost of storage have led to the development of software to clean, structure, and analyze data on desktop, laptop, distributed computing and cloud computing platforms. For example, NLP software tools automate many of the preprocessing steps, although word lists used by some NLP software to parse and filter text (e.g. lists of common words such as “and” and “the” that are automatically removed and lists used for part-of-speech tagging) may need to be tailored (Guan et al., 2018). Consistent with these shifts in technology, Zhang et al. (2019) report that there has been a shift away from traditional content analysis to more sophisticated machine learning text analytic approaches. It is noteworthy that some machine learning algorithms (e.g. multi-layered neural networks) used to model non-linear relationships (Chye Koh and Kee Low, 2004) may require little data preprocessing (Issa et al., 2016) as these complex, non-statistical methods are less sensitive to outliers, correlated variables, and data sets with fewer observations than variables, i.e. data distributions commonly encountered when working with textual data.

Studies presented in this eBook use software tools to analyze textual data. Boskou et al. (2019) chose RapidMiner Studio, Chang and Stone (2019b) Coh-Metrix (a computational linguistic analysis tool) and Zorio-Grima and Carmona (2019) text mining and latent semantic analysis packages developed for R. While RapidMiner Studio and R run on personal computers, the Coh-Metrix software is a web-based tool accessed via the internet. Many NLP and machine learning data analytic studies use tools such as R and Python that require programming skills. However, consistent with the evolution of other in-demand technologies, fewer programming skills are required as analytic tools improve (Alles, 2015). The analytic packages available for Python and R are rapidly evolving. So are commercial vendors’ connections to these open source tools and other cloud-based text analytics tools. Today, more and more solutions are being built with less help from data scientists (www.gartner.com/en/newsroom/press-releases/2019-02-18-gartner-identifies-top-10-data-and-analytics-technolo). Given the fast-paced changes in the analytic tools landscape, time is well spent reviewing current software options, as software tools’ functionality and ease of use vary widely.

Big data techniques and small sample studies

“Big data techniques can also be applied to traditional, smaller data sets to gain additional insights” (Gepp et al., 2018, p. 108). This is particularly relevant for textual analysis, as a small collection of documents is also a big collection of words and phrases (Zhang et al., 2019). Thus, data analytics techniques used to search for patterns, make predictions, and draw insights from larger data sets may prove useful for analyzing small samples of textual data. In this issue, machine learning algorithms were used to analyze 28 transparency reports (Zorio-Grima and Carmona, 2019) and data from 133 publicly listed Greek companies (Boskou et al., 2019). Issa et al. (2016) provide examples of ways textual analysis is being used today (i.e. for analyzing contracts and reviewing documents), cites recent textual analysis audit research, and discusses how textual data might be combined with data extracted from accounting information systems to automate audit tasks in the future.

Understanding and explaining models

Researchers need to understand the intuition underlying the models they adopt, including machine learning models that show increasing promise for audit analytics (Applebaum et al., 2017). Depending upon the accounting and audit literature survey, the greatest proportion of automated textual analysis studies are readability studies (Fisher, 2018) or sentiment studies (Zhang et al., 2019). Most of the readability studies use simple, easy-to-understand formulae based on counts of the number of words in sentences and syllables in words (Chang and Stone, 2019a; Lehavy et al., 2011; Li, 2008). Similarly, sentiment studies rely on counts of occurrences of words on lists associated with certain emotions. In contrast, other analytic models are complex. In this issue, Chang and Stone (2019b) use computational linguistic techniques to measure the co-relations of words, sentences, and paragraphs on multi-dimensions and Boskou et al. (2019) use algorithms developed using machine learning. Frequently text analytic studies use multiple models because, as Boskou et al. (2019) explain, the incremental costs of running multiple models once data has been cleaned and structured are small. When multiple models are used, model choice should include considerations of not only model fit but also model interpretability (Hall et al., 2017). Those who do not understand the intuition underlying models may draw unsupported conclusions from complex analytics reported by easy-to-use analytic software. The articles in this eBook explain the intuition underlying some complex machine learning algorithms (Boskou et al., 2019; Zorio-Grima and Carmona, 2019) and NLP techniques (Chang and Stone (2019a, 2019b)).

Biases

Algorithms developed by artificial intelligence (AI) will be biased if based on non-representative data. Furthermore, decision-makers bring cognitive biases (e.g. anchoring, availability and confirmation bias) to the interpretation of model results (Huerta and Jensen, 2017). Microsoft recognizes the risk of bias in AI processes and warned of this in its annual SEC filing in August 2018:

AI algorithms may be flawed. Data sets may be insufficient or contain biased information. Inappropriate or controversial data practices by Microsoft or others could impair the acceptance of AI solutions. These deficiencies could undermine the decisions, predictions or analysis AI applications produce, subjecting us to competitive harm, legal liability and brand or reputational harm (Simonite, 2019).

When adopting text analytic approaches that rely on AI processes, changes in business processes and societal trends that affect data distributions should be considered. Disclosures and language usage change over time.

Data visualization

A picture is worth a thousand words. Visualizations, along with descriptive statistics, are used to explore data, reveal patterns and identify data items needing attention. For most textual analysis projects, scanning data manually for outliers, errors and missing data are impractical. Visualization software can reduce multi-dimensional data to a visual image that humans can intuitively grasp. For example, word clouds can depict word frequencies (Zorio-Grima and Carmona, 2019) and the correlation of hundreds of words can be displayed in two dimensions on a network graph by showing their correlations as the distance between data points that represent different words, the size of which shows the word frequencies (Hall et al., 2017).

In contrast, to the straight-forward visualizations produced by spreadsheet software (i.e. bar graphs, trend lines, etc.), machine learning models are frequently needed to reduce a high-dimension text data sets to two- or three-dimension visualizations. In their seminal article, Hinton and Salakhutdinov (2006, p. 507) declare that models that reduce dimensionality to facilitate visualization have:

Been obvious since the 1980s […] provided that computers were fast enough, data sets were big enough and the initial weights were close enough to a good solution. All three conditions are now satisfied.

Accordingly, visualizations such as those used by Zorio-Grima and Carmona (2019) to illustrate the relationships between narrative disclosures in transparency reports are beginning to appear in audit and accounting literature. Data visualizations need to be carefully prepared and used cautiously as decision-makers draw conclusions swiftly from images and are subject to cognitive biases that affect their interpretation of visualizations (Dilla et al., 2010; Rose et al., 2016).

Conclusion

Adoption of automated textual analysis is accelerating as data analytic skills improve and easier-to-use data preparation and analysis tools become more readily available. The articles in this eBook not only contribute to audit literature but also demonstrate the promise of text analytics for future research. However, there is a risk that the inferences and conclusions drawn from text analytics may be unfounded when these tools are used by those unfamiliar with the challenges of preprocessing textual data, evaluating and explaining advanced models, and interpreting data visualizations. Collectively, the articles in this eBook invite the reader to make greater and more informed use of textual analysis in the future.