Search results

Most educational assessments tend to be constructed in a close-ended format, which is easier to score consistently and more affordable. However, recent work has leveraged computation text methods from the information sciences to make open-ended measurement more effective and reliable for older students. The purpose of this study is to determine whether models used by computational text mining applications need to be adapted when used with samples of elementary-aged children.

This study introduces domain-adapted semantic models for child-specific text analysis, to allow better elementary-aged educational assessment. A corpus compiled from a multimodal mix of spoken and written child-directed sources is presented, used to train a children’s language model and evaluated against standard non-age-specific semantic models.

Child-oriented language is found to differ in vocabulary and word sense use from general English, while exhibiting lower gender and race biases. The model is evaluated in an educational application of divergent thinking measurement and shown to improve on generalized English models.

The findings demonstrate the need for age-specific language models in the growing domain of automated divergent thinking and strongly encourage the same for other educational uses of computation text analysis by showing a measurable difference in the language of children.

Understanding children’s language more representatively in automated educational assessment allows for more fair and equitable testing. Furthermore, child-specific language models have fewer gender and race biases.

Research in computational measurement of open-ended responses has thus far used models of language trained on general English sources or domain-specific sources such as textbooks. To the best of the authors’ knowledge, this paper is the first to study age-specific language models for educational assessment. In addition, while there have been several targeted, high-quality corpora of child-created or child-directed speech, the corpus presented here is the first developed with the breadth and scale required for large-scale text modeling.

The increasing importance and proliferation of text data provide a unique opportunity and novel lens to study human communication across a myriad of business and marketing applications. For example, consumers compare and review products online, individuals interact with their voice assistants to search, shop, and express their needs, investors seek to extract signals from firms' press releases to improve their investment decisions, and firms analyze sales call transcripts to increase customer satisfaction and conversions. However, extracting meaningful information from unstructured text data is a nontrivial task. In this chapter, we review established natural language processing (NLP) methods for traditional tasks (e.g., LDA for topic modeling and lexicons for sentiment analysis and writing style extraction) and provide an outlook into the future of NLP in marketing, covering recent embedding-based approaches, pretrained language models, and transfer learning for novel tasks such as automated text generation and multi-modal representation learning. These emerging approaches allow the field to improve its ability to perform certain tasks that we have been using for more than a decade (e.g., text classification). But more importantly, they unlock entirely new types of tasks that bring about novel research opportunities (e.g., text summarization, and generative question answering). We conclude with a roadmap and research agenda for promising NLP applications in marketing and provide supplementary code examples to help interested scholars to explore opportunities related to NLP in marketing.

The rapid rise of large language models (LLMs) has propelled them to the forefront of applications in natural language processing (NLP). This paper aims to present a comprehensive examination of the research landscape in LLMs, providing an overview of the prevailing themes and topics within this dynamic domain.

Drawing from an extensive corpus of 198 records published between 1996 to 2023 from the relevant academic database encompassing journal articles, books, book chapters, conference papers and selected working papers, this study delves deep into the multifaceted world of LLM research. In this study, the authors employed the BERTopic algorithm, a recent advancement in topic modeling, to conduct a comprehensive analysis of the data after it had been meticulously cleaned and preprocessed. BERTopic leverages the power of transformer-based language models like bidirectional encoder representations from transformers (BERT) to generate more meaningful and coherent topics. This approach facilitates the identification of hidden patterns within the data, enabling authors to uncover valuable insights that might otherwise have remained obscure. The analysis revealed four distinct clusters of topics in LLM research: “language and NLP”, “education and teaching”, “clinical and medical applications” and “speech and recognition techniques”. Each cluster embodies a unique aspect of LLM application and showcases the breadth of possibilities that LLM technology has to offer. In addition to presenting the research findings, this paper identifies key challenges and opportunities in the realm of LLMs. It underscores the necessity for further investigation in specific areas, including the paramount importance of addressing potential biases, transparency and explainability, data privacy and security, and responsible deployment of LLM technology.

The analysis revealed four distinct clusters of topics in LLM research: “language and NLP”, “education and teaching”, “clinical and medical applications” and “speech and recognition techniques”. Each cluster embodies a unique aspect of LLM application and showcases the breadth of possibilities that LLM technology has to offer. In addition to presenting the research findings, this paper identifies key challenges and opportunities in the realm of LLMs. It underscores the necessity for further investigation in specific areas, including the paramount importance of addressing potential biases, transparency and explainability, data privacy and security, and responsible deployment of LLM technology.

This classification offers practical guidance for researchers, developers, educators, and policymakers to focus efforts and resources. The study underscores the importance of addressing challenges in LLMs, including potential biases, transparency, data privacy, and responsible deployment. Policymakers can utilize this information to shape regulations, while developers can tailor technology development based on the diverse applications identified. The findings also emphasize the need for interdisciplinary collaboration and highlight ethical considerations, providing a roadmap for navigating the complex landscape of LLM research and applications.

This study stands out as the first to examine the evolution of LLMs across such a long time frame and across such diversified disciplines. It provides a unique perspective on the key areas of LLM research, highlighting the breadth and depth of LLM’s evolution.

This study aims to adopt a systematic review approach to examine the existing literature on law and LLMs.It involves analyzing and synthesizing relevant research papers, reports and scholarly articles that discuss the use of LLMs in the legal domain. The review encompasses various aspects, including an analysis of LLMs, legal natural language processing (NLP), model tuning techniques, data processing strategies and frameworks for addressing the challenges associated with legal question-and-answer (Q&A) systems. Additionally, the study explores potential applications and services that can benefit from the integration of LLMs in the field of intelligent justice.

This paper surveys the state-of-the-art research on law LLMs and their application in the field of intelligent justice. The study aims to identify the challenges associated with developing Q&A systems based on LLMs and explores potential directions for future research and development. The ultimate goal is to contribute to the advancement of intelligent justice by effectively leveraging LLMs.

To effectively apply a law LLM, systematic research on LLM, legal NLP and model adjustment technology is required.

This study contributes to the field of intelligent justice by providing a comprehensive review of the current state of research on law LLMs.

Social media platforms that disseminate scientific information to the public during the COVID-19 pandemic highlighted the importance of the topic of scientific communication. Content creators in the field, as well as researchers who study the impact of scientific information online, are interested in how people react to these information resources and how they judge them. This study aims to devise a framework for extracting large social media datasets and find specific feedback to content delivery, enabling scientific content creators to gain insights into how the public perceives scientific information.

To collect public reactions to scientific information, the study focused on Twitter users who are doctors, researchers, science communicators or representatives of research institutes, and processed their replies for two years from the start of the pandemic. The study aimed in developing a solution powered by topic modeling enhanced by manual validation and other machine learning techniques, such as word embeddings, that is capable of filtering massive social media datasets in search of documents related to reactions to scientific communication. The architecture developed in this paper can be replicated for finding any documents related to niche topics in social media data. As a final step of our framework, we also fine-tuned a large language model to be able to perform the classification task with even more accuracy, forgoing the need of more human validation after the first step.

We provided a framework capable of receiving a large document dataset, and, with the help of with a small degree of human validation at different stages, is able to filter out documents within the corpus that are relevant to a very underrepresented niche theme inside the database, with much higher precision than traditional state-of-the-art machine learning algorithms. Performance was improved even further by the fine-tuning of a large language model based on BERT, which would allow for the use of such model to classify even larger unseen datasets in search of reactions to scientific communication without the need for further manual validation or topic modeling.

The challenges of scientific communication are even higher with the rampant increase of misinformation in social media, and the difficulty of competing in a saturated attention economy of the social media landscape. Our study aimed at creating a solution that could be used by scientific content creators to better locate and understand constructive feedback toward their content and how it is received, which can be hidden as a minor subject between hundreds of thousands of comments. By leveraging an ensemble of techniques ranging from heuristics to state-of-the-art machine learning algorithms, we created a framework that is able to detect texts related to very niche subjects in very large datasets, with just a small amount of examples of texts related to the subject being given as input.

With this tool, scientific content creators can sift through their social media following and quickly understand how to adapt their content to their current user’s needs and standards of content consumption.

This study aimed to find reactions to scientific communication in social media. We applied three methods with human intervention and compared their performance. This study shows for the first time, the topics of interest which were discussed in Brazil during the COVID-19 pandemic.

The study aims to provide concrete service remediation and enhancement for LLM developers such as getting user forgiveness and breaking through perceived bottlenecks. It also aims to improve the efficiency of app users' usage decisions.

This paper takes the user reviews of the app stores in 21 countries and 10 languages as the research data, extracts the potential factors by LDA model, exploratively takes the misalignment between user ratings and textual emotions as user forgiveness and perceived bottleneck and uses the Word2vec-SVM model to analyze the sentiment. Finally, attributions are made based on empathy.

The results show that AI-based LLMs are more likely to cause bias in user ratings and textual content than regular APPs. Functional and economic remedies are effective in awakening empathy and forgiveness, while empathic remedies are effective in reducing perceived bottlenecks. Interestingly, empathetic users are “pickier”. Further social network analysis reveals that problem solving timeliness, software flexibility, model updating and special data (voice and image) analysis capabilities are beneficial in breaking perceived bottlenecks. Besides, heterogeneity analysis show that eastern users are more sensitive to the price factor and are more likely to generate forgiveness through economic remedy, and there is a dual interaction between basic attributes and extra boosts in the East and West.

The “gap” between negative (positive) user reviews and ratings, that is consumer forgiveness and perceived bottlenecks, is identified in unstructured text; the study finds that empathy helps to awaken user forgiveness and understanding, while it is limited to bottleneck breakthroughs; the dataset includes a wide range of countries and regions, findings are tested in a cross-language and cross-cultural perspective, which makes the study more robust, and the heterogeneity of users' cultural backgrounds is also analyzed.

The term knowledge refers to the part of the world investigated by a specific discipline and that includes a specific taxonomy, vocabulary, concepts, theories, research methods and standards of justification. Our approach uses domain knowledge to improve the quality of business process models (BPMs) by exploiting the domain knowledge provided by large language models (LLMs). Among these models, ChatGPT stands out as a notable example of an LLM capable of providing in-depth domain knowledge. The lack of coverage presents a limitation in each approach, as it hinders the ability to fully capture and represent the domain’s knowledge. To solve such limitations, we aim to exploit GPT-3.5 knowledge. Our approach does not ask GPT-3.5 to create a visual representation; instead, it needs to suggest missing concepts, thus helping the modeler improve his/her model. The GPT-3.5 may need to refine its suggestions based on feedback from the modeler.

We initiate our semantic quality enhancement process of a BPM by first extracting crucial elements including pools, lanes, activities and artifacts, along with their corresponding relationships such as lanes being associated with pools, activities belonging to each lane and artifacts associated with each activity. These data are systematically gathered and structured into ArrayLists, a form of organized collection that allows for efficient data manipulation and retrieval. Once we have this structured data, our methodology involves creating a series of prompts based on each data element. We adopt three approaches to prompting: zero-shot, few-shot and chain of thoughts (CoT) prompts. Each type of prompting is specifically designed to interact with the OpenAI language model in a unique way, aiming to elicit a diverse array of suggestions. As we apply these prompting techniques, the OpenAI model processes each prompt and returns a list of suggestions tailored to that specific element of the BPM. Our approach operates independently of any specific notation and offers semi-automation, allowing modelers to select from a range of suggested options.

This study demonstrates the significant potential of prompt engineering techniques in enhancing the semantic quality of BPMs when integrated with LLMs like ChatGPT. Our analysis of model activity richness and model artifact richness across different prompt techniques and input configurations reveals that carefully tailored prompts can lead to more complete BPMs. This research is a step forward for further exploration into the optimization of LLMs in BPM development.

The limitation is the domain ontology that we are relying on to evaluate the semantic completeness of the new BPM. In our future work, the modeler will have the option to ask for synonyms, hyponyms, hypernyms or keywords. This feature will facilitate the replacement of existing concepts to improve not only the completeness of the BPM but also the clarity and specificity of concepts in BPMs.

To demonstrate our methodology, we take the “Hospitalization” process as an illustrative example. In the scope of our research, we have presented a select set of instructions pertinent to the “chain of thought” and “few-shot prompting.” Due to constraints in presentation and the extensive nature of the instructions, we have not included every detail within the body of this paper. However, they can be found in the previous GitHub link. Two appendices are given at the end. Appendix 1 describes the different prompt instructions. Appendix 2 presents the application of the instructions in our example.

In our research, we rely on the domain application knowledge provided by ChatGPT-3 to enhance the semantic quality of BPMs. Typically, the semantic quality of BPMs may suffer due to the modeler's lack of domain knowledge. To address this issue, our approach employs three prompt engineering methods designed to extract accurate domain knowledge. By utilizing these methods, we can identify and propose missing concepts, such as activities and artifacts. This not only ensures a more comprehensive representation of the business process but also contributes to the overall improvement of the model's semantic quality, leading to more effective and accurate business process management.

In the era of big data, various industries are generating large amounts of text data every day. Simplifying and summarizing these data can effectively serve users and improve efficiency. Recently, zero-shot prompting in large language models (LLMs) has demonstrated remarkable performance on various language tasks. However, generating a very “concise” multi-document summary is a difficult task for it. When conciseness is specified in the zero-shot prompting, the generated multi-document summary still contains some unimportant information, even with the few-shot prompting. This paper aims to propose a LLMs prompting for multi-document summarization task.

To overcome this challenge, the authors propose chain-of-event (CoE) prompting for multi-document summarization (MDS) task. In this prompting, the authors take events as the center and propose a four-step summary reasoning process: specific event extraction; event abstraction and generalization; common event statistics; and summary generation. To further improve the performance of LLMs, the authors extend CoE prompting with the example of summary reasoning.

Summaries generated by CoE prompting are more abstractive, concise and accurate. The authors evaluate the authors’ proposed prompting on two data sets. The experimental results over ChatGLM2-6b show that the authors’ proposed CoE prompting consistently outperforms other typical promptings across all data sets.

This paper proposes CoE prompting to solve MDS tasks by the LLMs. CoE prompting can not only identify the key events but also ensure the conciseness of the summary. By this method, users can access the most relevant and important information quickly, improving their decision-making processes.

Advances in artificial intelligence have ushered in new opportunities for marketers in the domain of content generation. We discuss approaches that have emerged to generate text and image content. Drawing on the customer equity framework, we then discuss the potential applications of automated content generation for customer acquisition, relationship development, and customer retention. We conclude by discussing important considerations that businesses must make prior to adopting automated content generation.

This study aims to explore Augmented Language Models (ALMs) for synthetic data generation in services marketing and research. It evaluates ALMs' potential in mirroring human responses and behaviors in service scenarios through comparative analysis with five empirical studies.

The study uses ALM-based agents to conduct a comparative analysis, leveraging SurveyLM (Bickley et al., 2023) to generate synthetic responses to the scenario-based experiment in Söderlund and Oikarinen (2018) and four more recent studies from the Journal of Services Marketing. The main focus was to assess the alignment of ALM responses with original study manipulations and hypotheses.

Overall, our comparative analysis reveals both strengths and limitations of using synthetic agents to mimic human-based participants in services research. Specifically, the model struggled with scenarios requiring high levels of visual context, such as those involving images or physical settings, as in the Dootson et al. (2023) and Srivastava et al. (2022) studies. Conversely, studies like Tariq et al. (2023) showed better alignment, highlighting the model's effectiveness in more textually driven scenarios.

To the best of the authors’ knowledge, this research is among the first to systematically use ALMs in services marketing, providing new methods and insights for using synthetic data in service research. It underscores the challenges and potential of interpreting ALM versus human responses, marking a significant step in exploring AI capabilities in empirical research.