Abstracts Track 2023

Area 1 - Natural Language Understanding

Nr: 55

An Application of Large Language Model Content Evaluation


Ge Zhan, Wenbo Zhang and Tianyang Zhang

Abstract: AI chatbots have become the fastest-growing intelligent terminals. AI chatbots provide multi-modal intelligent dialogue services through various cues such as voice, text, images, and emoticons, demonstrating the potential of these devices to provide emotional and respectful support, thereby affecting consumer emotions. Despite the recent surge in works on large language models (LLMs), there is a lack of studies with a semantic-analytics approach to evaluate the types of support provided by chatbot. LLM aims to recognize emotions and learn from historical conversations to understand human intentions and provide more human like responses to promote intimate relationships, emotional participation, connectivity and social engagement. The LLM- human interaction indicates that social support is possibly transferred between users and the machine. LLMs have high-level information retrieval and active adaptive learning capabilities, showing their potential in providing information and network support. Therefore, the multiple aspects of social support are worth paying attention to, as they inherently possess communication characteristics and reveal the key role of LLM in behaving like human beings. Most of the existing research on human-computer interaction focuses on the functional aspects of chatbots, with few studies focusing on social factors. A key research question should be how do the content (including emotional expression) generated by LLMs convey emotions to consumers? The paper introduces a semantic content analysis tool for LLM projects, which may facilitate the investigation of human-robot communication by visualizing and understanding speech flows. The web app integrates functions such as CMD terminal, LLM data training, content generation, summarization, analysis, and visualization. The web app has a browser based UI composed of modules. The browser based UI is a single page web application built by Python. In the terminal, the web app is like a LLM server: users can ask questions and view replies from the LLM. The app decodes speech texts into numeric values with the four dimensions of social support (emotional, esteem, information and network support). The research results of this project will help us to further understand the application of LLM and chatbot in supporting users, particularly in exploring user-chatbot interactions in different dimensions of social support. Based on the interaction and content characteristics of LLM, this study assists practitioners to identify the driving factors that affect consumer emotions and behavior, thus formulating corresponding strategies and policies, and generating insights about user emotional and behavioral dynamics. The web app timely adjust the evaluation and control methods of intelligent terminals, which have very important practical significance.