A Wide Evaluation of ChatGPT on Affective Computing Tasks

In the rapidly evolving field of artificial intelligence, foundation models like ChatGPT are emerging as powerful tools for solving a wide range of problems. A recent study by Mostafa M. Amin, Rui Mao, Erik Cambria, and Björn W. Schuller explores the capabilities of ChatGPT models, specifically GPT-4 and GPT-3.5, in tackling 13 affective computing tasks. These tasks range from sentiment analysis and toxicity detection to more complex issues like suicide tendency detection and personality assessment.

The researchers introduced a novel framework to evaluate the ChatGPT models on regression-based problems, such as intensity ranking problems. This was achieved by modelling these problems as pairwise ranking classifications. The framework uses a small-world graph generation algorithm to sample pairs for comparison, and a binary search-like prediction procedure to narrow down the possible range of answers.

The results of the study are fascinating. The models demonstrated emergent abilities in a variety of affective computing problems. GPT-3.5 and GPT-4 showed particularly strong performance in tasks related to sentiment, emotions, and toxicity detection. For instance, they excelled at identifying extremely negative emotions, especially in the context of well-being assessment over long texts and toxicity detection.

However, the study also identified areas where the ChatGPT models fell short. Tasks involving implicit signals, such as engagement measurement and subjectivity detection, proved more challenging for the models. The performance of GPT-3.5 was significantly weaker than end-to-end recurrent neural networks and transformers in most cases, and GPT-4 only slightly surpassed these traditional methods in sarcasm and subjectivity detection problems.

Overall, the study provides valuable insights into the potential and limitations of ChatGPT models in affective computing tasks. While the models show promise in many areas, there is still room for improvement, particularly in tasks involving implicit signals. As AI continues to evolve, studies like this one are crucial in guiding future research and development efforts.

Read the whole article here: http://arxiv.org/abs/2308.13911v1