EMit@Evalita 2023

The EMit (Emotions in Italian) shared task will be organized within Evalita 2023, the 8th evaluation campaign of Natural Language Processing and Speech tools for Italian, which will be held in Parma on the 7th-8th September 2023.

Introduction and motivation

The detection of emotions in texts has a long history in international evaluation campaigns but has never been addressed in EVALITA where the only shared task to deal with emotions was about emotional speech recognition systems [7]. The Affective Text shared task at SemEval 2007 was the first one to propose the classification of newspaper headlines according to 6 emotions (Anger, Disgust, Fear, Joy, Sadness, Surprise) [10] then, starting from 2017, this type of evaluation has become very frequent with a particular attention to the processing of tweets and dialogues. For example, Affect in Tweets at SemEval 2018 [6] included a subtask about the multilabel detection of 11 emotions in tweets written in English, Arabic and Spanish whereas the Emotion Detection task at TASS 2020 [11] and EmoEvalEs at IberLEF 2021 were only on Spanish tweets; instead, EmoContext at SemEval 2019 [3] and EmotionX at the SocialNLP workshop in 2018 [5] and 2019 focused on the emotion classification of dialogues in English. Last year, the Emotion Classification shared task at WASSA 2022 dealt with a different genre of text proposing the classification of emotions in essays written in reaction to news articles [1]. In this context, the EMit (Emotions in Italian) task aims at providing the first evaluation framework for emotion detection in Italian texts and make new annotated data available to the community.

The main proposed subtask is the detection of emotions in social media messages about TV shows, TV series, music videos and advertisements. Possible emotional labels are the 8 main emotions defined by Plutchik [8] (anger, anticipation, disgust, fear, joy, sadness, surprise, trust), and the additional label “love” that is one of the primary dyads in Plutchik's wheel of emotions.

Considering the specific attention on the entertainment sector, we designed an auxiliary subtask for identifying also if the target of the affective comments about tv programs/series, music videos and advertisement is related to the topic or to issues under control of the direction. The message could be classified as addressing the topic, or the direction (or both or neither). The subtask has been designed specifically on the events and players involved in such contents and in their creation. Such finer grained information can be of great importance in real application domains, for artists or broadcasters in the evaluation of the contents delivered, when the analysis of emotions in social media is used as a social signal of emotional reactions of the Italian audience.

Task description

EMit is organized according to two subtasks, both designed as multilabel classification problems:

  • Task A: Categorial Emotion Detection (required): given a text, the system decides the emotions expressed in it or the absence of emotions. In other words, the text could be classified as neutral, or expressing one or more of the 8 basic emotions defined by Plutchik [8] (anger, anticipation, disgust, fear, joy, sadness, surprise, trust) plus the additional emotion “love” that is one of the primary dyads in the Plutchik’s wheel of emotions.
  • Task B: Target Detection (optional): given a text, the system decides what is the target addressed by the author of the text. The text could be classified as addressing the topic, or the direction, or both or neither.

Important dates

  • 7th February 2023: development data available to participants for both tasks
  • 30th April 2023: registration closes
  • 2nd-19th May 2023: evaluation window and collection of participants’ results
  • 30th May 2023: assessment returned to participants
  • 14th June 2023: final reports due to task organizers
  • 25th July 2023: camera ready version deadline
  • 7th-8th September 2023: final workshop in Parma

Available resources

dataset

lexica

References

[1] Valentin Barriere, Shabnam Tafreshi, João Sedoc, and Sawsan Alqahtani. WASSA 2022 shared task: Predicting empathy, emotion and personality in reaction to news stories. In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, pages 214–227, Dublin, Ireland, May 2022. Association for Computational Linguistics.

[2] Federico Bianchi, Debora Nozza, and Dirk Hovy. FEEL-IT: Emotion and sentiment classification for the Italian language. In Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 76–83, Online, April 2021. Association for Computational Linguistics.

[3] Ankush Chatterjee, Kedhar Nath Narahari, Meghana Joshi, and Puneet Agrawal. SemEval-2019 task 3: EmoContext contextual emotion detection in text. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 39–48, Minneapolis, Minnesota, USA, June 2019. Association for Computational Linguistics.

[4] Francesco Fernicola, Shibingfeng Zhang, Federico Garcea, Paolo Bonora, and Alberto Barrón-Cedeno. Ariemozione: Identifying emotions in opera verses. In Proceedings of CLiC-it. CEUR-WS, 2020.

[5] Chao-Chun Hsu and Lun-Wei Ku. SocialNLP 2018 EmotionX challenge overview: Recognizing emotions in dialogues. In Proceedings of the sixth international workshop on natural language processing for social media, pages 27–31, 2018.

[6] Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. SemEval-2018 task 1: Affect in tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 1–17, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.

[7] Antonio Origlia and Vincenzo Galatà. Evalita 2014: Emotion recognition task (ERT). In Proceedings of the Fourth International Workshop EVALITA 2014, pages 112–115. Pisa University Press, 2014.

[8] Robert Plutchik and Henry Kellerman. Theories of emotion, volume 1. Academic Press, 1980.

[9] Rachele Sprugnoli. Multiemotions-it: A new dataset for opinion polarity and emotion analysis for italian. In 7th Italian Conference on Computational Linguistics, CLiC-it 2020, pages 402–408. Accademia University Press, 2020.

[10] Carlo Strapparava and Rada Mihalcea. SemEval-2007 task 14: Affective text. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 70–74, Prague, Czech Republic, June 2007. Association for Computational Linguistics.

[11] Manuel García Vega, Manuel Carlos Díaz-Galiano, Miguel Ángel García Cumbreras, Flor Miriam Plaza del Arco, Arturo Montejo-Ráez, Salud María Jiménez Zafra, Eugenio Martínez Cámara, César Antonio Aguilar, Marco Antonio Sobrevilla Cabezudo, Luis Chiruzzo, et al. Overview of TASS 2020: Introducing emotion detection. In IberLEF@ SEPLN, 2020.