SardiStance

stance detection in Italian tweets sardistance@evalita 2020

SardiStance is the first shared task regarding Stance Detection in Italian tweets. We have collected tweets written in Italian about the Sardines movement, and we invite at automatically detecting their stance.

The task will be organized within EVALITA 2020, the 7th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian, which will be held in Bologna, Italy, co-located with CLiC-it 2020, from November 30th, 2020 until December 3rd, 2020.

Organizers

Mirko Lai1

Alessandra T. Cignarella1,2

Cristina Bosco1

Viviana Patti1

Paolo Rosso2

1. Dipartimento di Informatica, Università degli Studi di Torino, Italy

2. PRHLT Research Center, Universitat Politècnica de València, Spain

Credits

Sergio Rabellino

ICT Staff, Dipartimento di Informatica, Università degli Studi di Torino, Italy


Introduction and Motivation

Recently, a special interest in the task of monitoring people's stance towards particular targets has grown; thus leading to the creation of a novel area of investigation named Stance Detection (SD). Research on this topic has an impact on different aspects such as public administration, policy-making, marketing strategies and security. In fact, through the constant monitoring of people's opinion, desires, complaints and beliefs on political agenda or public services, administrators could better meet population's needs. For example, a practical application of SD could improve the automatic identification of people's extremist tendencies or understand and prevent citizens' dissatisfaction and frustration.

Stance Detection (target-specific stance classification) is the task of automatically determining whether the author of a text is in favor, against, whether neither inference is likely or neutral/none towards a given target. The first shared task on SD was indeed held for English at SemEval in 2016, i.e. Task 6 "Detecting Stance in Tweets" (Mohammad et al., 2016). It consisted in detecting the orientation in favor or against six different targets of interest: "Hillary Clinton", "Feminist Movement", "Legalization of Abortion", "Atheism", "Donald Trump", and "Climate Change is a Real Concern". A more recent evaluation for SD systems was proposed at IberEval 2017 for both Catalan and Spanish (Taulé et al., 2017) where the target was only one, i.e. "Independence of Catalonia". The same organizers, proposed a similar shared task the following year at the evaluation campaign IberEval 2018 regarding the target "Catalan first of October Referendum" (Taulé et al., 2018) and encouraging Stance Detection with multimodal approaches.

Another type of stance classification, more general-purpose, is the open stance classification task, usually indicated with the acronym SDQC, by referring to the four categories exploited for indicating the attitude of a message with respect to a rumour: Support (S), Deny (D), Query (Q) and Comment (C) (Aker et al., 2017). Two shared tasks of this type were organized in the last years at SemEval in the 2017 and in the 2019 editions, considering tweets along with Reddit posts in English (Derczynski et al., 2017, Gorrell et al., 2019).


Target Audience

The task is open to everyone from industry and academia. We in particular encourage the participations of researchers, industrial teams, and students.


Task Description

With this task proposal we would like to invite participants to explore features based on the textual content of the tweet, such as structural, stylistic, and affective features, but also features based on contextual information that documents not emerge directly from the text, such as for instance knowledge about the domain of the political debate or information about the user's community.
Overall, we propose two different subtasks:

Task A - Textual Stance Detection:

The first task is a three-class classification task where the system has to predict whether a tweet is in favour, against or neutral/none towards the given target (following the guidelines below), exploiting only textual information, i.e. the text of the tweet.

Task B - Contextual Stance Detection:

The second task is the same as the first one: a three-class classification task where the system has to predict whether a tweet is in favour, against or neutral/none towards the given target. Here participants will have access to a wider range of contextual information based on the post such as: the number of retweets, the number of favours, the type of posting source (e.g. iOS or Android), and date of posting.

Furthermore we will share (and encourage its exploitation) contextual information related to the user, such as: number of tweets ever posted, user's bio (only emojis), user's number of followers, user's number of friends. Additionally we will share users' contextual information about their social network, such as: friends, replies, retweets, and quotes' relations.
The personal ids of the users will be anonymized but their network structures will be maintained intact.


Data

The dataset will include short documents taken from Twitter. From May 29th it will be possible to download it from here: data repository. The files are encrypted. Please register to our Google Group and compile this Google Form regarding Copyright Agreement, to obtain the passwords.


Evaluation

Each participating team will initially have access to the training data only. Later, the unlabelled test data will be released (see the timeframe below). After the assessment, the labels for the test data will be released as well.

The evaluation will be performed according to the standard metrics known in literature (accuracy, precision, recall and F1-score). The submissions will be ranked by F1-score (precision, recall and F-measure). Details on evaluation metrics to be applied for the evaluation of the participant results will be published in the Task guidelines soon.

How To Partecipate

Register your team by using the registration web form at http://www.evalita.it/2020 (available soon, see timeframe below).

Information about the submission of results and their format will be available in the Task guidelines.

We invite the potential participants to subscribe to our Google Group in order to be kept up to date with the latest news related to the task. Please share comments and questions with the mailing list. The organizers will assist you for any potential issues that could be raised.

Participants will be required to provide an abstract and a technical report including a brief description of their approach, an illustration of their experiments, in particular techniques and resources used, and an analysis of their results for the publication in the Proceedings of contest.

Important Dates

29th May 2020: training data available (data repository)

4th September 2020: test data available

24th September 2020: systems results due to the organizers

2nd October 2020: assessment returned to participants

6th November 2020: technical reports due to organizers (camera-ready)

1st or 2nd December 2020: final workshop in Bologna

Contact Us

Write to us HERE!

Or contact the two main organizers: Mirko Lai, mirko.lai@unito.it and Alessandra T. Cignarella, cigna@di.unito.it.