SardiStance is the first shared task regarding Stance Detection in Italian tweets. We have collected tweets written in Italian about the Sardines movement, and we invite at automatically detecting their stance.
The task will be organized within EVALITA 2020, the 7th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian, which will be held **ONLINE on December 16th and 17th, 2020**
in Bologna, Italy, co-located with CLiC-it 2020, from November 30th, 2020 until December 3rd, 2020.
1. Dipartimento di Informatica, Università degli Studi di Torino, Italy
2. PRHLT Research Center, Universitat Politècnica de València, Spain
ICT Staff, Dipartimento di Informatica, Università degli Studi di Torino, Italy
Recently, a special interest in the task of monitoring people's stance towards particular targets has grown; thus leading to the creation of a novel area of investigation named Stance Detection (SD). Research on this topic has an impact on different aspects such as public administration, policy-making, marketing strategies and security. In fact, through the constant monitoring of people's opinion, desires, complaints and beliefs on political agenda or public services, administrators could better meet population's needs. For example, a practical application of SD could improve the automatic identification of people's extremist tendencies or understand and prevent citizens' dissatisfaction and frustration.
Stance Detection (target-specific stance classification) is the task of automatically determining whether the author of a text is in favor, against, whether neither inference is likely or neutral/none towards a given target. The first shared task on SD was indeed held for English at SemEval in 2016, i.e. Task 6 "Detecting Stance in Tweets" (Mohammad et al., 2016). It consisted in detecting the orientation in favor or against six different targets of interest: "Hillary Clinton", "Feminist Movement", "Legalization of Abortion", "Atheism", "Donald Trump", and "Climate Change is a Real Concern". A more recent evaluation for SD systems was proposed at IberEval 2017 for both Catalan and Spanish (Taulé et al., 2017) where the target was only one, i.e. "Independence of Catalonia". The same organizers, proposed a similar shared task the following year at the evaluation campaign IberEval 2018 regarding the target "Catalan first of October Referendum" (Taulé et al., 2018) and encouraging Stance Detection with multimodal approaches.
Another type of stance classification, more general-purpose, is the open stance classification task, usually indicated with the acronym SDQC, by referring to the four categories exploited for indicating the attitude of a message with respect to a rumour: Support (S), Deny (D), Query (Q) and Comment (C) (Aker et al., 2017). Two shared tasks of this type were organized in the last years at SemEval in the 2017 and in the 2019 editions, considering tweets along with Reddit posts in English (Derczynski et al., 2017, Gorrell et al., 2019).
The task is open to everyone from industry and academia. We in particular encourage the participations of researchers, industrial teams, and students.
With this task proposal we would like to invite participants to explore features based on the textual content of the tweet, such as structural, stylistic, and affective features, but also features based on contextual information that documents not emerge directly from the text, such as for instance knowledge about the domain of the political debate or information about the user's community.
Overall, we propose two different subtasks:
Task A - Textual Stance Detection:
The first task is a three-class classification task where the system has to predict whether a tweet is in favour, against or neutral/none towards the given target (following the guidelines below), exploiting only textual information, i.e. the text of the tweet.
Task B - Contextual Stance Detection:
The second task is the same as the first one: a three-class classification task where the system has to predict whether a tweet is in favour, against or neutral/none towards the given target. Here participants will have access to a wider range of contextual information based on the post such as: the number of retweets, the number of favours, the type of posting source (e.g. iOS or Android), and date of posting.
Furthermore we will share (and encourage its exploitation) contextual information related to the user, such as: number of tweets ever posted, user's bio (only emojis), user's number of followers, user's number of friends. Additionally we will share users' contextual information about their social network, such as: friends, replies, retweets, and quotes' relations.
The personal ids of the users will be anonymized but their network structures will be maintained intact.
The dataset will include short documents taken from Twitter. From May 29th it will be possible to download it from here: data repository. The files are encrypted. Please register to our Google Group to obtain the passwords.
The test set will be available in the same repository from September 18th, 2020.
Each participating team will initially have access to the training data only. Later, the unlabelled test data will be released (see the timeframe below). After the assessment, the labels for the test data will be released as well.
The evaluation will be performed according to the standard metrics known in literature (accuracy, precision, recall and F1-score). We will provide official ranking for Task A and for Task B, and two separateranking for constrained and unconstrained runs. Systems will be evaluated using F1-score computed over the two main classes (favour and against). The submissions will be ranked by the averaged F1-score above the two classes.
For both sub-tasks, we will compute a first baseline using a simple machine learning model based on SVM combined with uni-gram feature. And as second baseline we will provide one measure computed by a model based on previous work from the authors (MultiTacos) [Lai, M., Cignarella, A. T., Hernandez Fariás, D. I., Bosco, C., Patti,V., and Rosso, P. Multilingual Stance Detection in Social Media: Four Political Debates on Twitter. Computer Speech and Language 63 (2020)].
Register your team by using the registration web form at http://www.evalita.it/2020 (available soon, see timeframe below).
Information about the submission of results and their format will be available in the Task guidelines.
We invite the potential participants to subscribe to our Google Group in order to be kept up to date with the latest news related to the task. Please share comments and questions with the mailing list. The organizers will assist you for any potential issues that could be raised.
Participants will be required to provide an abstract and a technical report including a brief description of their approach, an illustration of their experiments, in particular techniques and resources used, and an analysis of their results for the publication in the Proceedings of contest.
29th May 2020: training data available (data repository)
4th September 2020: test data available
18th September 2020: test data available
18th September 2020 - 2nd October 2020: evaluation window
2nd October 2020: systems results due to the organizers
16th October 2020: deadline for submission of system description papers
6th November 2020: technical reports due to organizers (camera-ready)
27th November 2020: video presentations sent to the EVALITA chair
1st or 2nd December 2020: final workshop in Bologna
16th-17th December 2020: final workshop (online)
Write to us HERE!
Or contact the two main organizers: Mirko Lai, email@example.com and Alessandra T. Cignarella, firstname.lastname@example.org.