The goal of stance detection is to determine the viewpoint expressed in a piece of text towards a target. These viewpoints or contexts are often expressed in many different languages depending on the user and the platform, which can be a local news outlet, a social media platform, a news forum, etc. Most research in stance detection, however, has been limited to working with a single language and on a few limited targets, with little work on cross-lingual stance detection. Moreover, non-English sources of labelled data are often scarce and present additional challenges. Recently, large multilingual language models have substantially improved the performance on many non-English tasks, especially such with limited numbers of examples. This highlights the importance of model pre-training and its ability to learn from few examples. In this paper, we present the most comprehensive study of cross-lingual stance detection to date: we experiment with 15 diverse datasets in 12 languages from 6 language families, and with 6 low-resource evaluation settings each. For our experiments, we build on pattern-exploiting training, proposing the addition of a novel label encoder to simplify the verbalisation procedure. We further propose sentiment-based generation of stance data for pre-training, which shows sizeable improvement of more than 6% F1 absolute in low-shot settings compared to several strong baselines.
$ python3 -m venv ~/.virtualenvs/stance-detection
$ source ~/.virtualenvs/stance-detection/bin/activate# And to install the packages
$ pip install -r requirements.txtWe release our few-shot splits (32, 64, 128, 256) in the data/fewshow folder. Moreover, we release the sentiment annotated Wiki snipets in the data/wikipedia folder. The full training, dev and test sets can be obtained from the links below.
- Stance Prediction and Claim Verification: An Arabic Perspective Data (ans)
- Integrating Stance Detection and Fact Checking in a Unified Corpus Data (arabicfc)
- Detecting Stance in Czech News Commentaries Data (czech)
- Stance Evolution and Twitter Interactions in an Italian Political Debate Data (conref)
- (Danish) Joint Rumour Stance and Veracity Prediction Data (dast)
- Multilingual stance detection in social media political debates Data (e-fra, r-ita)
- An English-Hindi Code-Mixed Corpus: Stance Annotation and Baseline System Data (hindi)
- Overview of NLPCC Shared Task 4: Stance Detection in Chinese Microblogs Data (nlpcc)
- Stance and Gender Detection in Tweets on Catalan Independence@Ibereval 2017 Data (iberval)
- Stance Prediction for Russian: Data and Analysis Data (rustance)
- SardiStance @ EVALITA2020 Data (sardistance)
- X-Stance: A Multilingual Multi-Target Dataset for Stance Detection Data
Some datasets may require additional steps to aquire, e.g., in order to obtain SardiStance you need to fill out a form, IberEval's test sets need to be obtained from the competition organizers.
- Stance Detection Benchmark Data (arc argmin fnc1 iac1 ibmcs perspectrum scd semeval2016t6 semeval2019t7 snopes)
- Will-They-Won't-They Data (wtwt)
- Emergent Data (emergent)
- Rumor has it Data (rumor)
- Multi-Target Stance Dataset Data (mtsd)
- Political Debates Data (poldeb)
- VAried Stance Topics Data (vast)
We used the data splits as described in Cross-Domain Label-Adaptive Stance Detection (code).
DATASETS=(arc argmin fnc1 iac1 ibmcs perspectrum scd semeval2016t6 semeval2019t7 snopes emergent mtsd poldeb rumor vast wtwt)
CROSS_LINGUAL_DATASETS=(conref-ita arabicfc ans nlpcc czech dast e-fra hindi iberval2017-ca iberval2017-es r-ita rustance sardistance xstance-de xstance-fr)
python src/stancedetection/models/trainer_le.py --data_dir "data/all/" \
--model_name_or_path ${MODEL_NAME} \
--output_dir ${OUTPUT_DIR} \
--task_names ${DATASET_NAME} \
--model_type xlm-r \
--replace_classification \
--do_train \
--do_eval \
--learning_rate ${LEARNING_RATE} \
--weight_decay 0.01 \
--per_gpu_train_batch_size 16 \
--per_gpu_eval_batch_size 128 \
--num_train_epochs 50000 \
--warmup_proportion ${WARMUP} \
--adam_epsilon 1e-08 \
--logging_steps 200 \
--max_steps ${MAX_STEPS} \
--max_seq_length ${MAX_SEQ_LEN} \
--evaluate_during_training \
--gradient_accumulation_steps 1 \
--seed ${SEED} \
--dataset_suffix "_${SHOTS}_${i}" \
--fp16 \
--cache_dir cache \
--balanced \
--lambda_mlm ${LAMBDA_MLM} \
--positive_samples_synonyms ${POSITIVE_SAMPLES_SYNONYMS} \
--negative_samples_synonyms ${NEGATIVE_SAMPLES_SYNONYMS} \
--negative_samples_rand ${NEGATIVE_SAMPLES_RAND} \
--p_replace_pos_label ${P_REPLACE_POS_LABEL} \
--p_replace_neg_label ${P_REPLACE_NEG_LABEL} \
--p_mask ${P_MASK} \
--p_random ${P_RANDOM} \
--p_delete 0.0 \
--p_split 0.0 \
--p_swap 0.0 \
--p_label_cond 0.0 \
--overwrite_output_dirPlease cite as [1]. There is also an arXiv version.
[1] Hardalov, M., Arora, A., Nakov, P., & Augenstein, I. (2022). "Few-Shot Cross-Lingual Stance Detection with Sentiment-Based Pre-Training", Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI-22).
@article{hardalov-etal-2022-fewshot,
title = {Few-Shot Cross-Lingual Stance Detection with Sentiment-Based Pre-Training},
author = {Hardalov, Momchil and Arora, Arnav and Nakov, Preslav and Augenstein, Isabelle},
year = 2022,
month = {Feb},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
volume = 36
}
The code in this repository is licenced under the CC-BY-NC-SA 4.0. The datasets are licensed under CC-BY-SA 4.0.