How to replicate the QA work on another dataset?

[Marwa199527]

I am trying to train the AraBERT on another Arabic question answering dataset which is a squad-like format. I am sure that I am doing something wrong but I do not have any idea what is it.

!python /content/arabert/arabert/arcd_preprocessing.py
--input_file="/content/drive/MyDrive/train-data.json"
--output_file="train-pre.json"
--do_farasa_tokenization=True
--use_farasapy=True \

So, I want to ask the following which might be where I got wrong?
1)How exactly you pass the model name.
2) I took the checkpoint, config, vocab files from SOQAL.

!python /content/arabert/arabert/run_squad.py
--vocab_file=multi_cased_L-12_H-768_A-12/vocab.txt
--bert_config_file=multi_cased_L-12_H-768_A-12/bert_config.json
--init_checkpoint=multi_cased_L-12_H-768_A-12/bert_model.ckpt
--do_train=True
--train_file=/content/train-pre.json
--do_predict=False
--predict_file=/content/dev-pre.json
--train_batch_size=32
--predict_batch_size=24
--learning_rate=3e-5
--num_train_epochs=4
--do_lower_case=False
--use_tpu=False
--tpu_name=$TPU_ADDRESS
--max_seq_length=384
--doc_stride=128
--output_dir=/content/out

This is the final result:

{"exact_match": 0.06265664160401002, "f1": 10.098177855997694, "exact_sentence": 45.30075187969925}

Note:
The prediction took a much longer time than the training which end in just 5 minutes or less.

Could you please walk me step by step to do this in the right way?

[WissamAntoun]

How big is your dataset? Also please check that the answer is correct in the data, and that the answer_start character is also correct

I will be releasing a colab notebook for question answering on SQUAD datasets soon, which has better customizability than the official BERT one.

[Marwa199527]

Thanks for your reply @WissamAntoun !
So the data is big enough, I checked all the above and there is no problem could you please tell me when you will release the colab notebook?

[WissamAntoun]

@Marwa199527 here is an early version of it https://colab.research.google.com/drive/1hik0L_Dxg6WwJFcDPP1v74motSkst4gE?usp=sharing

the utils_qa.py , trainer_qa.py, run_qa.py might not be fully compatible with transformers==4.2.1 so you might need ti either install the latest version or to install from source

[Marwa199527]

Hi @WissamAntoun ,
For the most recent version of your colab I got this error
Traceback (most recent call last): File "run_qa.py", line 546, in <module> main() File "run_qa.py", line 291, in main answer_column_name = "answer" if "answer" in column_names else column_names[2] IndexError: list index out of range

[WissamAntoun]

make sure that your dataset has a field called 'answers' , and it has the same format as squad

[salama4]

i facing similar problem, where i want to train poly-encoder model, for question-answering and i need to use arabert inside it, but i dont know what is the syntax for doing that.

poly-encoder model syntax:-
parlai train_model \ --init-model zoo:pretrained_transformers/poly_model_huge_reddit/model \ -t convai2 \ --model transformer/polyencoder --batchsize 256 --eval-batchsize 10 \ --warmup_updates 100 --lr-scheduler-patience 0 --lr-scheduler-decay 0.4 \ -lr 5e-05 --data-parallel True --history-size 20 --label-truncate 72 \ --text-truncate 360 --num-epochs 8.0 --max_train_time 200000 -veps 0.5 \ -vme 8000 --validation-metric accuracy --validation-metric-mode max \ --save-after-valid True --log_every_n_secs 20 --candidates batch --fp16 True \ --dict-tokenizer bpe --dict-lower True --optimizer adamax --output-scaling 0.06 \ --variant xlm --reduction-type mean --share-encoders False \ --learn-positional-embeddings True --n-layers 12 --n-heads 12 --ffn-size 3072 \ --attention-dropout 0.1 --relu-dropout 0.0 --dropout 0.1 --n-positions 1024 \ --embedding-size 768 --activation gelu --embeddings-scale False --n-segments 2 \ --learn-embeddings True --polyencoder-type codes --poly-n-codes 64 \ --poly-attention-type basic --dict-endtoken __start__ \ --model-file <YOUR MODEL FILE>
could i have any help please?