Unexpected behavior in masked prediction

[kwcckw]

Hi,

I was trying to make some masked prediction with "bert-base-arabertv2", but looks like there's some unexpected behavior. I'm getting a lot of '+' sign in the prediction:

Is this common and i should remove it? Or that's mean the model need to be fine tuned?

[WissamAntoun]

Are you pre-processing the sentence (doing pre-segmentation) before you give it to the model?

[kwcckw]

also from this discussion:
#114

you mentioned :

the preprocessor will automatically apply farasa segmentation when the model passed into it requires segmentation

So would this be the root cause of the issue where the segmenter is not applied in some cases?

[WissamAntoun]

I tested these examples, and I got the same results.

From what I can see, most of these results are linguistically right and plausible.
I can also see that the pre-segmented model tends to prefer predicting a prefix/suffix. I tried the large model and it should perform better in some cases.

On the other hand, my suggestion is to give the model longer context, as I see that some examples were taken from the middle of a sentence. Also if this behavior seems to presist I suggest that you try the v02 models without pre-segmentation

[kwcckw]

i tried with base v02, this is the snippet of code:

    from transformers import AutoModelWithLMHead, AutoTokenizer
    from transformers import pipeline
    from farasa.segmenter import FarasaSegmenter
    from preprocess import ArabertPreprocessor
    
    model_name = "bert-base-arabertv02"
    
    modelv02 = AutoModelWithLMHead.from_pretrained(model_name)
    tokenizerv02 = AutoTokenizer.from_pretrained(model_name)
    arabert_prep = ArabertPreprocessor(model_name=model_name)
    
    pipeline_fill_mask_arabertv2 = pipeline("fill-mask",model=modelv02, tokenizer=tokenizerv02)
    
    
    text = " عاصمة لبنان هي [MASK]"
    
    textv2 = arabert_prep.preprocess(text)
    
    result = pipeline_fill_mask_arabertv2(textv2)
    
    print(textv2)
    print(result)
    print(result[0]['token_str'])

And the results:

    [{'sequence': '[CLS] عاصمة لبنان هي : [SEP]',
      'score': 0.3804662525653839,
      'token': 31,
      'token_str': ':'},
     {'sequence': '[CLS] عاصمة لبنان هي بيروت [SEP]',
      'score': 0.2570646405220032,
      'token': 2079,
      'token_str': 'بيروت'},
     {'sequence': '[CLS] عاصمة لبنان هي. [SEP]',
      'score': 0.06474816054105759,
      'token': 20,
      'token_str': '.'},
     {'sequence': '[CLS] عاصمة لبنان هي العاصمة [SEP]',
      'score': 0.021024178713560104,
      'token': 2113,
      'token_str': 'العاصمة'},
     {'sequence': '[CLS] عاصمة لبنان هي طرابلس [SEP]',
      'score': 0.013106552883982658,
      'token': 4861,
      'token_str': 'طرابلس'}]

The result with highest score doesn't looks correct, could you advise again?

Thank you.

[WissamAntoun]

The model thinks that ":" is the most likely token, which is actually fine here. For the best/desired results, you should look into creating a better prompt for the model, which forces the model to output full words. This is done mostly by trial and error.

See in this case if you add a dot after the [MASK] you will get Beirut with a confidence of 84%. Or with a ":" before the MASK and a dot after you will get 93% confidence.

[
  {
    "sequence": "عاصمة لبنان هي بيروت.",
    "score": 0.8453249931335449,
    "token": 2079,
    "token_str": "بيروت"
  },
  {
    "sequence": "عاصمة لبنان هي طرابلس.",
    "score": 0.03817316144704819,
    "token": 4861,
    "token_str": "طرابلس"
  },
  {
    "sequence": "عاصمة لبنان هي العاصمة.",
    "score": 0.016537988558411598,
    "token": 2113,
    "token_str": "العاصمة"
  },
  {
    "sequence": "عاصمة لبنان هي صيدا.",
    "score": 0.011134223081171513,
    "token": 5840,
    "token_str": "صيدا"
  },
  {
    "sequence": "عاصمة لبنان هي المدينة.",
    "score": 0.008538899943232536,
    "token": 1665,
    "token_str": "المدينة"
  }
]

[
  {
    "sequence": "عاصمة لبنان هي : بيروت.",
    "score": 0.9301252365112305,
    "token": 2079,
    "token_str": "بيروت"
  },
  {
    "sequence": "عاصمة لبنان هي : طرابلس.",
    "score": 0.018252452835440636,
    "token": 4861,
    "token_str": "طرابلس"
  },
  {
    "sequence": "عاصمة لبنان هي : بعبدا.",
    "score": 0.012975244782865047,
    "token": 16349,
    "token_str": "بعبدا"
  },
  {
    "sequence": "عاصمة لبنان هي : العاصمة.",
    "score": 0.006037214770913124,
    "token": 2113,
    "token_str": "العاصمة"
  },
  {
    "sequence": "عاصمة لبنان هي : صيدا.",
    "score": 0.0033810429740697145,
    "token": 5840,
    "token_str": "صيدا"
  }
]

[kwcckw]

ah i see, thanks again for the clarification.