pred_loss decrease fast while avg_acc stay at 50%

[jiqiujia]

I try to run the code on a small dataset and I find that pred_loss decrease fast while avg_acc stay at 50%. It is strange to me since decrease in pred_loss should indicates increase in accuracy.

[wenhaozheng-nju]

I also meet the same problem in small dataset.

[NiHaoUCAS]

me too

[codertimo]

Hmm interesting.. Is this the result of 0.0.1a4 version?
And How did you guys print out that result?

[NiHaoUCAS]

Hmm interesting.. Is this the result of 0.0.1a4 version?
And How did you guys print out that result?

0.0.01a3 vesion
the result is print out by bert cmd , no any modify.

[jiqiujia]

Hmm interesting.. Is this the result of 0.0.1a4 version?
And How did you guys print out that result?

I try using 0.0.1a4 and the result is the same

[codertimo]

Hmmm... anyone have any clues?

[yangze01]

I try using different data: continuous sentence pair from same document, concat continuous sentence as longer sentence , query and document pair, the result is the same. I also found that there is a big gap between next_loss and mask_loss although they use the same loss function.

[cairoHy]

Probably the criterion loss function is the problem.

# shape [10, 2], not very accurate output
out = torch.tensor([[ -8.4014,  -0.0002],
        [-10.3151,  -0.0000],
        [ -8.8440,  -0.0001],
        [ -7.5148,  -0.0005],
        [-11.0145,  -0.0000],
        [-10.9770,  -0.0000],
        [-13.3770,  -0.0000],
        [ -9.5733,  -0.0001],
        [ -9.5957,  -0.0001],
        [ -9.0712,  -0.0001]])
# shape [10], next sentence label
label = torch.tensor([1,1,0,1,0,0,1,0,0,1])
original_criterion = nn.NLLLoss(ignore_index=0)
criterion = nn.NLLLoss()
original_loss = original_criterion(out, label)
loss = criterion(out, label)

with the above code snippet, the original_loss is 0.0002, and loss is 5.0005.

I changed following code in trainer/pretrain.py:

self.criterion = nn.NLLLoss(ignore_index=0)

to:

self.criterion = nn.NLLLoss()

And as the magnitude of next_loss is smaller than mask_loss, I also over weight the next_loss, and get 58% next accuracy after train on my corpus for one epoch.

[jiqiujia]

Probably the criterion loss function is the problem.

# shape [10, 2], not very accurate output
out = torch.tensor([[ -8.4014,  -0.0002],
        [-10.3151,  -0.0000],
        [ -8.8440,  -0.0001],
        [ -7.5148,  -0.0005],
        [-11.0145,  -0.0000],
        [-10.9770,  -0.0000],
        [-13.3770,  -0.0000],
        [ -9.5733,  -0.0001],
        [ -9.5957,  -0.0001],
        [ -9.0712,  -0.0001]])
# shape [10], next sentence label
label = torch.tensor([1,1,0,1,0,0,1,0,0,1])
original_criterion = nn.NLLLoss(ignore_index=0)
criterion = nn.NLLLoss()
original_loss = original_criterion(out, label)
loss = criterion(out, label)

with the above code snippet, the original_loss is 0.0002, and loss is 5.0005.

I changed following code in trainer/pretrain.py:

self.criterion = nn.NLLLoss(ignore_index=0)

to:

self.criterion = nn.NLLLoss()

And as the magnitude of next_loss is smaller than mask_loss, I also over weight the next_loss, and get 58% next accuracy after train on my corpus for one epoch.

That's right. I just figure it out. Also note that for masklm, we still need ignore_index=0 since we only want to predict the masked words.

[codertimo]

@cairoHy Wow thank you for your smart analysis.

I just fixed this issue on 0.0.1a5 version branch. And changes is under here.

BERT-pytorch/bert_pytorch/trainer/pretrain.py

Lines 61 to 62 in 2a0b282

	self.masked_criterion = nn.NLLLoss(ignore_index=0)
	self.next_criterion = nn.NLLLoss()

BERT-pytorch/bert_pytorch/trainer/pretrain.py

Lines 98 to 102 in 2a0b282

	# 2-1. NLL(negative log likelihood) loss of is_next classification result
	next_loss = self.next_criterion(next_sent_output, data["is_next"])
	# 2-2. NLLLoss of predicting masked token word
	mask_loss = self.masked_criterion(mask_lm_output.transpose(1, 2), data["bert_label"])

[codertimo]

Thanks everyone who join this investigation :)
It was totally my fault and sorry for your inconvenience during bug fixing.

Additionally, is here anyone can test the new code with your own corpus?
Any feedback would be welcome, and you can reinstall new version using under command.

git clone https://github.com/codertimo/BERT-pytorch.git
git checkout 0.0.1a5
pip install -U .

specially thanks for @jiqiujia @cairoHy @NiHaoUCAS @wenhaozheng-nju