Weights Not Loading for text_branch in Training Script

[WangHaoyuuu]

I am experiencing an issue with a training script for an audio-visual model where the text_branch components are not loading any pre-trained weights as expected. The unloaded components include all layers of text_branch, such as word embeddings, position embeddings, attention layers, and all associated weights and biases. Is this necessary?

here is the warning：

text_branch.embeddings.word_embeddings.weight Unloaded
text_branch.embeddings.position_embeddings.weight Unloaded
text_branch.embeddings.token_type_embeddings.weight Unloaded
text_branch.embeddings.LayerNorm.weight Unloaded
text_branch.embeddings.LayerNorm.bias Unloaded
text_branch.encoder.layer.0.attention.self.query.weight Unloaded
text_branch.encoder.layer.0.attention.self.query.bias Unloaded
text_branch.encoder.layer.0.attention.self.key.weight Unloaded
text_branch.encoder.layer.0.attention.self.key.bias Unloaded
text_branch.encoder.layer.0.attention.self.value.weight Unloaded
text_branch.encoder.layer.0.attention.self.value.bias Unloaded
text_branch.encoder.layer.0.attention.output.dense.weight Unloaded
text_branch.encoder.layer.0.attention.output.dense.bias Unloaded
text_branch.encoder.layer.0.attention.output.LayerNorm.weight Unloaded
text_branch.encoder.layer.0.attention.output.LayerNorm.bias Unloaded
text_branch.encoder.layer.0.intermediate.dense.weight Unloaded
text_branch.encoder.layer.0.intermediate.dense.bias Unloaded
text_branch.encoder.layer.0.output.dense.weight Unloaded
text_branch.encoder.layer.0.output.dense.bias Unloaded

Here is my shell script:
#!/bin/bash

CUDA_VISIBLE_DEVICES=1 python -m training.main
--save-frequency 5
--save-top-performance 3
--save-most-recent
--dataset-type="webdataset"
--datasetpath='/home/ubuntu/AudioLDM-training-finetuning/data/dataset'
--precision="fp32"
--batch-size=32
--lr=1e-4
--wd=0.0
--epochs=45
--workers=1
--use-bn-sync
--amodel HTSAT-tiny
--tmodel roberta
--warmup 3200
--datasetnames "Clotho"
--datasetinfos "train"
--top-k-checkpoint-select-dataset="Clotho-test"
--top-k-checkpoint-select-metric="mAP@10"
--logs 'logs'
--seed 3407
--gather-with-grad
--optimizer "adam"
--data-filling "repeatpad"
--data-truncating "rand_trunc"
--pretrained-audio '/home/ubuntu/AudioLDM-training-finetuning/data/checkpoints/HTSAT-fullset-imagenet-tiny-map=0.467.ckpt'

Here are my evaluation results:

It is very bad.

Could you please help me?

[zjsong]

I am experiencing an issue with a training script for an audio-visual model where the text_branch components are not loading any pre-trained weights as expected. The unloaded components include all layers of text_branch, such as word embeddings, position embeddings, attention layers, and all associated weights and biases. Is this necessary?

here is the warning：

text_branch.embeddings.word_embeddings.weight Unloaded text_branch.embeddings.position_embeddings.weight Unloaded text_branch.embeddings.token_type_embeddings.weight Unloaded text_branch.embeddings.LayerNorm.weight Unloaded text_branch.embeddings.LayerNorm.bias Unloaded text_branch.encoder.layer.0.attention.self.query.weight Unloaded text_branch.encoder.layer.0.attention.self.query.bias Unloaded text_branch.encoder.layer.0.attention.self.key.weight Unloaded text_branch.encoder.layer.0.attention.self.key.bias Unloaded text_branch.encoder.layer.0.attention.self.value.weight Unloaded text_branch.encoder.layer.0.attention.self.value.bias Unloaded text_branch.encoder.layer.0.attention.output.dense.weight Unloaded text_branch.encoder.layer.0.attention.output.dense.bias Unloaded text_branch.encoder.layer.0.attention.output.LayerNorm.weight Unloaded text_branch.encoder.layer.0.attention.output.LayerNorm.bias Unloaded text_branch.encoder.layer.0.intermediate.dense.weight Unloaded text_branch.encoder.layer.0.intermediate.dense.bias Unloaded text_branch.encoder.layer.0.output.dense.weight Unloaded text_branch.encoder.layer.0.output.dense.bias Unloaded

Here is my shell script: #!/bin/bash

CUDA_VISIBLE_DEVICES=1 python -m training.main --save-frequency 5 --save-top-performance 3 --save-most-recent --dataset-type="webdataset" --datasetpath='/home/ubuntu/AudioLDM-training-finetuning/data/dataset' --precision="fp32" --batch-size=32 --lr=1e-4 --wd=0.0 --epochs=45 --workers=1 --use-bn-sync --amodel HTSAT-tiny --tmodel roberta --warmup 3200 --datasetnames "Clotho" --datasetinfos "train" --top-k-checkpoint-select-dataset="Clotho-test" --top-k-checkpoint-select-metric="mAP@10" --logs 'logs' --seed 3407 --gather-with-grad --optimizer "adam" --data-filling "repeatpad" --data-truncating "rand_trunc" --pretrained-audio '/home/ubuntu/AudioLDM-training-finetuning/data/checkpoints/HTSAT-fullset-imagenet-tiny-map=0.467.ckpt'

Here are my evaluation results:

It is very bad.

Could you please help me?

It seems like that the aforementioned weights would be loaded successfully when using the updated version (1.1.6).

However, the warning is still there:

Not sure if this matters.