Seeds for Reproducing Results

[fzohra]

Hi,

I am trying to reproduce the reported results when training on 4 A100 GPUs. I am using the provided configurations for all evaluations (fully supervised, zero-shot, few-shot, and base-to-novel), but the accuracy tends to fall by 1-2%.

Can you please share the seeds used obtained the reported results.

Also, what are the hardware specs used for training the results reported in the paper?

Thank you!

[muzairkhattak]

Hi @fzohra,

Thank you for showing interest in our work!

Regarding your queries, please note the following:

• Firstly, have you tried to reproduce the results using our pretrained models? It will be good to verify if your GPU machine can correctly reproduce results using the provided pretrained models.
• The fluctuation in the results specially in the few-shot training setting could be due to random variations when tested on different machines. We have also observed same trend when we trained same models on different machines.
• Regarding the seeds, we are directly using the splits provided in datasets_splits/hmdb_splits. After training, we evaluate models on 3 validation splits and report the numbers averaged over these 3 validation splits. The SEED by default is fixed to 1024.
• Lastly, we also used 4 A100 GPUs to train the models in few-shot and base-to-novel benchmarks. The overall batch size should be the same as provided in the config files.

I hope this is helpful for you. Please let us know if you have any additional questions!
Thank you.

[fzohra]

Thanks for the reply!
I'm currently using v100 and 4 gpus and am trying to reproduce the results for the base to novel evaluation on ssv2. I've increased batch size to 8 when training to maintain an effective batch size of 64.

I'm using the splits defined in datasets_splits/base2novel_splits/ssv2_splits.

Using the provided pretained weights (seed1, seed2, and seed3), I am able to reproduce the results doing inference on the three base and novel classes. On the base classes the average is Acc@1 16.148 Acc@5 39.474. On the novel classes, the result is Acc@1 12.337 Acc@5 31.536.

However, when I train the model, the results are lower. As you can see in the following table, the accuracy on the base validation sets for each split after 11 epochs of training is 14.356, which is less than the 16.148. I don't have the numbers anymore for when I was reproducing on an a100's but it was a similar scenario. Would really like to reproduce the baseline to be close to the pre-trained weights.

		base to novel split 1	base to novel split 2	base to novel split 3	base avg
vifi-clip [reproduced]	ssv2 (bs=8, ga=2, gpu=4)
	epoch 0	Acc@1 7.346 Acc@5 18.542	Acc@1 7.358 Acc@5 18.588	Acc@1 7.386 Acc@5 18.650	Acc@1 7.363 Acc@5 18.593
	epoch 1	Acc@1 7.938 Acc@5 20.188	Acc@1 8.035 Acc@5 20.444	Acc@1 8.081 Acc@5 20.569	Acc@1 8.018 Acc@5 20.400
	epoch 2	Acc@1 9.038 Acc@5 23.366	Acc@1 9.066 Acc@5 23.303	Acc@1 9.163 Acc@5 23.109	Acc@1 9.089 Acc@5 23.259
	epoch 3	Acc@1 9.789 Acc@5 26.071	Acc@1 10.165 Acc@5 26.213	Acc@1 10.592 Acc@5 27.084	Acc@1 10.182 Acc@5 26.456
	epoch 4	Acc@1 11.851 Acc@5 30.194	Acc@1 12.153 Acc@5 30.091	Acc@1 12.534 Acc@5 30.997	Acc@1 12.179 Acc@5 30.427
	epoch 5	Acc@1 13.183 Acc@5 33.468	Acc@1 13.087 Acc@5 33.514	Acc@1 13.423 Acc@5 33.440	Acc@1 13.231 Acc@5 33.474
	epoch 6	Acc@1 13.588 Acc@5 34.157	Acc@1 13.314 Acc@5 34.248	Acc@1 14.009 Acc@5 34.476	Acc@1 13.637 Acc@5 34.294
	epoch 7	Acc@1 14.009 Acc@5 34.937	Acc@1 13.610 Acc@5 35.023	Acc@1 14.487 Acc@5 35.000	Acc@1 14.035 Acc@5 34.987
	epoch 8	Acc@1 14.226 Acc@5 35.239	Acc@1 13.787 Acc@5 35.490	Acc@1 14.470 Acc@5 35.228	Acc@1 14.161 Acc@5 35.319
	epoch 9	Acc@1 14.288 Acc@5 35.330	Acc@1 13.952 Acc@5 35.786	Acc@1 14.710 Acc@5 35.495	Acc@1 14.317 Acc@5 35.537
	epoch 10	Acc@1 14.402 Acc@5 35.364	Acc@1 13.969 Acc@5 35.871	Acc@1 14.698 Acc@5 35.552	Acc@1 14.356 Acc@5 35.596