cannot run hg38_hyena_seqlen_warmup_reload

[sustcsonglin]

Hi,

Thanks for your great work! I am trying to do hg38/hg38_hyena_seqlen_warmup_reload.yaml experiment. Got the following error msg:

I had some initial search on this issue and found this. I set monitor: test/loss and it still doesn't work. But i have no problem running 'g38/hg38_hyena.yaml'.

Do you have any insights on this issue? Is this related to the sentence length warmup callback? because I can run 'g38/hg38_hyena.yaml' without this callback. i am using pytorch_lightning v1.8.6

[sustcsonglin]

After some inspections, I found the reason! link1 disables loading the test dataloader during training, but the monitor metric was not changed. Hope this configuration will be fixed

[sustcsonglin]

Another question: does this configuration file consider the trainer.devices for setting accumulate_grad_batches? I didn't find sth like dividing by trainer.devices here