Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Train Log file is missing information show at the console #658

Open
AHarouni opened this issue Sep 18, 2024 · 0 comments
Open

Train Log file is missing information show at the console #658

AHarouni opened this issue Sep 18, 2024 · 0 comments

Comments

@AHarouni
Copy link

Describe the bug
Log files generated doesn't include all lines shown on the console

To Reproduce
I testing this on vista3d and whole body bundle. Run a training job see output on the console showing lines as

2024-09-14 03:19:12,752 - INFO - Epoch: 881/900, Iter: 24/25 -- train_loss: 0.4078 
2024-09-14 03:19:13,500 - INFO - Epoch: 881/900, Iter: 25/25 -- train_loss: 0.4699 
2024-09-14 03:19:13,500 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[881] Complete. Time taken: 00:00:19.345
2024-09-14 03:19:14,636 - INFO - Epoch: 882/900, Iter: 1/25 -- train_loss: 0.3655 
2024-09-14 03:19:15,386 - INFO - Epoch: 882/900, Iter: 2/25 -- train_loss: 0.3317 
2024-09-14 03:19:16,135 - INFO - Epoch: 882/900, Iter: 3/25 -- train_loss: 0.3545 
2024-09-14 03:19:16,884 - INFO - Epoch: 882/900, Iter: 4/25 -- train_loss: 0.2083 
2024-09-14 03:19:17,633 - INFO - Epoch: 882/900, Iter: 5/25 -- train_loss: 0.2773 
....
2024-09-14 03:19:31,124 - INFO - Epoch: 882/900, Iter: 23/25 -- train_loss: 0.2539 
2024-09-14 03:19:31,874 - INFO - Epoch: 882/900, Iter: 24/25 -- train_loss: 0.1814 
2024-09-14 03:19:32,622 - INFO - Epoch: 882/900, Iter: 25/25 -- train_loss: 0.2402 
2024-09-14 03:19:32,622 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[882] Complete. Time taken: 00:00:19.122
2024-09-14 03:19:33,923 - INFO - Epoch: 883/900, Iter: 1/25 -- train_loss: 0.2866 

however the log file will show only


2024-09-14 03:19:13,500 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[881] Complete. Time taken: 00:00:19.345
2024-09-14 03:19:32,622 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[882] Complete. Time taken: 00:00:19.122

Expected behavior

The file log should show exactly the same as what is on the console.

Additional context

I have tried changing the loggin.config file but it seems the monai bundle is loading logging config from else where.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant