feat: ✨ ClearML training loss logging

[odulcy-mindee]

No description provided.

[felixdittrich92]

Thanks :)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.61%. Comparing base (b0d2728) to head (f670db7).
Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1844      +/-   ##
==========================================
+ Coverage   96.58%   96.61%   +0.02%     
==========================================
  Files         165      165              
  Lines        7940     7940              
==========================================
+ Hits         7669     7671       +2     
+ Misses        271      269       -2     

Flag	Coverage Δ
unittests	96.61% <ø> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kuraga]

Sorry, weren't there plans to make logging more universal?
(clearml_log argument name doesn't have a place for other loggers...)

[felixdittrich92]

Hi @kuraga 👋🏼,

What do you mean with other loggers ?
Sry I'm a bit out of scope with the Clearml topic (still using W&B on my own 😅)
CC @odulcy-mindee

[kuraga]

@felixdittrich92 Hm... Is W&B exactly the answer? 😄

I like a lightweight lazyscribe BTW...

I don't ask "to implement other loggers" but about some abstract layer...

[felixdittrich92]

@felixdittrich92 Hm... Is W&B exactly the answer? 😄

I like a lightweight lazyscribe BTW...

I don't ask "to implement other loggers" but about some abstract layer...

Ah got it I was thinking the question is related to something with ClearML 😅 For me personally W&B is still the answer xD But yeah everyone what they prefer

Mh yes I understood the request ..the thing is that the training scripts / references folder is more a "collection" of scripts everyone can modify to there own needs which has some advantages but on the other side also disadvantages (for example adding globally avaialble features shared between the different training tasks like logging, early stopping, and so on)

with an Trainer object like you know from the transformers lib for example it would be much easier to implement such features but on the other hand it would wrap everything, reduce the transparency and make it much harder for (experienced) users to transform the training process to there own needs 😅

[kuraga]

the training scripts / references folder is more a "collection" of scripts

Aaahh, didn't catch it. My fault. Training of docTR is Mindee's inner job at most. Thanks, @felixdittrich92 :)

P.S. Add READMEs to /references, etc. :)

[felixdittrich92]

Mh not directly we use the same scripts for pre-training all the models but users can use it also for fine-tuning on there own needs / own datasets :)

The inner task specific folders have already README's :)
For example: https://github.com/mindee/doctr/tree/main/references/recognition

[felixdittrich92]

And the corresponding doc: https://mindee.github.io/doctr/latest/using_doctr/custom_models_training.html

[kuraga]

Well, if the mechanism is public I'll illustrate my

I don't ask "to implement other loggers" but about some abstract layer...

: the clearml_log: bool argument could be logger: logging.Logger :)

[kuraga]

Update: hhm, now I see clearml.Logger does not inherit logging.Logger...

[ThanhNX0401]

Is there any reason why docTR don't output the Training loss log to WB? How can i tell if the model is overfit or underfit?

[felixdittrich92]

Is there any reason why docTR don't output the Training loss log to WB? How can i tell if the model is overfit or underfit?

Oh you are right we should do this for every train script the same way it's done with the clearml_log passed to the train function. Would you like to open a PR ?

[felixdittrich92]

@ThanhNX0401 Fixed now on main branch 😊