You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There's duplicate code in different LightningModules that calculates metrics.
To avoid this metric setup, update, logging, reset can be moved to a lightning callback which can then be used by different modules.
The callback should have an init where you specify what metrics you want to track: f1, recall precision, accuracy, ROC-AUC, PR-AUC x averaging micro, macro, weighted, per-class.
Macro averaging should happen on the full validation dataset, not minibatched like it currently is.
By tracking validation_step outputs and not relying on on_validation_epoch_end results for preds we can address #1 as well
One wrinkle is, how do we manage plot callbacks. Many of them rely on averaging=None metrics. Do they become helper functions to this one main metrics callback? That doesn't seem right. I suppose they should also use aggregate validation_step outputs.
The text was updated successfully, but these errors were encountered:
There's duplicate code in different LightningModules that calculates metrics.
To avoid this metric setup, update, logging, reset can be moved to a lightning callback which can then be used by different modules.
The callback should have an init where you specify what metrics you want to track: f1, recall precision, accuracy, ROC-AUC, PR-AUC x averaging micro, macro, weighted, per-class.
Macro averaging should happen on the full validation dataset, not minibatched like it currently is.
By tracking validation_step outputs and not relying on on_validation_epoch_end results for preds we can address #1 as well
One wrinkle is, how do we manage plot callbacks. Many of them rely on averaging=None metrics. Do they become helper functions to this one main metrics callback? That doesn't seem right. I suppose they should also use aggregate validation_step outputs.
The text was updated successfully, but these errors were encountered: