joblib.Memory allows to cache results. In scikit-learn, the Pipeline exposes a memory parameter to allow for such behaviour. It would be cool to go at the estimator level to have some aggressive caching. But it is not a straightforward task because sometimes hashing the inputs is more costly than just calling the function itself.

So if skore could make a sensible caching mechanism into place.

Example regarding the caching: https://joblib.readthedocs.io/en/stable/auto_examples/memory_basic_usage.html#sphx-glr-auto-examples-memory-basic-usage-py

In #997, there is an in-memory caching mechanism. Persisting it on the disk would be useful to avoid the recomputing some of the results.

For this to work, we need to move the fitting logic out of CrossValidationReport.__init__. Indeed, right now

report = CrossValidationReport(estimator, ...)

immediately fits the estimators, which can take a long time; if the fitting is interrupted, then the whole __init__ fails, so report is not defined and there is no way to access anything.

Instead the fitting could be done as soon as some metric/plot method is called.
Another solution (maybe quicker to implement) could be to expose the _fit_estimator_reports method, and have the user call it explicitly.

immediately fits the estimators, which can take a long time; if the fitting is interrupted, then the whole init fails, so report is not defined and there is no way to access anything.

I don't think so. The fitting is using a joblib generator. It means that theoretically, we can catch the exception and terminate the process properly with the current results. It is not related with the __init__ itself.

The more challenging and subsequent feature is being able to "resume" the execution from what have computed to be able to not recompute some previous cached information. Maybe in this case, we should have a resume(...) method because you will already have the instance at hand:

report = CrossValidationReport(...)
# report is interrupted or crash for some reason
report.resume()
# eventually to restart from scractch
report.restart()

(disclosure I'm not sure about the naming because they might not be explicit what are we restarting or resuming).

I can confirm that with the "call _fit_estimator_reports explicitly" solution works. I can push my local experiment if you're interested.

Doing this brought up the same question as @glemaitre: What happens if _fit_estimator_reports is called again: should it resume or restart? IMO, for a first iteration, restarting from scratch every time is fine. If you want to keep your previous "run" then you can put it in storage.

Even if it's theoretically possible to do everything in the __init__, I still think the fit pattern from sklearn is preferable. __init__ should not be a long computation.

With that said, I'll now look at catching exceptions directly in _fit_estimator_reports.

I can confirm that with the "call _fit_estimator_reports explicitly" solution works. I can push my local experiment if you're interested.

I'm interested to see what about just to be sure in which "user setting" you are.

With that said, I'll now look at catching exceptions directly in _fit_estimator_reports.

My thought would be to catch it up in the loop consuming the generator (but I need to see you other code to understand exactly the use case).

See 3acfb1f for the "call _fit_estimator_reports explicitly" attempt

See 928aa5e for the "catch exceptions in _fit_estimator_reports" attempt

I took some time to try to write an automatic test, and I'm not getting anywhere. The clone-ing of the estimator and the parallelism when we run the cross-validation makes it difficult. Any ideas @glemaitre @rouk1 @thomass-dev ? Otherwise I'll just write a manual testing guide in the PR.

Try by mocking clone?

Try by mocking clone?

Thanks, worked perfectly :)