Rewrite of the load_checkpoint function

[Xirider]

This is a rewrite of how we determine which checkpoint to load when starting/restarting a training run. (Originally there was also a refactor of how our different checkpoint paths are processed, but I seperated this out for now)

Previously we had a quite brittle logic for this, with edge cases where metaseq would not load the correct checkpoint. For example here: #544

The new logic checks first all possible sources for checkpoints (restore-file, finetune-from, local checkpoints, nfs / azure checkpoints), and assigns them priority based on their progress in training and prefers local caches.

It then takes the most recent checkpoint and copies it to local disk.

To test this you need both metaseq / metaseq-internal PR's. Here: https://github.com/fairinternal/metaseq-internal/pull/842

I tested:

• multi-node and single node local start
• with and without nfs cloud upload path
• crashing a run and correct continuing
• with and without finetune-from
• with and without resume-file
• evals

What I didn't test yet, is if starting with an azure blob path is working.

[Xirider]

Will test this on azure soon.

[suchenzang]

Heads up: #646 will likely go in first since tests are passing there (after loss parity check is added). There will probably be merge conflicts after, but hopefully not too bad.

[suchenzang]

Nit on getting the checkpoint saving/uploading tests to pass (may need to modify to adjust to new logic, just need to be clear why the modification to the test is needed)