Ulysses SP for HF Integration

[stas00]

This is the Deepspeed counterpart of snowflakedb/ArcticTraining#45 - as the new feature(s) require changes on both sides.

For PR reviewers:

Readiness status:

• Code
• Tests
• Docs - working on it

Features:

• add support for delaying grad addition via param.ds_grad_is_ready flag (used when performing tiled compute in an autograd function)
• add light sp-only mpu version (Jeff Rasley)
• improved debug
• added all_gather_object to dist
• UlyssesSPAttentionHF (port of UlyssesAttention from Megatron-Deepspeed plus modern MHA-variations)
• UlyssesSPDataLoaderAdapter - DL adapter to shard the normal DL batches to be used by UlyssesSPAttentionHF
• SequenceTiledCompute - generic autograd function to perform compute after tiling on the sequence dimension
• TiledMLP - a specific autograd function to perform tiled MLP (it's much easier to understand before trying to grok SequenceTiledCompute)
• added a differentiable _DimZeroAllToAll (Samyam Rajbhandari)
• torch-dist-check now allows torch.distributed.nn (which is needed since deepspeed's dist is not up to date with torch.distributed.nn)