Skip to content

Release MS-AMP v0.1.0

Compare
Choose a tag to compare
@tocean tocean released this 21 Apr 09:01
· 59 commits to main since this release
ac659d5

MS-AMP 0.1.0 Release Notes

MS-AMP package

  • Support the new FP8 feature that is introduced by latest accelerators (e.g. H100).
  • Speed up math-intensive operations, such as linear layers, by using Tensor Cores.
  • Speed up memory-limited operations by accessing one byte compared to half or single-precision.
  • Reduce memory requirements for training models, enabling larger models or larger minibatches.
  • Speed up communication for distributed model by transmitting lower precision gradients.
  • Support two optimization levels: O1 and O2.
  • Support two optimizers: Adam and AdamW.

Examples using MS-AMP