DeepSpeed
Microsoft open source library
DeepSpeed' is an open-source optimization library for the distributed training and inference of deep learning models using PyTorch.
01Library
The library is designed to reduce computing power and memory use and to train large distributed models with better parallelism on existing computer hardware. DeepSpeed is optimized for low latency, high throughput training. It includes the Zero Redundancy Optimizer (ZeRO) for training models with 1 trillion or more parameters. Features include mixed precision training, single-GPU, multi-GPU, and multi-node training as well as custom model parallelism. The DeepSpeed source code is licensed under Apache License and available on GitHub.
The team claimed to achieve up to a 6.2x throughput improvement, 2.8x faster convergence, and 4.6x less communication.
Sources and credits
This article is adapted from the Wikipedia article “DeepSpeed”, written by its contributors and licensed under CC BY-SA 4.0. Fathomly has changed the layout, removed citation markers, navigation and maintenance notices, and adjusted punctuation. This adapted version is shared under the same license. For references, see the original article.
Images, from Wikimedia Commons:
- DeepSpeed logo.svg by https://github.com/jeffra, MIT
Fathomly is not affiliated with or endorsed by the Wikimedia Foundation. Spotted a problem? Tell us.
