ZeRO: Memory Optimizations toward Training Trillion Parameter Models

Preprints and otherRajbhandari et al. 2020Cite as [@rajbhandari2020zero]

Rajbhandari, Samyam, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. “ZeRO: Memory Optimizations toward Training Trillion Parameter Models.” arXiv:1910.02054 [cs.LG]. arxiv.org/abs/1910.02054

arXiv · PDF

BibTeX

@misc{rajbhandari2020zero,
  title = {{{ZeRO}}: {{Memory}} Optimizations toward Training Trillion Parameter Models},
  author = {Rajbhandari, Samyam and Rasley, Jeff and Ruwase, Olatunji and He, Yuxiong},
  year = 2020,
  eprint = {1910.02054},
  primaryclass = {cs.LG},
  archiveprefix = {arXiv}
}

Download