Inria-Skoltech Moliere Associated Team

Molière (Memory Optimization for paraLlel traIning of dEep neuRal nEtworks) is an Inria Associated Team between Hiepacs project team  and the Center for Computational and Data-Intensive Science and Engineering at Skoltech

The main goal  concerns the development of algorithms to better manage memory constraints and increase the scalability of DNN training algorithms, by combining Checkpointing and Tensor Train Decomposition techniques with data parallelism and model parallelism.

Research directions

  • Re-materialization (Checkpointing)
  • Tensor Train (TT) Decomposition
  • MultiGrid Reduction In Time (MGRIT)
  • Pipelined Model Parallelism

Comments are closed.