Molière (Memory Optimization for paraLlel traIning of dEep neuRal nEtworks) is an Inria Associated Team between Hiepacs project team and the Center for Computational and Data-Intensive Science and Engineering at Skoltech
The main goal concerns the development of algorithms to better manage memory constraints and increase the scalability of DNN training algorithms, by combining Checkpointing and Tensor Train Decomposition techniques with data parallelism and model parallelism.
Research directions
- Re-materialization (Checkpointing)
- Tensor Train (TT) Decomposition
- MultiGrid Reduction In Time (MGRIT)
- Pipelined Model Parallelism
