Can you provide a sample code for training with DeepSpeed ZeRO3?

#10

by SupercarryNg - opened May 17

May 17

When I try to use DeepSpeed ZeRO3 to do some fine-tuning training for a specific task, I can't specify the device map=“sequential”, how do I set it up correctly?

jerryli1981

Jun 11

https://github.com/alibaba/Pai-Megatron-Patch/blob/main/examples/deepseek_v2/README.md#Megatron-Core-MoE%E6%A8%A1%E5%9E%8B%E8%AE%AD%E7%BB%83%E6%B5%81%E7%A8%8B

SupercarryNg

Jun 11

Thanks a lot!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment