I'm wondering if we can take the ZenFlow finetuning example, and extend this example into a test bed of different DeepSpeed technologies.
The ZenFlow finetuning example:
https://github.com/deepspeedai/DeepSpeedExamples/tree/master/training/DeepSpeed-ZenFlow/finetuning
The reason is that DeepSpeed is mostly config driven, which means with the same training script, it only takes a config file change to run DeepSpeed training in different mode.
How about put different config file in the directory, i.e. "z2_config.json", "z3_config.json", "z2_offload_config.json", "zenflow_config.json", etc. into the directory, then change the run.sh to make it take the config as input? This might be a good start point for anyone to quickstart deepspeed with a real example.
I'm wondering if we can take the ZenFlow finetuning example, and extend this example into a test bed of different DeepSpeed technologies.
The ZenFlow finetuning example:
https://github.com/deepspeedai/DeepSpeedExamples/tree/master/training/DeepSpeed-ZenFlow/finetuning
The reason is that DeepSpeed is mostly config driven, which means with the same training script, it only takes a config file change to run DeepSpeed training in different mode.
How about put different config file in the directory, i.e. "z2_config.json", "z3_config.json", "z2_offload_config.json", "zenflow_config.json", etc. into the directory, then change the run.sh to make it take the config as input? This might be a good start point for anyone to quickstart deepspeed with a real example.