mirror of
https://github.com/modelscope/modelscope.git
synced 2025-12-23 19:49:24 +01:00
de491cfe6b
1. llama base finetune：使用trainer从llama finetune至alpaca，效果确认并提供example 2. deepspeed通用性完善：mpu解耦；deepspeed的训练信息支持从ms log透出（目前ms打印的log有误）；支持从modelscope configuration.json 进行 deepspeed config 配置；deepspeed optimizer和lr_scheduler 初始化支持；解决deepspeed和ddp同时使用报错；解决保存ckpt时报错 Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12651323
10 lines
282 B
Bash
10 lines
282 B
Bash
DATA_PARALLEL_SIZE=4
|
|
|
|
|
|
export PYTHONPATH=$PYTHONPATH:./
|
|
torchrun --nproc_per_node $DATA_PARALLEL_SIZE examples/pytorch/llama/finetune_llama.py \
|
|
--work_dir './tmp' \
|
|
--model 'skyline2006/llama-7b' \
|
|
--deepspeed 'default_offload_opt_param.json' \
|
|
--eval_interval 100
|