mirror of
https://github.com/modelscope/modelscope.git
synced 2025-12-17 00:37:43 +01:00
43a57fe110
support loading dataset for llama: 1.load dataset by MsDataset when parameters train dataset name and val dataset name were set. but there is no suitable dataset in hub. 2.load dataset by MsDataset when only parameter train dataset name was set, and then split into train dataset and validation dataset . 3.load dataset by MsDataset when user set parameter src_txt, which is a file path such as 'alpaca_data.json', and then split into training dataset and validation dataset. 4.load dataset by build dataset from file in flex training. Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13505335