fix copytree python37 bug (#464)

* fix copytree python37 bug

* add copytree_py37 function

examples/pytorch/llm/README.md

@@ -15,14 +15,14 @@
 </p>
 
 ## Note!!!
-1. This README.md file is copied from [ms-swift README.md](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/README.md)
-2. This directory has been migrated to [ms-swift](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm), and the files in this directory are no longer maintained.
+1. This README.md file is **copied from** [ms-swift](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/README.md)
+2. This directory has been **migrated** to [ms-swift](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm), and the files in this directory are **no longer maintained**.
 
 ## Features
 1. supported sft method: lora, qlora, full, ...
 2. supported models: [**qwen-7b**](https://github.com/QwenLM/Qwen-7B), baichuan-7b, baichuan-13b, chatglm2-6b, llama2-7b, llama2-13b, llama2-70b, openbuddy-llama2-13b, ...
-3. supported feature: quantization, ddp, model parallelism(device map), gradient checkpoint, gradient accumulation steps, push to modelscope hub, custom datasets, notebook compatibility, tensorboard, warmup, lr scheduler, resume from ckpt, ...
-4. supported datasets: alpaca-en(gpt4), alpaca-zh(gpt4), finance-en, multi-alpaca-all, multi-alpaca-ar, multi-alpaca-de, multi-alpaca-es, multi-alpaca-fr, multi-alpaca-id, multi-alpaca-ja, multi-alpaca-ko, multi-alpaca-pt, multi-alpaca-ru, multi-alpaca-th, multi-alpaca-vi, code-en, instinwild-en, instinwild-zh, ...
+3. supported feature: quantization, ddp, model parallelism(device map), gradient checkpoint, gradient accumulation steps, push to modelscope hub, custom datasets, ...
+4. supported datasets: alpaca-en(gpt4), alpaca-zh(gpt4), finance-en, multi-alpaca-all, code-en, instinwild-en, instinwild-zh, ...
 
 ## Prepare the Environment
 ```bash
@@ -67,12 +67,3 @@ bash scripts/qwen_7b/full/infer.sh
 ## Extend Datasets
 1. If you need to extend the model, you can modify the `MODEL_MAPPING` in `utils/models.py`. `model_id` can be specified as a local path. In this case, `revision` doesn't work.
 2. If you need to extend or customize the dataset, you can modify the `DATASET_MAPPING` in `utils/datasets.py`. You need to customize the `get_*_dataset` function, which returns a dataset with two columns: `instruction`, `output`.
-
-
-## TODO
-1. Support multi-round
-2. RLHF
-3. more models: Qwen-7B-Chat (use same prompt)
-4. more datasets
-5. metrics
-6. ...

examples/pytorch/llm/README_CN.md

@@ -15,14 +15,14 @@
 </p>
 
 ## 请注意!!!
-1. 该README_CN.md拷贝自[ms-swift README_CN.md](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/README_CN.md)
-2. 该目录已经迁移至[ms-swift](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm), 此目录中的文件不再维护.
+1. 该README_CN.md**拷贝**自[ms-swift](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/README_CN.md)
+2. 该目录已经**迁移**至[ms-swift](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm), 此目录中的文件**不再维护**.
 
 ## 特性
 1. 支持的sft方法: lora, qlora, 全参数微调, ...
 2. 支持的模型: [**qwen-7b**](https://github.com/QwenLM/Qwen-7B), baichuan-7b, baichuan-13b, chatglm2-6b, llama2-7b, llama2-13b, llama2-70b, openbuddy-llama2-13b, ...
-3. 支持的特性: 模型量化, DDP, 模型并行(device_map), gradient checkpoint, 梯度累加, 支持推送modelscope hub, 支持自定义数据集, 兼容notebook, tensorboard, warmup, lr scheduler, 断点续训, ...
-4. 支持的数据集: alpaca-en(gpt4), alpaca-zh(gpt4), finance-en, multi-alpaca-all, multi-alpaca-ar, multi-alpaca-de, multi-alpaca-es, multi-alpaca-fr, multi-alpaca-id, multi-alpaca-ja, multi-alpaca-ko, multi-alpaca-pt, multi-alpaca-ru, multi-alpaca-th, multi-alpaca-vi, code-en, instinwild-en, instinwild-zh, ...
+3. 支持的特性: 模型量化, DDP, 模型并行(device_map), gradient checkpoint, 梯度累加, 支持推送modelscope hub, 支持自定义数据集, ...
+4. 支持的数据集: alpaca-en(gpt4), alpaca-zh(gpt4), finance-en, multi-alpaca-all, code-en, instinwild-en, instinwild-zh, ...
 
 
 ## 准备实验环境
@@ -68,11 +68,3 @@ bash scripts/qwen_7b/full/infer.sh
 ## 拓展数据集
 1. 如果你想要拓展模型, 你可以修改`utils/models.py`文件中的`MODEL_MAPPING`. `model_id`可以指定为本地路径, 这种情况下, `revision`参数不起作用.
 2. 如果你想要拓展或使用自定义数据集, 你可以修改`utils/datasets.py`文件中的`DATASET_MAPPING`. 你需要自定义`get_*_dataset`函数, 并返回包含`instruction`, `output`两列的数据集.
-
-## TODO
-1. 支持 多轮对话
-2. RLHF
-3. 支持更多的模型: Qwen-7B-Chat (使用相同的prompt)
-4. 支持更多的数据集
-5. 指标与评估
-6. ...

examples/pytorch/llm/llm_infer.py

@@ -103,7 +103,8 @@ def llm_infer(args: InferArguments) -> None:
         top_k=args.top_k,
         top_p=args.top_p,
         do_sample=True,
-        pad_token_id=tokenizer.eos_token_id)
+        pad_token_id=tokenizer.pad_token_id,
+        eos_token_id=tokenizer.eos_token_id)
     logger.info(f'generation_config: {generation_config}')
 
     if args.eval_human:

examples/pytorch/llm/llm_sft.py

@@ -2,7 +2,7 @@
 """
 conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia -y
 pip install sentencepiece charset_normalizer cpm_kernels tiktoken -U
-pip install transformers datasets -U
+pip install transformers datasets scikit-learn -U
 pip install matplotlib tqdm tensorboard torchmetrics -U
 pip install accelerate transformers_stream_generator -U
 

examples/pytorch/llm/utils/utils.py

@@ -301,6 +301,7 @@ def inference(input_ids: List[int],
     print(f'{tag}{tokenizer.decode(input_ids)}', end='')
     input_ids = torch.tensor(input_ids)[None].cuda()
     attention_mask = torch.ones_like(input_ids)
+    model.eval()
     generate_ids = model.generate(
         input_ids=input_ids,
         attention_mask=attention_mask,

modelscope/utils/checkpoint.py

@@ -3,6 +3,7 @@
 import io
 import os
 import re
+import sys
 import time
 from collections import OrderedDict
 from shutil import copytree, ignore_patterns, rmtree
@@ -17,6 +18,7 @@ from torch.optim.lr_scheduler import _LRScheduler
 from modelscope.fileio import File, LocalStorage
 from modelscope.utils.config import Config, JSONIteratorEncoder
 from modelscope.utils.constant import ConfigFields, ModelFile
+from modelscope.utils.file_utils import copytree_py37
 from modelscope.utils.logger import get_logger
 from modelscope.utils.torch_utils import is_master
 
@@ -622,7 +624,11 @@ def save_pretrained(model,
     ignore_file_set.add('.*')
     if hasattr(model,
                'model_dir') and model.model_dir is not None and is_master():
-        copytree(
+        if sys.version_info.minor >= 8:
+            copytree_func = copytree
+        else:  # == 7
+            copytree_func = copytree_py37
+        copytree_func(
             model.model_dir,
             target_folder,
             ignore=ignore_patterns(*ignore_file_set),

modelscope/utils/file_utils.py

@@ -1,7 +1,9 @@
 # Copyright (c) Alibaba, Inc. and its affiliates.
 
 import inspect
+import os
 from pathlib import Path
+from shutil import Error, copy2, copystat
 
 
 # TODO: remove this api, unify to flattened args
@@ -42,3 +44,76 @@ def read_file(path):
     with open(path, 'r') as f:
         text = f.read()
     return text
+
+
+def copytree_py37(src,
+                  dst,
+                  symlinks=False,
+                  ignore=None,
+                  copy_function=copy2,
+                  ignore_dangling_symlinks=False,
+                  dirs_exist_ok=False):
+    """copy from py37 shutil. add the parameter dirs_exist_ok."""
+    names = os.listdir(src)
+    if ignore is not None:
+        ignored_names = ignore(src, names)
+    else:
+        ignored_names = set()
+
+    os.makedirs(dst, exist_ok=dirs_exist_ok)
+    errors = []
+    for name in names:
+        if name in ignored_names:
+            continue
+        srcname = os.path.join(src, name)
+        dstname = os.path.join(dst, name)
+        try:
+            if os.path.islink(srcname):
+                linkto = os.readlink(srcname)
+                if symlinks:
+                    # We can't just leave it to `copy_function` because legacy
+                    # code with a custom `copy_function` may rely on copytree
+                    # doing the right thing.
+                    os.symlink(linkto, dstname)
+                    copystat(srcname, dstname, follow_symlinks=not symlinks)
+                else:
+                    # ignore dangling symlink if the flag is on
+                    if not os.path.exists(linkto) and ignore_dangling_symlinks:
+                        continue
+                    # otherwise let the copy occurs. copy2 will raise an error
+                    if os.path.isdir(srcname):
+                        copytree_py37(
+                            srcname,
+                            dstname,
+                            symlinks,
+                            ignore,
+                            copy_function,
+                            dirs_exist_ok=dirs_exist_ok)
+                    else:
+                        copy_function(srcname, dstname)
+            elif os.path.isdir(srcname):
+                copytree_py37(
+                    srcname,
+                    dstname,
+                    symlinks,
+                    ignore,
+                    copy_function,
+                    dirs_exist_ok=dirs_exist_ok)
+            else:
+                # Will raise a SpecialFileError for unsupported file types
+                copy_function(srcname, dstname)
+        # catch the Error from the recursive copytree so that we can
+        # continue with other files
+        except Error as err:
+            errors.extend(err.args[0])
+        except OSError as why:
+            errors.append((srcname, dstname, str(why)))
+    try:
+        copystat(src, dst)
+    except OSError as why:
+        # Copying file access times may fail on Windows
+        if getattr(why, 'winerror', None) is None:
+            errors.append((src, dst, str(why)))
+    if errors:
+        raise Error(errors)
+    return dst

tests/utils/test_file_utils.py

@@ -0,0 +1,49 @@
+import os
+import tempfile
+import unittest
+
+from modelscope.utils.file_utils import copytree_py37
+
+
+class TestCopyTree(unittest.TestCase):
+
+    def setUp(self):
+        self.tmp_dir = tempfile.TemporaryDirectory()
+        self.tmp2_dir = tempfile.TemporaryDirectory()
+        dir_path = self.tmp_dir.name
+        print(f'self.tmp_dir: {self.tmp_dir.name}')
+        print(f'self.tmp_dir2: {self.tmp2_dir.name}')
+        fnames = ['1.py', '2.py', '3.py']
+        self.folders = ['.', 'a', 'b', 'c']
+        folder_dirs = [
+            os.path.join(dir_path, folder) for folder in self.folders
+        ]
+        for folder in folder_dirs:
+            os.makedirs(folder, exist_ok=True)
+            for fname in fnames:
+                fpath = os.path.join(folder, fname)
+                with open(fpath, 'w') as f:
+                    f.write('hello world')
+
+        for folder in folder_dirs:
+            print(f'folder: {os.listdir(folder)}')
+
+    def tearDown(self):
+        self.tmp_dir.cleanup()
+        self.tmp2_dir.cleanup()
+
+    def test_copytree_py37_exist_ok_true(self):
+        copytree_py37(
+            self.tmp_dir.name, self.tmp2_dir.name, dirs_exist_ok=True)
+        copytree_py37(
+            self.tmp_dir.name, self.tmp2_dir.name, dirs_exist_ok=True)
+        dir_path = self.tmp2_dir.name
+        new_folder_dirs = [
+            os.path.join(dir_path, folder) for folder in self.folders
+        ]
+        for folder in new_folder_dirs:
+            print(f'new_folder: {os.listdir(folder)}')
+
+
+if __name__ == '__main__':
+    unittest.main()