Mirrors/modelscope
1
0
Fork 0
mirror of https://github.com/modelscope/modelscope.git synced 2025-12-25 12:39:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
2,825 Commits 263 Branches 106 Tags
fix/patch_dynamic_module
Commit Graph

2825 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
wenmeng zhou
2809bdbff6 Feat/chatglm2 32k support (#433) 
* add support for chatglm2 32k

* fix
 2023-08-01 10:58:29 +08:00
suluyan.sly
dc17c7d6ee remove cv requirements: basicsr 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13471366
* rm req basicsr
 2023-07-31 18:20:23 +08:00
mulin.lyh
da2badaacd [to #51274424]fix: easycv extension and tinycudann conflict 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13471150
* fix easycv extension and tinycudann conflict
 2023-07-31 18:08:53 +08:00
wenmeng zhou
f2396ddb6a Merge pull request #432 from modelscope/merge_master-gitlab_0731 
Merge master gitlab 0731
 2023-07-31 17:14:41 +08:00
yuze.zyz
f856c04785 Ignore http error 
忽略log打印，防止在check_model_is_latest时打出信息使用户不明所以。
将log改为Error栈
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13463005
 2023-07-31 17:14:12 +08:00
wenmeng.zwm
7fbd6ecc40 Merge branch 'master-gitlab' into merge_master-gitlab_0731 2023-07-31 16:16:16 +08:00
wenmeng.zwm
48a39244f1 Merge branch master-merge-github-230728 into master 
Title: Merge branch 'master-github' into master-merge-github-230728 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13456249
 2023-07-31 16:09:07 +08:00
Xingjun.Wang
ef1b429bef Merge pull request #430 from XDUWQ/delete_nerf_log 
delete unuseful logs
 2023-07-31 14:48:39 +08:00
XDUWQ
4233916867 delete unuseful logs 2023-07-31 14:36:12 +08:00
suluyana
b68b90ba15 skip plugin 2023-07-30 00:30:30 +08:00
suluyana
9ece90ee84 skip plugin test case 2023-07-29 21:35:21 +08:00
suluyan.sly
d7cd2ce28e Merge branch 'master-github' into master-merge-github-230728 2023-07-29 10:42:54 +08:00
Jintao
312b63fe06 fix checkpoint, same device bug (#427) 2023-07-29 00:06:27 +08:00
zsl01670416
847607ab66 Merge branch 'debug_chatglm6b_json_dataset' fix conflict between hf dataset and to_hf_dataset 
The type of dataset built from file is hf dataset, which can not use function to_hf_dataset.
 2023-07-28 19:12:21 +08:00
suluyan.sly
05e1357c32 Merge branch 'master-github' into master-merge-github-230728 2023-07-28 16:40:34 +08:00
tastelikefeet
972298813b 1. add check model for training 2. add model_dir for automodel (#422) 2023-07-28 16:34:02 +08:00
wenmeng.zwm
3b485d5835 fix plugin python module missing files 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13453749
* fix plugin python module missing files
 2023-07-28 16:14:56 +08:00
frozoul
2566d028cd cv/cv nerf 3d reconstruction 4k nerf damo (#389) 
* add 4k-nerf core files

* update configure file

* update dataloader and model path

* update unittest

* Delete test_4k.py

* update unittest

* update unittest

* update pre-commit

* update dataloader

* update cuda code path

* check with pre-commit

---------

Co-authored-by: zhongshu.wzs <zhongshu.wzs@alibaba-inc.com>
Co-authored-by: wenmeng zhou <wenmeng.zwm@alibaba-inc.com>
 2023-07-28 10:37:13 +08:00
tongmu.wh
475924a421 correct language recognition taks name 
modelscope平台同学最终定下语种识别任务名为speech-language-recognition，对应进行代码中的相关改动
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13444517
* correct language recognition taks name
 2023-07-27 21:03:04 +08:00
Xingjun.Wang
3bd395a0c9 Merge pull request #423 from modelscope/dev/del_redundancy_code 
del redundant code
 2023-07-27 20:53:01 +08:00
xingjun.wang
6748cd7370 del redundancy code in func: get_dataset_meta_file_list 2023-07-27 20:49:41 +08:00
Wang Qiang
34ea2b474a Upgrade stable diffusion version to a more powerful version 2.1 (#415) 2023-07-27 20:40:38 +08:00
suluyan.sly
76921561e6 Merge branch master-merge-github-230727 into master 
Title: Merge branch 'master-github' into master-merge-github-230727 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13442869
 2023-07-27 20:18:35 +08:00
mulin.lyh
a501f7a417 fix easycv cpu extension build issue, cpu not support 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13441612
* fix easycv cpu extension build issue, cpu not support
 2023-07-27 17:42:37 +08:00
suluyan.sly
1c6f5fe775 Merge branch 'master-github' into master-merge-github-230727 
Conflicts:
       examples/pytorch/baichuan/finetune_baichuan.py
       examples/pytorch/chatglm6b/finetune.py
 2023-07-27 17:29:27 +08:00
Wang Qiang
dca6143b8b support safetensors weight pipeline (#421) 2023-07-27 16:49:01 +08:00
tastelikefeet
9802dfe93b fix device error (#419) 2023-07-27 16:23:21 +08:00
mengyang.fmy
18f998a85c add text-to-360pano-image pipeline, mod cv requirements 
7月份计划上线的360全景图生成模型，自研

模型权重文件地址https://www.modelscope.cn/models/damo/cv_diffusion_text-to-360panorama-image_generation/summary


#### 依赖项说明

##### 由于要使用xformers，torch版本最好使用1.13.1
```
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116
```
##### 对应的diffusers和xformers版本如下
```
pip install -U diffusers==0.18.0
pip install xformers==0.0.16
pip install triton, accelerate, transformers
```

##### ModelScope Library 需要使用cv
```
pip install modelscope
pip install "modelscope[cv]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html
```

##### 此外，还需要安装第三方的一个库，Real-ESRGAN， 安装方法如下
```
# Install basicsr - https://github.com/xinntao/BasicSR
# We use BasicSR for both training and inference
pip install basicsr
# facexlib and gfpgan are for face enhancement
pip install facexlib
pip install gfpgan
pip install Pillow
pip install tqdm
pip install realesrgan==0.3.0
```
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13346430
* add text-to-360pano-image pipeline

* add text-to-360pano-image pipeline, mod cv requirements

* rm redundant files and cv requirements; add standard input and output definations

* fix diffusers==0.18.0 and run test

* fix diffusers==0.18.0 in multi-modal and run test again

* add model_revision='v1.0.0'

* fix yapf

* add trycatch for enabling xformers

* fix key error

* add install xformers in test/setup

* skip highres.fix in ci

* feat: Fix conflict, auto commit by WebIDE
 2023-07-27 11:33:39 +08:00
Wang Qiang
66cf72a75c Merge pull request #376 from XDUWQ/custom_diffusion 
Custom method for finetuning stable diffusion
 2023-07-27 10:41:38 +08:00
suluyana
534a020e93 Merge pull request #418 from tastelikefeet/feat/fix_device_split 
fix CI
 2023-07-27 10:03:41 +08:00
yuze.zyz
00c79428d5 fix 2023-07-26 21:30:38 +08:00
Jintao
4ca937d2ba support openbuddy-llama2-13b (#416) 2023-07-26 18:12:55 +08:00
Zackary Shen
ba4db97507 upload cv_nerf_3d-reconstruction_vector-quantize-compression (#407) 
* add vq_compression model

* add vq_compression model

* check pre-commit for lint test

* fix by flake8

* update

* update

* update

* the last update

* the laast update

* update test_level>=0

---------

Co-authored-by: 剑匣 <zackary.sz@alibaba-inc.com>
 2023-07-26 17:20:13 +08:00
XDUWQ
3a629bb2b4 pre 2023-07-26 16:58:58 +08:00
XDUWQ
99aa707995 fix bugs 2023-07-26 16:35:40 +08:00
XDUWQ
25d67a0b83 pre 2023-07-26 16:08:24 +08:00
XDUWQ
70da8b7809 fix bugs 2023-07-26 15:54:37 +08:00
XDUWQ
a7678bbb77 precommit 2023-07-26 15:06:42 +08:00
XDUWQ
c0b62cb0fb fix bugs 2023-07-26 15:03:46 +08:00
Xingjun.Wang
41c2c95083 dataset fix version (#413) 
* modify datasets version
 2023-07-25 22:18:25 +08:00
mulin.lyh
a780935317 torch1.11 and torch2.0.1 build script refactor 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13408751

* [to #51122016]feat: build image support torch1.x 2.x cuda11.3.0 11.7.1

* remove base  version

* env cudatookit-version

* build script merge

* torch 1.11.0 fix
 2023-07-25 21:15:00 +08:00
jiaqi.sjq
1f05acef88 fix pip install error when pysptk>=0.1.19 2023-07-25 21:13:31 +08:00
tongmu.wh
ba1a333ba6 add language recognition pipelines and models 
新增语种识别pipeline和model
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13385083
* add language recognition pipelines and models

* add a clustering method for speaker diarization

* define input and output type for language recognition
 2023-07-25 21:07:56 +08:00
tastelikefeet
0db3d1d53b Fix bug of amp and device_map (#397) 
* fix amp

* remove useless code

* Fix bug
 2023-07-25 19:28:00 +08:00
Jintao
f03898626e ckpt output directory ignore *.safetensors (#410) 
* ckpt output file ignore *.safetensors

* update
 2023-07-25 19:27:11 +08:00
hemu.zp
fc54593a56 fix baichuan eval and support sequence_length 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13404289
* fix baichuan eval

* support sequence_length and ppl

* fix typo

* fix bug for palm

* fix bug
 2023-07-25 19:10:45 +08:00
zsl01670416
9926ad685b support getting labels from dataset in sbert text classification and building dataset from file in chatglm-6b 
1.Add getting labels from dataset in "text_classificationfinetune_text_classification.py" to simplify user's operation in flex training. Parameters "--num_labels" and "--labels" were removed in "run_train.sh".
2.In "chatglm6b / finetune.py", building dataset from file  is necessary to support flex training.
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13382745
* support getting labels from dataset in sbert text classification and building dataset from file in chatglm-6b

* support getting labels from dataset in sbert text classification and building dataset from file in chatglm-6b

* remove repetitive labels in a concise manner of using set

* reserve parameter labels in finetune_text_classification

* Merge branch 'master' of http://gitlab.alibaba-inc.com/Ali-MaaS/MaaS-lib

reserve parameter labels in finetune_text_classification

* Merge branch 'support_text_cls_labels_chatglm_json'
reserve parameter labels in finetune_text_classification
 2023-07-25 19:02:32 +08:00
zeyinzi.jzyz
672c4899e9 add sd swift tuner 
SD-Tuner base on Swift (LoRA/Adapter/Prompt)
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13380798

* sd swift tuner

* fix pre-checker
 2023-07-25 19:00:49 +08:00
shuli.cly
526e1371f5 Merge the speaker-turn-detection codes, local test finished 
# Speaker Diarization Speaker-Turn Detection CR

和Dialogue-Detection一样，本模型是Speaker Diarization（`audio/speaker diarization`，语音/说话人日志）任务下的一个子模块。

本次提交的是基于文本进行判断的模型，本地模型的初始模型基于huggingface训练的，此提交中复用了部分 `nlp/token-classification` 模型的代码。为了方便后续维护以及与nlp方面代码解耦，在model、pipeline以及preprocessor中 **单独** 创建了相应模块并重新register。
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13364720
* std first commit

* local test pass for speaker-turn-detection

* update speaker-turn-detection pipeline task outputs format; update pipeline outputs; update test scripts
 2023-07-25 18:57:47 +08:00
hemu.zp
80f76ca475 Support stream output for transformers model 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13271136
* support stream for transformers model

* set test_level >= 2

* support hf model and chatglm2

* remove streaming_output for chatglm2
 2023-07-25 17:41:32 +08:00