Mirrors/modelscope
1
0
Fork 0
mirror of https://github.com/modelscope/modelscope.git synced 2025-12-25 12:39:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
1,766 Commits 263 Branches 106 Tags
88de9f78aa2a311f68e3158ae81c8b06cbde4e05
Commit Graph

1766 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
suluyan.sly
88de9f78aa [to #50538422]fix: support load from model id 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13092954
 2023-06-29 10:26:52 +08:00
yuze.zyz
e951598a82 Merge branch merge_master_github_0628 into master 
Title: Merge branch 'master-github' into merge_master_github_0628 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13104791
 2023-06-29 09:47:52 +08:00
wenmeng.zwm
0271b9c256 Merge branch 'master-github' into merge_master_github_0628 2023-06-28 20:27:34 +08:00
Wang Qiang
d8381bf9fd Stable diffusion examples of lora and dreambooth. (#341) 
* stable diffusion examples of lora and dreambooth

* pre-commit
 2023-06-28 20:13:28 +08:00
Wang Qiang
a018cd6107 Dreambooth method for finetuning stable diffusions (#339) 
* Copyright

* dreambooth

* dreambooth test trainer

* fix bugs

* pre-commit

---------

Co-authored-by: 翊靖 <yijing.wq@alibaba-inc.com>
 2023-06-28 20:10:28 +08:00
xingjun.wxj
0af8752024 Unfreeze datasets version setting 
1. Unfreeze datasets version setting
2. Fix some bugs in class NativeIterableDataset
3. Remove auto-converting (image/audio) object in col:FILE
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13080190
 2023-06-28 15:50:57 +08:00
mulin.lyh
48b62d8947 [to #50521098]fix: fix diffuser version conflict cv and multi-modal 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13079583
* [to #50521098]fix: fix diffuser version conflict cv and multi-modal
 2023-06-28 15:36:12 +08:00
mulin.lyh
1ea9b58447 fix torch2.0 compatible issue 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13086361

* fix face-aligment compatible

* fix torch2.0 compatible issue
 2023-06-28 14:15:48 +08:00
Wang Qiang
6942144ad7 Stable Diffusion model checkpoint export to onnx. (#340) 
* stable diffusion export onnx

* fix pre commit bugs

* fix bugs

* safety checker support

* test export stable diffusion
 2023-06-28 13:26:19 +08:00
yuze.zyz
8f18274f75 Add teardown for tests 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12643554

* add teardown for tests

* add teardown for dialog_modeling_trainer,document_grounded_dialog_generate_trainer,document_grounded_dialog_rerank_trainer,document_grounded_dialog_retrieval_trainer,training_args,translation_evaluation_trainer,translation_trainer
 2023-06-28 09:44:44 +08:00
mulin.lyh
eb0f0216c6 fix torch 2.x compatible issue 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13045011

* fix torch 2.x compatible issue

* fix torch 2.x compatible issue

* fix complex-valued input tensor matching the output from stft with return_complex=True.

* skip plugin test temporarily for modify torch version

* fix test_speech_signal_process.py compatible issue

* fix lint issue

* upgrade funasr to 0.6.5
 2023-06-27 14:40:51 +08:00
yuze.zyz
a58be34384 Add Lora/Adapter/Prompt and support for chatglm6B and chatglm2-6B 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12770413

* add prompt and lora

* add adapter

* add prefix

* add tests

* adapter smoke test passed

* prompt test passed

* support model id in petl

* migrate chatglm6b

* add train script for chatglm6b

* move gen_kwargs to finetune.py

* add chatglm2

* add model definination
 2023-06-27 14:38:18 +08:00
xingjun.wxj
1dbff6cb48 Support jsonl format in meta data 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13071970

* support jsonl in meta


* add UT and refine fetch_meta_files_from_url
 2023-06-27 11:58:19 +08:00
wenmeng.zwm
e9fba3c042 Merge branch merge_master_github_0626 into master 
Title: merge_master_github_0626 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13071322
 2023-06-26 23:31:22 +08:00
suluyan.sly
7e28ca226b [to #50334474] llama tuned model -> pipeline 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13071602
* prepare for inference
 2023-06-26 22:49:44 +08:00
wenmeng.zwm
6ab85256af to [#50529030] fix easycv lr hook error 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13069804
 2023-06-26 21:54:01 +08:00
wenmeng.zwm
341cdd20a8 add missing deepspeed_hook 2023-06-26 21:11:08 +08:00
wenmeng.zwm
9e51920fdb Merge branch 'master-github' into merge_master_github_0626 2023-06-26 21:04:05 +08:00
suluyan.sly
b6c40aaa87 [to #50538422] feat: make diff & recover for llama based weights 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13055390
 2023-06-26 19:41:19 +08:00
co63oc
21f59f317e Fix some words (#318) 2023-06-26 16:57:47 +08:00
wenmeng zhou
6dea1d5646 Fix/citest timeout (#308) 
* timeout for citest set to 240min

* update docker image

* fix ci template not packed in whl

* update docker image version to 1.6.1 and add python3.8 support

* randome choose a model for controlnet to avoid oom
 2023-06-26 11:23:10 +08:00
mulin.lyh
698c794070 [to #50537864]fix: fix select case issue 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13055098
 2023-06-25 22:44:29 +08:00
hemu.zp
1421629392 Support device_map for transformers model 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13048856

* support device_map

* support device_map for T5

* fix bug
 2023-06-25 20:15:11 +08:00
chenyafeng.cyf
29062d9f94 eres2net_aug v2 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13046524

* eres2net_aug v2
 2023-06-25 18:07:04 +08:00
wenmeng zhou
84070cc2f5 Merge pull request #336 from modelscope/master_merge_github_0619 
Master merge GitHub 0619 from internal
 2023-06-25 17:00:51 +08:00
wenmeng.zwm
674c67ce48 move data to lfs 2023-06-21 17:57:34 +08:00
tongmu.wh
f03c93cda5 add speaker diarization pipeline and improve some speaker pipelines 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12891685

* add new speaker diarization pipeline. improve the capability of speaker pipelines
 2023-06-21 17:56:05 +08:00
wenmeng.zwm
3724697b65 update test data 2023-06-21 17:30:19 +08:00
wenmeng.zwm
54132e9299 update data/test to newest version 2023-06-21 16:35:10 +08:00
xingjun.wang
cb73bd14c0 merge internal master 2023-06-21 10:32:08 +08:00
xingjun.wxj
0db0ec5586 Merge code from github 
1. Merge(add) daily regression from github PR (daily_regression.yaml)
2. Add lora stable diffusion from github PR
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13010802
* fix: device arg not work, rename device to ngpu (#272)

* Correcting the lora stable diffusion example script (#300)

* add vad model and punc model in README.md 

add vad model and punc model

* Merge pull request #302 from modelscope/langgz-patch-1

add vad model and punc model in README.md

* add 1.6

* modify ignore

* Merge pull request #307 from modelscope/dev_rs_16

Merge release 1.6

* undo datetime to 2099

* Merge pull request #311 from modelscope/fix_master_version

undo datetime to 2099

* add daily regression workflow

* modify workflow name

* fix cron format issue

* lora trainer

* Merge pull request #315 from liuyhwangyh/add_regression_workflow

add daily regression workflow
 2023-06-21 10:22:06 +08:00
lanjinpeng.ljp
bc5c16aa10 support third_party key in pipeline 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12940228

* support third_party key in pipeline
 2023-06-20 19:25:24 +08:00
mulin.lyh
fa7562fd96 [to #50384694]fix: ci hang when build torch extension 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12976382
 2023-06-19 17:56:04 +08:00
hemu.zp
ad5d1aeb62 Fix gpt3 finetune nan 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12981998
* fix gpt3 finetune nan
 2023-06-19 11:25:27 +08:00
xingjun.wxj
cc3c384d5e Fix issues for downloading mplug-youku dataset 
1. Optimize downloading meta-csv files for large-scale dataset like mPLUG-youku (> 1GB for meta csv mapping)
2. Add head and overall progress bar for NativeIterableDataset
3. Modify the try-catch info for oss_utils
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12952842
 2023-06-15 15:42:21 +08:00
suluyan.sly
de491cfe6b [to #48855014] llama finetune + deepspeed 
1. llama base finetune：使用trainer从llama finetune至alpaca，效果确认并提供example
2. deepspeed通用性完善：mpu解耦；deepspeed的训练信息支持从ms log透出（目前ms打印的log有误）；支持从modelscope configuration.json 进行 deepspeed config 配置；deepspeed optimizer和lr_scheduler 初始化支持；解决deepspeed和ddp同时使用报错；解决保存ckpt时报错
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12651323
 2023-06-14 10:00:23 +08:00
zhifu.gzf
8b4e9dcdfb ngpu bug and rm easyasr 
修复ngpu指定无效的问题；移除easyasr，全部涉及模型都下架了；将funasr版本限制为>=0.6.0
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12933049
* ngpu bug and rm easyasr
 2023-06-13 16:36:21 +08:00
mulin.lyh
d6bf9e80fb [to #50129142]feat: support pre build torch extension build image, first extension megatron_util 2023-06-12 11:03:28 +08:00
hemu.zp
96c2d42f09 Add StreamingMixin 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12445731
* StreamingMixin poc

* update design

* Merge branch 'master' into feat/StreamingMixin

* add dicstr

* make postprocessor input consistent
 2023-06-08 19:40:14 +08:00
wucong.lyb
a77d390238 fix extra_args 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12839071
* fix extra_args

* update extra_args
 2023-06-08 19:39:05 +08:00
hejunjie.hjj
ae672b73df bug fix for single human parsing 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12869114
* fix bug: single human parsing, 当src_h, src_w相等时，new_h, new_w = 0, 0, 出现bug
 2023-06-08 19:36:18 +08:00
zsl01670416
75c90d8a98 add model revision in training_args and modify dataset loading in finetune text classification 
1.add parameter model_revision in training_args.py.
2.add parameter model_revision in kwargs for finetune_text_classification.py and finetune_text_generation.py.
3.modify dataset loading in finetune_text_classification.py for flex training.
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12869552
* add model revision in training_args and modify dataset loading in finetune text classification
 2023-06-08 19:36:05 +08:00
mulin.lyh
25a7398aea docker file py38 and py37 compatible merge 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12722824
* debug

* add missing deps

* add stanza

* fix numpy issue

* numpy version

* tf case

* numpy <= 1.22.0

* fix tf 1.15.5 case

* add python38 and python37 compatible docker file

* cv add fvcore

* add easycv

* remove debug code

* fix style issue

* fix python3.7 python3.8 compatible issue

* comment

* modify run config

* scipy<=1.7.3

* revert scipy<=1.7.3

* fix compatible bug

* fix compatible bug

* add easycv

* update version: 1.6.1rc0
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12754942
fix python3.8 and python3.8 compatible issue

* numpy version change

* numpy version

* numpy version

* fix numpy version

* fix numpy version

* update version: 1.6.1rc0
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12754942
fix python3.8 and python3.8 compatible issue

* restore setup.py

* restore setup.py

* add build base image

* fix style

* numpy version fix

* optimize build image, split base and modelscope image

* fix style issue

* modify example path

* modify examples folder

* add SETUPTOOLS_USE_DISTUTILS=stdlib comments
 2023-06-08 10:01:58 +08:00
pangda
73f8d39897 Add ner example for flextrain 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12835095
* update named_entity_recognition example
 2023-06-05 15:17:45 +08:00
liuyhwangyh
66fc614e15 Merge pull request #315 from liuyhwangyh/add_regression_workflow 
add daily regression workflow
 2023-06-03 06:14:52 +08:00
liuyhwangyh
2498bb60b6 fix cron format issue 2023-06-02 16:01:36 +08:00
liuyhwangyh
d61529dfa5 modify workflow name 2023-06-01 14:25:02 +08:00
mulin.lyh
8dc5246fdb add daily regression workflow 2023-06-01 09:36:06 +08:00
xixing.tj
1b7e0f50f4 add ocr detection new model db-nas 
新增5M的DB-NAS ocr detection 文字检测模型
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12760623
* add ocr detection new model db-nas

* add comment
 2023-05-31 21:32:46 +08:00
hemu.zp
898e3a42eb Fix bug for DeepspeedHook.register_processor 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12686921
 2023-05-31 21:29:36 +08:00