Mirrors/modelscope
1
0
Fork 0
mirror of https://github.com/modelscope/modelscope.git synced 2026-02-24 12:10:09 +01:00
Code Issues Packages Projects Releases Wiki Activity
1,842 Commits 267 Branches 107 Tags
dev/dataset_fix_version
Commit Graph

1842 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
wenmeng zhou
84070cc2f5 Merge pull request #336 from modelscope/master_merge_github_0619 
Master merge GitHub 0619 from internal
 2023-06-25 17:00:51 +08:00
wenmeng.zwm
674c67ce48 move data to lfs 2023-06-21 17:57:34 +08:00
tongmu.wh
f03c93cda5 add speaker diarization pipeline and improve some speaker pipelines 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12891685

* add new speaker diarization pipeline. improve the capability of speaker pipelines
 2023-06-21 17:56:05 +08:00
wenmeng.zwm
3724697b65 update test data 2023-06-21 17:30:19 +08:00
wenmeng.zwm
54132e9299 update data/test to newest version 2023-06-21 16:35:10 +08:00
xingjun.wang
cb73bd14c0 merge internal master 2023-06-21 10:32:08 +08:00
xingjun.wxj
0db0ec5586 Merge code from github 
1. Merge(add) daily regression from github PR (daily_regression.yaml)
2. Add lora stable diffusion from github PR
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13010802
* fix: device arg not work, rename device to ngpu (#272)

* Correcting the lora stable diffusion example script (#300)

* add vad model and punc model in README.md 

add vad model and punc model

* Merge pull request #302 from modelscope/langgz-patch-1

add vad model and punc model in README.md

* add 1.6

* modify ignore

* Merge pull request #307 from modelscope/dev_rs_16

Merge release 1.6

* undo datetime to 2099

* Merge pull request #311 from modelscope/fix_master_version

undo datetime to 2099

* add daily regression workflow

* modify workflow name

* fix cron format issue

* lora trainer

* Merge pull request #315 from liuyhwangyh/add_regression_workflow

add daily regression workflow
 2023-06-21 10:22:06 +08:00
lanjinpeng.ljp
bc5c16aa10 support third_party key in pipeline 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12940228

* support third_party key in pipeline
 2023-06-20 19:25:24 +08:00
mulin.lyh
fa7562fd96 [to #50384694]fix: ci hang when build torch extension 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12976382
 2023-06-19 17:56:04 +08:00
hemu.zp
ad5d1aeb62 Fix gpt3 finetune nan 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12981998
* fix gpt3 finetune nan
 2023-06-19 11:25:27 +08:00
xingjun.wxj
cc3c384d5e Fix issues for downloading mplug-youku dataset 
1. Optimize downloading meta-csv files for large-scale dataset like mPLUG-youku (> 1GB for meta csv mapping)
2. Add head and overall progress bar for NativeIterableDataset
3. Modify the try-catch info for oss_utils
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12952842
 2023-06-15 15:42:21 +08:00
suluyan.sly
de491cfe6b [to #48855014] llama finetune + deepspeed 
1. llama base finetune：使用trainer从llama finetune至alpaca，效果确认并提供example
2. deepspeed通用性完善：mpu解耦；deepspeed的训练信息支持从ms log透出（目前ms打印的log有误）；支持从modelscope configuration.json 进行 deepspeed config 配置；deepspeed optimizer和lr_scheduler 初始化支持；解决deepspeed和ddp同时使用报错；解决保存ckpt时报错
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12651323
 2023-06-14 10:00:23 +08:00
zhifu.gzf
8b4e9dcdfb ngpu bug and rm easyasr 
修复ngpu指定无效的问题；移除easyasr，全部涉及模型都下架了；将funasr版本限制为>=0.6.0
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12933049
* ngpu bug and rm easyasr
 2023-06-13 16:36:21 +08:00
mulin.lyh
d6bf9e80fb [to #50129142]feat: support pre build torch extension build image, first extension megatron_util 2023-06-12 11:03:28 +08:00
hemu.zp
96c2d42f09 Add StreamingMixin 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12445731
* StreamingMixin poc

* update design

* Merge branch 'master' into feat/StreamingMixin

* add dicstr

* make postprocessor input consistent
 2023-06-08 19:40:14 +08:00
wucong.lyb
a77d390238 fix extra_args 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12839071
* fix extra_args

* update extra_args
 2023-06-08 19:39:05 +08:00
hejunjie.hjj
ae672b73df bug fix for single human parsing 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12869114
* fix bug: single human parsing, 当src_h, src_w相等时，new_h, new_w = 0, 0, 出现bug
 2023-06-08 19:36:18 +08:00
zsl01670416
75c90d8a98 add model revision in training_args and modify dataset loading in finetune text classification 
1.add parameter model_revision in training_args.py.
2.add parameter model_revision in kwargs for finetune_text_classification.py and finetune_text_generation.py.
3.modify dataset loading in finetune_text_classification.py for flex training.
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12869552
* add model revision in training_args and modify dataset loading in finetune text classification
 2023-06-08 19:36:05 +08:00
mulin.lyh
25a7398aea docker file py38 and py37 compatible merge 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12722824
* debug

* add missing deps

* add stanza

* fix numpy issue

* numpy version

* tf case

* numpy <= 1.22.0

* fix tf 1.15.5 case

* add python38 and python37 compatible docker file

* cv add fvcore

* add easycv

* remove debug code

* fix style issue

* fix python3.7 python3.8 compatible issue

* comment

* modify run config

* scipy<=1.7.3

* revert scipy<=1.7.3

* fix compatible bug

* fix compatible bug

* add easycv

* update version: 1.6.1rc0
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12754942
fix python3.8 and python3.8 compatible issue

* numpy version change

* numpy version

* numpy version

* fix numpy version

* fix numpy version

* update version: 1.6.1rc0
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12754942
fix python3.8 and python3.8 compatible issue

* restore setup.py

* restore setup.py

* add build base image

* fix style

* numpy version fix

* optimize build image, split base and modelscope image

* fix style issue

* modify example path

* modify examples folder

* add SETUPTOOLS_USE_DISTUTILS=stdlib comments
 2023-06-08 10:01:58 +08:00
pangda
73f8d39897 Add ner example for flextrain 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12835095
* update named_entity_recognition example
 2023-06-05 15:17:45 +08:00
liuyhwangyh
66fc614e15 Merge pull request #315 from liuyhwangyh/add_regression_workflow 
add daily regression workflow
 2023-06-03 06:14:52 +08:00
liuyhwangyh
2498bb60b6 fix cron format issue 2023-06-02 16:01:36 +08:00
liuyhwangyh
d61529dfa5 modify workflow name 2023-06-01 14:25:02 +08:00
mulin.lyh
8dc5246fdb add daily regression workflow 2023-06-01 09:36:06 +08:00
xixing.tj
1b7e0f50f4 add ocr detection new model db-nas 
新增5M的DB-NAS ocr detection 文字检测模型
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12760623
* add ocr detection new model db-nas

* add comment
 2023-05-31 21:32:46 +08:00
hemu.zp
898e3a42eb Fix bug for DeepspeedHook.register_processor 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12686921
 2023-05-31 21:29:36 +08:00
yuanzhi.zyz
10c39b5ce1 add new ocr recognition model (LightweightEdge) and some functions 
1. 增加了新轻量化端侧识别模型 LightweightEdge，并把原来CRNN和ConvNextViT的代码整理了
2. 增加batch inference支持
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12787905
 2023-05-31 21:16:22 +08:00
Xingjun.Wang
5634a18a58 Merge pull request #311 from modelscope/fix_master_version 
undo datetime to 2099
 2023-05-30 14:27:16 +08:00
xingjun.wang
6611445bab undo datetime to 2099 2023-05-30 14:04:12 +08:00
zhangzhicheng.zzc
c7aede97ba add pattern to skip easycv.thirdparty 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12785287

* add pattern to remove

* update pattern to remove

* avoid import with build
 2023-05-30 11:16:39 +08:00
yuze.zyz
003c33648a Fix CI: test merge dataset failed 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12771691
* fix CI

* add teardown
 2023-05-29 10:32:33 +08:00
xingjun.wxj
5b99bb24fe update version: 1.6.1rc0 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12754942
 2023-05-25 20:28:36 +08:00
yuze.zyz
e2a20fd817 fix merge_hook 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12718423
* fix merge_hook
 2023-05-25 20:05:04 +08:00
yuze.zyz
a7a3eb5dc5 Support flex train feature 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12737668
 2023-05-25 19:39:24 +08:00
yuze.zyz
c08b924968 Fix ner tokenizer which won't accept kwargs 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12664244
* fix slow tokenizer
 2023-05-25 18:55:47 +08:00
feiyu.gfy
f82361d71b 修正lineless_table_recognition功能遇到空白图片崩溃的bug 
用户反馈lineless_table_recognition功能遇到空白图片会发生崩溃。
增加处理机制以防止崩溃。

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12741201
* fix bug
 2023-05-25 18:28:05 +08:00
lee.lcy
a524e01e47 feat: support onnx export for domain_specific_object_detection 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12682688
 2023-05-25 10:44:09 +08:00
xingjun.wxj
f2640a5a12 fix private dataset auth issue 
1. Fix private datasets auth issue
2. Add arg: token (optional) in MsDataset.load() for FlexTrain
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12721569
 2023-05-24 19:48:20 +08:00
Xingjun.Wang
92e12b5f19 Merge pull request #307 from modelscope/dev_rs_16 
Merge release 1.6
 2023-05-24 11:12:18 +08:00
xingjun.wang
d3fa166fe0 modify ignore 2023-05-23 21:21:15 +08:00
xingjun.wang
48c0d2a9af add 1.6 v1.6.0 2023-05-22 10:53:18 +08:00
yuze.zyz
d38fa563a8 Re commit lossing files for flextrain 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12674746
 2023-05-19 10:48:33 +08:00
mulin.lyh
6abdce929d fix py38 issue 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12660340
 2023-05-17 22:11:23 +08:00
chenyafeng.cyf
f6ea3eadea eres2net 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12640199
 2023-05-16 22:28:20 +08:00
yuze.zyz
c6ef9024d7 Fix distributed hook to lazyimport; fix an import bug 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12638653
 2023-05-16 21:37:12 +08:00
yuze.zyz
38fa4cceed fix transformer example and fix some bugs 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12626375
 2023-05-16 15:27:17 +08:00
zsl01670416
71a80173e5 modify examples text_classification, text_generation, token_classification for improved trainer 
1.text_classification/run_train.sh
2.text_generation/run_train_mt5.sh, run_train_palm.sh
3.token_classification/finetune_token_classification.py, run_train_mgeo.sh, run_train_structbert.sh
above files were modified to adapt to improved trainer
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12629010
 2023-05-16 14:32:43 +08:00
hemu.zp
5804ad2dc1 update multi_modal_embedding example 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12626062
 2023-05-16 14:31:26 +08:00
zhifu.gzf
6c8c3a53f8 fix bug in **kwargs duplicated 
修复**kwargs中参数重复的bug
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12633542
 2023-05-16 13:49:01 +08:00
wenmeng.zwm
3ac31290f2 add description for modelscope 2023-05-15 19:26:21 +08:00