Mirrors/modelscope
1
0
Fork 0
mirror of https://github.com/modelscope/modelscope.git synced 2026-02-24 20:19:51 +01:00
Code Issues Packages Projects Releases Wiki Activity
1,279 Commits 267 Branches 107 Tags
v1.2.0
Commit Graph

152 Commits

Author SHA1 Message Date
mxf164419
ebb9bb15f8 add easyrobust-models for image classification 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11372821
 2023-01-16 18:04:12 +08:00
jiangyu.xzy
742ad4b355 sv inference & asr trainer: add new inputs 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11424993

* add sv_inference for speaker embedding extraction

* add raw inputs support for speaker_verification

* add train meta_params
 2023-01-13 13:52:52 +00:00
hemu.zp
9b353fd10e [to #42322933] add requirements rouge 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11422746
 2023-01-13 02:31:56 +00:00
bin.xue
78f812dbb6 [to #42322933] add speech separation finetune 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11379892
 2023-01-12 07:02:46 +08:00
jiangyu.xzy
3989b8da32 asr inference: support new models, punctuation, vad, sv 
asr推理支持新模型，以及支持标点后处理、端点检测、说话人确认
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11366442

    * support asr new models & vad-punc models

* fix conflict

* fix format

* Merge branch 'master' into asr/asr_inference

* add punc pipeline

* add punc model

* add sv infer

* fix format

* Merge branch 'master' into asr/asr_inference

* fix sv pipeline

* remove useless comments

* fix asr_test

* fix format

* fix output format

* change test level of models which would be removed later.
 2023-01-11 23:28:14 +08:00
gongbiao.gb
5debc88edf add vop retrieval 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11302991
 2023-01-11 22:22:12 +08:00
ly103369
0e54e80b23 add Damo Chinese Stable Diffusion and fix bugs in DiffusersPipeline 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11386214

* chinese sd & fix sdwrapper

* impl preprocess and postprocess in DiffusersPipeline

* add accelerate requirements
 2023-01-11 21:49:58 +08:00
guangpan.cd
03cce308c7 Integrate StabeDiffusionPipeline from diffusers into MaaS-lib 
Integrate `StabeDiffusionPipeline` from [`diffusers`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py) into MaaS-lib.
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11344883
 2023-01-10 13:54:52 +08:00
xingjun.wxj
43edddd31f [to #42322933] msdataset module refactor and add 1230 features 
1. 优化本地数据集加载链路  
2. local与remote解耦，无网络环境下也可以使用SDK  
3. 升级hf datasets及其相关依赖到最新版(2.7.0+)
4. 解决元数据感知不到数据文件变更的问题  
5. 系统分层设计
6. 本地缓存管理问题  
7. 优化error log输出信息  
8. 支持streaming load	
* a. 支持数据文件为zip格式的streaming
* b. 支持Image/Text/Audio/Biodata等格式数据集的iter
* c. 兼容训练数据在meta中的历史数据集的streaming load
* d. 支持数据文件为文件夹格式的streaming load

9. finetune任务串接进一步规范
* a. 避免出现to_hf_dataset这种使用，将常用的tf相关的func封装起来  
* b. 去掉了跟hf混用的一些逻辑，统一包装到MsDataset里面

10. 超大数据集场景优化
* a. list oss objects: 直接拉取meta中的csv mapping，不需要做 list_oss_objects的api调用（前述提交已实现）
* b. 优化sts过期加载问题（前述提交已实现）

11. 支持dataset_name格式为：namespace/dataset_name的输入方式

参考Aone链接： https://aone.alibaba-inc.com/v2/project/1162242/task/46262894
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11264406
 2023-01-10 07:01:34 +08:00
jiaqi.sjq
453ff1dae3 [to #42322933] support byte input feature and refine fp implementations 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11338137
 2023-01-09 20:56:52 +08:00
yichang.zyc
d78fe495ee add structure tasks: sudoku & text2sql 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11314581
 2023-01-09 15:53:49 +08:00
lanjinpeng.ljp
b2a78b5ad0 支持视频多目标跟踪模型 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11249098
 2023-01-09 07:11:15 +08:00
ni.chongjia
ac53ce3e36 modify format of itn_pipeline 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11257394

    * dev for asr itn inference pipeline

* add task interface

* add pipeline input

* add modemodelscope/pipelines/audio/itn_inference_pipeline.py

* add modelscope/pipelines/audio/itn_inference_pipeline.py

* modelscope/pipelines/audio/itn_inference_pipeline.py

* update modelscope/pipelines/audio/itn_inference_pipeline.py

* modify itn_inference_pipeline.py

* modify itn_inference_pipeline.py

* modify itn_inference_pipeline.py

* remove itn.py

* modify some names

* add modify itn_inference_pipeline.py

* modify itn_inference_pipeline.py

* modify itn_inference_pipeline.py

* modify itn_inference_pipeline.py

* modify itn

* add tests/pipelines/test_inverse_text_processing.py

* modify asr_inference_pipeline.py for the original files

* modify format

* add commits files

* Merge remote-tracking branch 'origin' into remotes/origin/asr/itn_nichongjia

* Merge remote-tracking branch 'origin' into remotes/origin/asr/itn_nichongjia

* modify the pipelines

* Merge branch 'master' into remotes/origin/asr/itn_nichongjia

* [to #47031187]fix: hub test suites can not parallel 
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11276872

    * [to #47031187]fix: hub test suites can not parallel

* google style docs and selected file generator 

ref: https://yuque.alibaba-inc.com/pai/rwqgvl/go8sc8tqzeqqfmsz
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11150212

    * google style docs and selected file generator

* merge

* Merge remote-tracking branch 'origin' into remotes/origin/asr/itn_nichongjia

* Merge branch 'master' into remotes/origin/asr/itn_nichongjia

* add requirements for fun_text_processing
 2023-01-05 16:36:17 +08:00
mulin.lyh
ab07dc5b5a google style docs and selected file generator 
ref: https://yuque.alibaba-inc.com/pai/rwqgvl/go8sc8tqzeqqfmsz
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11150212

    * google style docs and selected file generator
 2023-01-03 16:27:29 +08:00
bin.xue
0fdf37312f [to #42322933] feat:add speech separation pipeline 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11255740
 2023-01-03 13:18:44 +08:00
james.wjg
a12ef720a6 全景分割easycv接入支持finetune（mask2former-r50）- 12.30 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11179263
 2023-01-02 11:27:11 +08:00
hemu.zp
fd0099c92d [to #42322933] Refactor megatron-util 
Rename import lib 'megatron' to 'megatron_util' and add error message for users when import failed.

Use initialize_megatron as a unified initialization entry in megatron-util, which can accept configuration input of ConfigDict in MaaS-lib.

Wrap the initialization process into the utils/megatron_utils.py file, add default parameters for the existing large model to be compatible with the uploaded configuration file.

The version of megatron_cfg currently supports v3 (default), v1 and moe.
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11231840
 2022-12-29 15:01:45 +08:00
dadong.gxd
42557b0867 add cv_pointnet2_sceneflow-estimation_general 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11201880
 2022-12-29 08:09:57 +08:00
jiaqi.sjq
8896087034 [to #42322933] support kantts infer and finetune 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11111331#tab=detail
 2022-12-20 10:45:34 +08:00
wenmeng.zwm
070ec00720 remove opencv-python from framework requirements and remove easynlp from nlp default requirements 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11119626
 2022-12-16 14:27:23 +08:00
wenmeng.zwm
e2bf864f63 update audio requirements to use funasr>=0.1.4 2022-12-07 11:37:38 +08:00
wenmeng.zwm
4dd99b8f6e Revert "move opencv dependency from framwork to cv " 
This reverts commit e970a6eb43.
 2022-11-30 18:29:03 +08:00
jiangyu.xzy
9bfc77c178 support asr new models 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10919277

* support new asr paraformer model

* support asr conformer model
 2022-11-30 17:08:35 +08:00
yichang.zyc
ebb9636179 fix 不必要的init和优化vqa的preprocessor 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10868091
 2022-11-28 23:12:23 +08:00
shiyi.zxh
b386a4ee50 adapt to different wav input 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10886461
 2022-11-28 17:48:10 +08:00
jiangyu.xzy
2b62084146 add funasr based asr inference 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10868583
 2022-11-25 17:49:24 +08:00
shiyi.zxh
7661470350 ofa增加asr任务 
ofa增加asr任务infer
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10761019
 2022-11-25 12:16:33 +08:00
james.wjg
541e460f8b add support for cv/language_guided_video_summarization 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10636269
 2022-11-16 15:20:26 +08:00
yingda.chen
e970a6eb43 move opencv dependency from framwork to cv 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10708368
 2022-11-12 20:09:48 +08:00
mulin.lyh
1ca24299da [to #45892407]fix: fix pytorch_lighting incompatible with taming-transformers-rom1504 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10604329

    * [to #45892407]fix: fix pytorch_lighting incompatible with taming-transformers-rom1504
 2022-11-02 13:51:59 +08:00
mulin.lyh
4429991646 Merge branch dev/msdataset_event_tracking into master 
Title: [to #42322933] add event tracking 

1. add event tracking for dataset downloading pv/uv
2. change datasets version: <=2.5.2
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10593016
 2022-11-01 19:36:49 +08:00
干劲
aecb88044e up 2022-11-01 17:44:37 +08:00
干劲
853e5235d5 fix requirements 2022-11-01 17:32:04 +08:00
干劲
30c8c27145 up requirements 2022-11-01 17:06:30 +08:00
班扬
79c44a6810 add event tracking 2022-11-01 15:41:01 +08:00
班扬
84032f90e3 add event tracking 2022-11-01 15:34:58 +08:00
liugao.lg
40b6770956 [to #42322933]fix ocr prepreocess & conflict 
修复ocr预处理逻辑不一致问题
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10581697
 2022-11-01 10:22:11 +08:00
Yingda Chen
3464324f6b [to #42322933] limit datasets version for now 2022-10-31 22:15:37 +08:00
wenmeng.zwm
e07f3cdbf5 remove fasttext 2022-10-29 15:05:26 +08:00
Yufeng
261c04b8b5 add Mglm (#5) 
* mglm init

* add mglm requirements

Co-authored-by: Yufeng <zhuyufeng@gmail.com>
Co-authored-by: wenmeng.zwm <wenmeng.zwm@alibaba-inc.com>
 2022-10-28 17:12:47 +08:00
ran.zhou
13f7e9ceca [to #42322933]SEA multilingual NLP (NER & word segmentation) 
添加东南亚小语种NLP支持，包括：
1. 针对泰语，越南语NER的预处理
2. 基于XLMR-CRF架构的分词模型和pipeline
3. 针对泰语分词的预处理

添加了相应pipeline的unittest
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10492404
 2022-10-26 14:52:22 +08:00
yuanzheng.yuanzhen
bab54bbce8 [to #42322933]support uni fold 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10481410
 2022-10-25 22:59:19 +08:00
yichang.zyc
6ddafb3218 [to #42322933]caption finetune done, add belu metric 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10318299
 2022-10-25 12:55:41 +08:00
mulin.lyh
7257f6c6fb [to #45631658]feat support eas deploy 
服务端文档链接(可能需要登录)：
https://test.modelscope.cn/api/v1/deployer/docs
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10478609
 2022-10-24 15:12:48 +08:00
shouzhou.bx
01d521dd78 [to #42322933]add face 2d keypoints finetune test case 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10421808

    * add face 2d keypoints & human wholebody keypoint finrtune test case
 2022-10-20 11:51:52 +08:00
shuying.shu
cb570d586c add referring video object segmentation pipeline 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10400324
 2022-10-18 16:10:10 +08:00
zhangzhicheng.zzc
7e7303a658 [to #42322933] remove fasttext from nlp requirements 2022-10-15 08:51:06 +08:00
Yingda Chen
deb847614a [to #42322933] limit espnet version 2022-10-14 21:59:52 +08:00
hemu
876058556d fix generate 2022-10-14 18:15:52 +08:00
Yingda Chen
355da866c5 [to #42322933] limit tranformers version temporarily 2022-10-14 18:07:29 +08:00