Mirrors/modelscope
1
0
Fork 0
mirror of https://github.com/modelscope/modelscope.git synced 2025-12-25 12:39:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
1,243 Commits 263 Branches 106 Tags
075eb1eddb84d67804b009bb343e973fe6c0bfde
Commit Graph

1243 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
zhangzhicheng.zzc
075eb1eddb [to #47017903] add file io with lock 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11402082
 2023-01-12 13:22:47 +08:00
hemu.zp
06296c1819 [to #42322933] Fix evaluation oom 
Add merge method for all metrics, parallel metrics can be merged when using data parallel. No longer save all data in the evaluation process to avoid oom.

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11399082
 2023-01-12 13:02:54 +08:00
xianzhe.xxz
393aa01e2b 支持DAMO-YOLO系列模型的Finetune功能。 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11249980

* add tinynas-detection trainer, evaluater and dataloader.

* add timmer and general torch dist tools.

* replace loguru with modelscope standard logger.

* merge duplicate tinynas-detection model files.

* add compatibility of json config files.
 2023-01-12 11:08:17 +08:00
mulin.lyh
f540e5f571 [to #47239563]feat: add local model file check 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11321065
 2023-01-12 10:06:12 +08:00
ly261666
66f7134db8 [to #42322933] Add FLRGB Face Liveness RGB Model 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11264617
 2023-01-12 10:05:56 +08:00
klayzhang.zb
f9a01acecf [to #42322933] text-error-correction support batch inference 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11398895
 2023-01-12 09:33:56 +08:00
bin.xue
78f812dbb6 [to #42322933] add speech separation finetune 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11379892
 2023-01-12 07:02:46 +08:00
jiangyu.xzy
3989b8da32 asr inference: support new models, punctuation, vad, sv 
asr推理支持新模型，以及支持标点后处理、端点检测、说话人确认
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11366442

    * support asr new models & vad-punc models

* fix conflict

* fix format

* Merge branch 'master' into asr/asr_inference

* add punc pipeline

* add punc model

* add sv infer

* fix format

* Merge branch 'master' into asr/asr_inference

* fix sv pipeline

* remove useless comments

* fix asr_test

* fix format

* fix output format

* change test level of models which would be removed later.
 2023-01-11 23:28:14 +08:00
gongbiao.gb
5debc88edf add vop retrieval 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11302991
 2023-01-11 22:22:12 +08:00
huizheng.hz
466200f355 NAFNet Image Deblurring pipeline and finetune support 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11300932

* fix psnr/ssim metrics for NAFNet (image denoise)

* add subset_name when loading dataset (NAFNet image denoising)
 2023-01-11 22:18:03 +08:00
pangda
ae5caf584a [to #42322933] add UT for adaseq span-based ner pipeline 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11398124
 2023-01-11 22:13:03 +08:00
hemu.zp
a277b343af [to #42322933] Add beam search and pair finetune for GPT-3 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11397726

* test finetune weather

* support ppl and generation metrics
 2023-01-11 22:04:11 +08:00
ly103369
0e54e80b23 add Damo Chinese Stable Diffusion and fix bugs in DiffusersPipeline 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11386214

* chinese sd & fix sdwrapper

* impl preprocess and postprocess in DiffusersPipeline

* add accelerate requirements
 2023-01-11 21:49:58 +08:00
zhangzhicheng.zzc
b1b65c2597 add megatron bert 2023-01-11 21:45:42 +08:00
shenweichao.swc
1f9d6e51f6 [to #42322933] Add panovit-layout-estimation-pipeline 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11234460
 2023-01-11 20:43:09 +08:00
zhicheng.sc
962cd4258a change demo_service to is_cvt_h264 
按照前端和后端同学要求，将在前端传入的参数名称更改为有具体语义和可复用的变量名
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11394996

    * change demo_service to is_cvt_h264
 2023-01-11 20:07:35 +08:00
wenmeng.zwm
8b03375702 fix create logger with module file path and avoid import mmcv in pipeline base 
1. fix create logger with module file path
2. avoid import mmcv in collate_fn

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11397356
 2023-01-11 19:55:42 +08:00
shuying.shu
bfc4f713e5 adjust output for demo service 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11383023
 2023-01-11 16:56:04 +08:00
fengyutong.fyt
1e5df96b46 [to #42322933] Add vision middleware to maas lib 2023-01-11 15:16:22 +08:00
zhangzhicheng.zzc
42898badf7 [to #42322933] update ast_index logic 2023-01-11 10:43:56 +08:00
pangda
346af6773f support plugin mechanism for second-party/third-party modules 2023-01-11 10:35:09 +08:00
shengzhe.sz
9667548d9e add panorama_depth_estimation 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11366392

* add panorama_depth_estimation: pipeline, model, test

* modelhub：https://modelscope.cn/models/damo/cv_unifuse_panorama-depth-estimation/summary
 2023-01-11 10:23:24 +08:00
zhangzhicheng.zzc
637c2c69e8 fix best ckpt saver not actually save best ckpt error 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11386286
 2023-01-11 07:10:25 +08:00
zhangzhicheng.zzc
fd24a243e6 token classfication model output unified 
Before this commit, different token classification models return the logits with various format, such as logits with speical token, with out speical token or predicts squeezed ignored of mask label.
After the commit, all of the results return the logits or predicts with respect of mask label.

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11370609
 2023-01-10 23:13:29 +08:00
liaojie.laj
a2b8d95188 modify video reader in pipeline 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11378034
 2023-01-10 23:01:57 +08:00
wenmeng.zwm
ed859e5274 Title: merge master-github and fix conflict 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11370549
 2023-01-10 14:03:08 +08:00
mulin.lyh
04b9704b48 [to #47190649]fix: merge v1.1 docker file 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11371422

    * v1.1 docker file modify
 2023-01-10 14:02:15 +08:00
tanfan.zjh
62e575a376 faq问答模型支持finetune/faq问答模型支持多语言 
- faq问答模型支持finetune
- faq问答模型支持多语言
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11343276
 2023-01-10 13:59:40 +08:00
guangpan.cd
03cce308c7 Integrate StabeDiffusionPipeline from diffusers into MaaS-lib 
Integrate `StabeDiffusionPipeline` from [`diffusers`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py) into MaaS-lib.
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11344883
 2023-01-10 13:54:52 +08:00
wenmeng.zwm
9ce750f4a9 merge master-github and fix conflict 2023-01-10 11:12:37 +08:00
maojialiang.mjl
73066fe04c [to #42322933] Add cv-bnext-image-classification-pipeline to maas lib 
add binary quantization classification to maas lib
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11257734
 2023-01-10 07:42:24 +08:00
xingjun.wxj
43edddd31f [to #42322933] msdataset module refactor and add 1230 features 
1. 优化本地数据集加载链路  
2. local与remote解耦，无网络环境下也可以使用SDK  
3. 升级hf datasets及其相关依赖到最新版(2.7.0+)
4. 解决元数据感知不到数据文件变更的问题  
5. 系统分层设计
6. 本地缓存管理问题  
7. 优化error log输出信息  
8. 支持streaming load	
* a. 支持数据文件为zip格式的streaming
* b. 支持Image/Text/Audio/Biodata等格式数据集的iter
* c. 兼容训练数据在meta中的历史数据集的streaming load
* d. 支持数据文件为文件夹格式的streaming load

9. finetune任务串接进一步规范
* a. 避免出现to_hf_dataset这种使用，将常用的tf相关的func封装起来  
* b. 去掉了跟hf混用的一些逻辑，统一包装到MsDataset里面

10. 超大数据集场景优化
* a. list oss objects: 直接拉取meta中的csv mapping，不需要做 list_oss_objects的api调用（前述提交已实现）
* b. 优化sts过期加载问题（前述提交已实现）

11. 支持dataset_name格式为：namespace/dataset_name的输入方式

参考Aone链接： https://aone.alibaba-inc.com/v2/project/1162242/task/46262894
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11264406
 2023-01-10 07:01:34 +08:00
liaojie.laj
fcf6e6431f submit video frame interpolation model 
增加视频插帧模型
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11188339
 2023-01-10 06:57:19 +08:00
shuying.shu
338a5a4994 fix bug for batch inference 
具体bug见https://github.com/modelscope/modelscope/issues/66
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11350862
 2023-01-10 06:11:54 +08:00
ryan.yy
c77213d919 图像换脸模型上MaaS 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11347556
 2023-01-10 05:45:55 +08:00
huizheng.hz
340a14a456 save a video with h264 vcodec for video_super_resolution pipeline 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11327516
 2023-01-09 22:23:55 +08:00
pengteng.spt
d3eb9a8401 speedup kws ctc decoder 2023-01-09 22:05:16 +08:00
yuze.zyz
672a4ba107 Refactor tinynas objectdetection & img-classification 
Refactor tinynas model & pipeline:
1. Move preprocess method out of model to image.py
2. Pipeline calls the model.__call__ method instead of inference method
3. Remove some obsolete code
4. Add a default preprocessor to preprocessor.py instead of change config in modelhub.
5. Standardize the return value of model

Refactor general image classification pipeline:
1. Change the preprocessor build method of ofa to avoid dependencies between multi-modal and cv.
2. Move preprocess method out of pipeline to image.py
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11185418
 2023-01-09 21:33:42 +08:00
zhicheng.sc
2cb89609f0 Add video stabilization model 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11204574

* add video stabilization model
 2023-01-09 21:23:26 +08:00
yuze.zyz
e6320f29d3 Small features: 
1. Exporting: Support text-classification of bert and tensorflow2.0 models, test cases have been added.
2. Downloading of preprocessor.from_pretrained will ignores some large files which not needed by extension file name.
3. Move sentence-piece-preprocessor to the subclass of text-generation-preprocessor and keep the original name for compatibility.
4. Remove some useless codes in nlp-trainer and trainer.
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11206922
 2023-01-09 21:22:07 +08:00
jiaqi.sjq
453ff1dae3 [to #42322933] support byte input feature and refine fp implementations 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11338137
 2023-01-09 20:56:52 +08:00
pangda
9794fbf1c7 [to #42322933] token_cls default sequence_length: 128 -> 512 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11303820
 2023-01-09 20:30:11 +08:00
yichang.zyc
d78fe495ee add structure tasks: sudoku & text2sql 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11314581
 2023-01-09 15:53:49 +08:00
Yingda Chen
728a5b3a21 remove useless import 2023-01-08 22:05:28 -08:00
chong.zhang
f56742b9af add new asr model_id: speech_UniASR_asr_2pass-pt-16k-common-vocab1617-tensorflow1-offline, speech_UniASR_asr_2pass-pt-16k-com... 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11311291
 2023-01-09 09:32:37 +08:00
ly261666
ffeeae82c6 [to #42322933] remove dup contents 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11275055
 2023-01-09 09:31:57 +08:00
hemu.zp
871b345e79 [to #42322933] GPT-3 model supports batch input 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11322820
 2023-01-09 09:31:44 +08:00
james.wjg
c0c14177bc 增加一个 trainer 单元测试 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11277400
 2023-01-09 07:38:38 +08:00
lanjinpeng.ljp
b2a78b5ad0 支持视频多目标跟踪模型 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11249098
 2023-01-09 07:11:15 +08:00
zhangyanzhao.zyz
8f5dc7aea4 文本向量pipeline支持仅输入source sentences; 新增medical/ecom领域语义相关性/文本向量表示模型UT。 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11283935
 2023-01-09 07:02:56 +08:00