Mirrors/modelscope
1
0
Fork 0
mirror of https://github.com/modelscope/modelscope.git synced 2025-12-25 04:29:22 +01:00
Code Issues Packages Projects Releases Wiki Activity
1,173 Commits 263 Branches 106 Tags
871b345e79a557f10fbc260db50abecefc2f8024
Commit Graph

1173 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
hemu.zp
871b345e79 [to #42322933] GPT-3 model supports batch input 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11322820
 2023-01-09 09:31:44 +08:00
james.wjg
c0c14177bc 增加一个 trainer 单元测试 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11277400
 2023-01-09 07:38:38 +08:00
lanjinpeng.ljp
b2a78b5ad0 支持视频多目标跟踪模型 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11249098
 2023-01-09 07:11:15 +08:00
zhangyanzhao.zyz
8f5dc7aea4 文本向量pipeline支持仅输入source sentences; 新增medical/ecom领域语义相关性/文本向量表示模型UT。 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11283935
 2023-01-09 07:02:56 +08:00
wenmeng.zwm
68534bf554 Revert "skip unifold ut due to MMseqs2 api error" 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11334131
 2023-01-09 06:56:30 +08:00
ziyuan.tw
9552a8533e add ConvNeXt model 
增加ConvNeXt模型和修复代码bug：模型需要输入BGR格式图像，但读取图片代码默认输出为RGB格式，造成归一化预处理错误，模型精度下降。
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11192762
 2023-01-09 06:56:05 +08:00
wenmeng.zwm
8f6a0f64e2 add support for eval configuration and fix logger problem 
1. add support for configuration for gpu_collect and cache_dir which is used for cpu result gathering, configuration example

```json
"evaluation": {
    "gpu_collect":  false,
    "cache_dir": "path/to/your/local/cache"
}
```

2. fix logger file missing  when log_file is passed to get_logger and add log_file for trainer
3.  automatically create work_dir in rank0 worker
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11342068

    * add support for configuration for tmpdir and gpu_collect
 2023-01-09 02:51:35 +08:00
mulin.lyh
aa541468d1 temp skip tests/pipelines/test_video_depth_estimation.py for download demo_kitti.gif failed 2023-01-08 10:28:35 +08:00
hemu.zp
e4be1dde72 [to #42322933] add ppl metric 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11303248
 2023-01-07 22:52:55 +08:00
xuangen.hlh
db654dab5b fix error of 'unexpected keyword argument device_id' 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11347364
 2023-01-07 03:25:54 +08:00
mulin.lyh
53a9342a29 skip tests/trainers/test_dialog_modeling_trainer.py 2023-01-06 14:41:49 +08:00
mulin.lyh
1ec601aea2 skip tests/trainers/test_dialog_intent_trainer.py for list model file 500 error 2023-01-06 09:15:50 +08:00
ni.chongjia
ac53ce3e36 modify format of itn_pipeline 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11257394

    * dev for asr itn inference pipeline

* add task interface

* add pipeline input

* add modemodelscope/pipelines/audio/itn_inference_pipeline.py

* add modelscope/pipelines/audio/itn_inference_pipeline.py

* modelscope/pipelines/audio/itn_inference_pipeline.py

* update modelscope/pipelines/audio/itn_inference_pipeline.py

* modify itn_inference_pipeline.py

* modify itn_inference_pipeline.py

* modify itn_inference_pipeline.py

* remove itn.py

* modify some names

* add modify itn_inference_pipeline.py

* modify itn_inference_pipeline.py

* modify itn_inference_pipeline.py

* modify itn_inference_pipeline.py

* modify itn

* add tests/pipelines/test_inverse_text_processing.py

* modify asr_inference_pipeline.py for the original files

* modify format

* add commits files

* Merge remote-tracking branch 'origin' into remotes/origin/asr/itn_nichongjia

* Merge remote-tracking branch 'origin' into remotes/origin/asr/itn_nichongjia

* modify the pipelines

* Merge branch 'master' into remotes/origin/asr/itn_nichongjia

* [to #47031187]fix: hub test suites can not parallel 
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11276872

    * [to #47031187]fix: hub test suites can not parallel

* google style docs and selected file generator 

ref: https://yuque.alibaba-inc.com/pai/rwqgvl/go8sc8tqzeqqfmsz
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11150212

    * google style docs and selected file generator

* merge

* Merge remote-tracking branch 'origin' into remotes/origin/asr/itn_nichongjia

* Merge branch 'master' into remotes/origin/asr/itn_nichongjia

* add requirements for fun_text_processing
 2023-01-05 16:36:17 +08:00
kangxiaoyang.kxy
2260dd45fa 1230-image-colorization 
submit new algorithm for image colorization and corresponding pipeline.
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11179627
 2023-01-05 15:02:49 +08:00
kaisong.sks
2d68c6772a 用户满意度分析推理 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11235766

    * add

* add

* Merge remote-tracking branch 'origin' into nlp_user-satisfaction-estimation

* add

* add

* add

* Merge branch 'master' into nlp_user-satisfaction-estimation

* add

* Merge branch 'master' into nlp_user-satisfaction-estimation

* add

* add

* Merge branch 'master' into nlp_user-satisfaction-estimation

* Merge branch 'master' into nlp_user-satisfaction-estimation

* Merge branch 'master' into nlp_user-satisfaction-estimation

* Merge remote-tracking branch 'origin' into nlp_user-satisfaction-estimation

* add

* Merge branch 'master' into nlp_user-satisfaction-estimation
 2023-01-05 10:58:36 +08:00
liugao.lg
6dd0bea98d ofa-finetune 支持训练推理配置文件一体化 
ofa的finetune的配置文档教程冗余，用户使用copy内容太多，修改复杂，提供简洁的finetune代码流程
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11282842
 2023-01-05 07:25:47 +08:00
changxu.ccx
60bd40742a [to #42322933] Add vldoc to maas lib 
Test
```python
python tests/pipelines/test_document_vl_embedding.py
```
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11199555
 2023-01-04 19:11:30 +08:00
caorongyu.cry
72c39fb161 add space-t trainer 
1. 增加fine-tuning流程
2. 增加evalution流程
3. 关联数据集nlp_convai_text2sql_pretrain_cn_trainset
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11276053

    * add space-t trainer

* revise for trainer

* Merge branch 'master' into dev/tableqa_finetune

* revise for trainer

* Merge remote-tracking branch 'origin' into dev/tableqa_finetune
 2023-01-04 09:46:37 +08:00
pangda
2dbc93a931 [to #42322933] add UT for chunking model 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11289061
 2023-01-04 04:41:00 +08:00
mulin.lyh
ab07dc5b5a google style docs and selected file generator 
ref: https://yuque.alibaba-inc.com/pai/rwqgvl/go8sc8tqzeqqfmsz
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11150212

    * google style docs and selected file generator
 2023-01-03 16:27:29 +08:00
mulin.lyh
0675bd5c88 [to #47031187]fix: hub test suites can not parallel 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11276872

    * [to #47031187]fix: hub test suites can not parallel
 2023-01-03 16:26:59 +08:00
bin.xue
0fdf37312f [to #42322933] feat:add speech separation pipeline 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11255740
 2023-01-03 13:18:44 +08:00
dadong.gxd
01c498cd14 add cv_casmvs_multi-view-depth-esimation_general 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11204285
 2023-01-03 08:24:41 +08:00
shouzhou.bx
4698051fa5 update task of hand detect 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11271647
 2023-01-02 12:14:49 +08:00
james.wjg
a12ef720a6 全景分割easycv接入支持finetune（mask2former-r50）- 12.30 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11179263
 2023-01-02 11:27:11 +08:00
mulin.lyh
41cd220e01 temp skip failed case 2022-12-30 19:24:19 +08:00
lee.lcy
ed28b849eb [to #42322933] add domain specific object detection models 
添加垂类目标检测模型。
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11265502
 2022-12-30 14:19:16 +08:00
hejunjie.hjj
d560291525 [to #42322933] add maskdino model (1230) 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11187240
 2022-12-30 07:43:57 +08:00
wenmeng.zwm
b8ec677739 add training args support and image classification fintune example 
design doc: https://yuque.antfin.com/pai/rwqgvl/khy4uw5dgi39s6ke

usage:
```python

    from modelscope.trainers.training_args import (ArgAttr, MSArgumentParser,
                                               training_args)


    training_args.topk = ArgAttr(cfg_node_name=['train.evaluation.metric_options.topk',
                                                'evaluation.metric_options.topk'],
                                 default=(1,), help='evaluation using topk, tuple format, eg (1,), (1,5)')
    training_args.train_data = ArgAttr(type=str, default='tany0699/cats_and_dogs', help='train dataset')
    training_args.validation_data = ArgAttr(type=str, default='tany0699/cats_and_dogs', help='validation dataset')
    training_args.model_id = ArgAttr(type=str, default='damo/cv_vit-base_image-classification_ImageNet-labels', help='model name')

    parser = MSArgumentParser(training_args)
    cfg_dict = parser.get_cfg_dict()
    args = parser.args
    
    train_dataset = create_dataset(args.train_data, split='train')
    val_dataset = create_dataset(args.validation_data, split='validation')

    def cfg_modify_fn(cfg):
        cfg.merge_from_dict(cfg_dict)
        return cfg

    kwargs = dict(
        model=args.model_id,          # model id
        train_dataset=train_dataset,  # training dataset
        eval_dataset=val_dataset,     # validation dataset
        cfg_modify_fn=cfg_modify_fn     # callback to modify configuration
    )

    trainer = build_trainer(name=Trainers.image_classification, default_args=kwargs)
    # start to train
    trainer.train()
```
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11225071
 2022-12-30 07:35:15 +08:00
xiaodongdeng.dxd
1e4a3dcbe5 add_api_docs_for_ofa 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11097614
 2022-12-30 01:13:21 +08:00
hemu.zp
fd0099c92d [to #42322933] Refactor megatron-util 
Rename import lib 'megatron' to 'megatron_util' and add error message for users when import failed.

Use initialize_megatron as a unified initialization entry in megatron-util, which can accept configuration input of ConfigDict in MaaS-lib.

Wrap the initialization process into the utils/megatron_utils.py file, add default parameters for the existing large model to be compatible with the uploaded configuration file.

The version of megatron_cfg currently supports v3 (default), v1 and moe.
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11231840
 2022-12-29 15:01:45 +08:00
ly261666
6583e6f398 [to #42322933] Add FLIR Face Liveness Model 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11250177
 2022-12-29 14:55:51 +08:00
pengteng.spt
cddebf567f add kws nearfield finetune 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11179425

* add kws nearfield finetune

* work on rank-0 only if evaluating

* split kaldi relevant code into runtime utils

* add evaluate but not files checking

* test evaluate on cpu

* add default value for cmvn_file
 2022-12-29 10:14:41 +08:00
dadong.gxd
42557b0867 add cv_pointnet2_sceneflow-estimation_general 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11201880
 2022-12-29 08:09:57 +08:00
yeqinghao.yqh
f7a7504782 Add HiTeA model for VideoQA and Caption (12.30) 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11201652
 2022-12-29 08:06:34 +08:00
hemu.zp
f58060b140 [to #42322933] add GPT-2 model 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11194200
 2022-12-29 07:59:40 +08:00
lee.lcy
e8a354d226 [to #42322933] add real-time human detection model 
add real-time human detection model
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11191113
 2022-12-29 07:58:45 +08:00
xingjun.wxj
6423445697 [to #42322933] Modify object_exists in oss utils download. 
1. Move self.bucket.object_exists to try-except block. 
        
CR Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11245983
 2022-12-28 19:59:02 +08:00
yichang.zyc
0c79b57fcc support batch infer 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11170755
 2022-12-28 12:17:36 +08:00
hemu.zp
addda1f613 [to #42322933] Fix saved checkpoint can't run with pipeline for gpt3 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11225426
 2022-12-28 06:37:54 +08:00
qianmu.ywh
59b7f411b8 add video depth estimation pipeline 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11189483
 2022-12-28 06:34:02 +08:00
wjq264216
e57424eaf0 [to #42322933] Add ocr-detection-vlpt-pipeline to maas lib 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11213036
 2022-12-28 06:26:15 +08:00
huizheng.hz
cb9f1bfb8d 1230-video-super-resolution 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11090960
 2022-12-28 06:18:50 +08:00
wenmeng.zwm
4ee2ad3617 skip unifold ut due to MMseqs2 api error 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11218838
 2022-12-26 22:37:45 +08:00
wenmeng.zwm
a01e29ceb0 fix check video type cv2.VideoCapture and add unittest 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11181951
 2022-12-26 09:59:59 +08:00
suluyan.sly
086f629fbc [to #42322933]fix a bug for plug inference 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11199968

]
 2022-12-26 02:58:17 +08:00
shuying.shu
048207d79b fix memory leak bug in eval 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11195714
 2022-12-25 08:40:19 +08:00
jiaqi.sjq
147da4b3d9 [to #41669377] tts pipeline change infer result to wav 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11191925
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11191925
 2022-12-24 09:36:02 +08:00
xingjun.wxj
85b9d656eb [to #42322933] Add auto reload sts token and bigdata args 
1. Add auto-reload sts token for oss download function.
2. Improve big-data (e.g. Uni-Fold-Data) downloading performance.

        
CR Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11188490
 2022-12-23 14:40:37 +08:00
shuying.shu
b680691c2c adjust pipeline output 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11136535
 2022-12-23 01:53:17 +08:00