Mirrors/modelscope
1
0
Fork 0
mirror of https://github.com/modelscope/modelscope.git synced 2025-12-17 16:57:42 +01:00
Code Issues Packages Projects Releases Wiki Activity
1,649 Commits 263 Branches 106 Tags
release/1.6
Commit Graph

54 Commits

Author SHA1 Message Date
wenmeng.zwm
49e8d20560 update develop doc 2023-03-21 18:06:20 +08:00
yzhao
99e94bc2c2 Merge branch 'master-github' into master-merge-github20230310 2023-03-10 13:52:31 +08:00
xingjun.wxj
e02a260c93 Refactor the task_datasets module 
Refactor the task_datasets module:

1. Add new module modelscope.msdatasets.dataset_cls.custom_datasets.
2. Add new function: modelscope.msdatasets.ms_dataset.MsDataset.to_custom_dataset().
2. Add calling to_custom_dataset() func in MsDataset.load() to adapt new custom_datasets module.
3. Refactor the pipeline for loading custom dataset: 
	1) Only use MsDataset.load() function to load the custom datasets.
	2) Combine MsDataset.load() with class EpochBasedTrainer.
4. Add new entry func for building datasets in EpochBasedTrainer: see modelscope.trainers.trainer.EpochBasedTrainer.build_dataset()
5. Add new func to build the custom dataset from model configuration, see: modelscope.trainers.trainer.EpochBasedTrainer.build_dataset_from_cfg()
6. Add new registry function for building custom datasets, see: modelscope.msdatasets.dataset_cls.custom_datasets.builder.build_custom_dataset()
7. Refine the class SiameseUIETrainer to adapt the new custom_datasets module.
8. Add class TorchCustomDataset as a superclass for custom datasets classes.
9. To move modules/classes/functions:
	1) Move module msdatasets.audio to custom_datasets
	2) Move module msdatasets.cv to custom_datasets
	3) Move module bad_image_detecting to custom_datasets
	4) Move module damoyolo to custom_datasets
	5) Move module face_2d_keypoints to custom_datasets
	6) Move module hand_2d_keypoints to custom_datasets
	7) Move module human_wholebody_keypoint to custom_datasets
	8) Move module image_classification to custom_datasets
	9) Move module image_inpainting to custom_datasets
	10) Move module image_portrait_enhancement to custom_datasets
	11) Move module image_quality_assessment_degradation to custom_datasets
	12) Move module image_quality_assmessment_mos to custom_datasets
	13) Move class LanguageGuidedVideoSummarizationDataset to custom_datasets
	14) Move class MGeoRankingDataset to custom_datasets
	15) Move module movie_scene_segmentation custom_datasets
	16) Move module object_detection to custom_datasets
	17) Move module referring_video_object_segmentation to custom_datasets
	18) Move module sidd_image_denoising to custom_datasets
	19) Move module video_frame_interpolation to custom_datasets
	20) Move module video_stabilization to custom_datasets
	21) Move module video_super_resolution to custom_datasets
	22) Move class GoproImageDeblurringDataset to custom_datasets
	23) Move class EasyCVBaseDataset to custom_datasets
	24) Move class ImageInstanceSegmentationCocoDataset to custom_datasets
	25) Move class RedsImageDeblurringDataset to custom_datasets
	26) Move class TextRankingDataset to custom_datasets
	27) Move class VecoDataset to custom_datasets
	28) Move class VideoSummarizationDataset to custom_datasets
10. To delete modules/functions/classes:
	1) Del module task_datasets
	2) Del to_task_dataset() in EpochBasedTrainer
	3) Del build_dataset() in EpochBasedTrainer and renew a same name function.
11. Rename class Datasets to CustomDatasets in metainfo.py

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11872747
 2023-03-10 09:03:32 +08:00
chenxujun
20b3a679e7 Fix some words (#141) 2023-03-02 11:06:56 +08:00
wenmeng.zwm
677e49eaf3 update api doc 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11582587
 2023-02-10 07:48:11 +00:00
mulin.lyh
86128cd3a8 [to #47612577]feat: image supoort 3d models, python3.8 test version 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11542009


* mmcv-full<=1.7.0 for mmdet3d

* fix code style issue

* add speechbrain==0.5.7

* add compatible comments
 2023-02-07 02:28:20 +00:00
wenmeng.zwm
9ce750f4a9 merge master-github and fix conflict 2023-01-10 11:12:37 +08:00
mulin.lyh
ab07dc5b5a google style docs and selected file generator 
ref: https://yuque.alibaba-inc.com/pai/rwqgvl/go8sc8tqzeqqfmsz
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11150212

    * google style docs and selected file generator
 2023-01-03 16:27:29 +08:00
wenmeng.zwm
5a3d58ad49 Merge branch 'master-gitlab' into merge_master_internal_1207 2022-12-07 19:59:07 +08:00
wenmeng.zwm
82af080c18 upate git-lfs install instruction 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10963252

    * upate git-lfs install instruction
 2022-12-03 17:42:20 +08:00
Yingda Chen
33934fccf7 Merge pull request #40 from co63oc/patch-2 
Update develop.md

no code change, gated tests not required.
 2022-12-03 11:34:09 +08:00
chenxujun
ea5c2a2155 Update faq.md 
Fix typo
 2022-12-03 08:53:55 +08:00
chenxujun
79da49e078 Update develop.md 
Fix typo
 2022-12-03 08:50:51 +08:00
wenmeng.zwm
9f2de967c5 merge master and fix conflict 2022-11-16 18:10:39 +08:00
yingda.chen
4e4faa9a30 specifiy file encoding when open text for read 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10708723
 2022-11-14 14:16:08 +08:00
wenmeng.zwm
687766d9f8 refactor readme and remove internal links 2022-10-17 15:04:07 +08:00
wenmeng.zwm
0128d44517 [to #43115513] remove additional empty section header 2022-08-23 19:30:51 +08:00
Yingda Chen
2ebe68eded [to #42322933]remove docker build instruction 2022-08-08 22:44:30 +08:00
wenmeng.zwm
638cdc632e [to #43115513] update quick_start doc and add pai-easynlp back 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9655518

* update quick_start doc and add pai-easynlp back

* update doc with sndfile
 2022-08-05 15:31:32 +08:00
wenmeng.zwm
49192f94be [to #43726282] fix bugs and refine docs 
1. remove pai-easynlp temporarily due to its hard dependency on scipy==1.5.4
2. fix sentiment classification output
3. update quickstart and trainer doc

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9646399
 2022-08-04 22:38:31 +08:00
wenmeng.zwm
b3f4ac8acc [to #43115042] add trainer usage doc 
1. add trainer doc
2. support local configuration file for trainer
3. update nlp trainer test

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9541239
 2022-07-29 10:28:50 +08:00
wenmeng.zwm
590d531484 [to #43115513] requirements refine and preparation for v0.3 release 
*  remove tensorflow numpy from audio requirements
*  add audio requirements to all
* auto set model to eval model for pipeline
*  add audio requirement check hint for easyasr and kwsbp
*  fix docs build error

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9561021
 2022-07-28 22:04:18 +08:00
mulin.lyh
c4fb2445d6 [to #43259593]feat: add list model api 
添加list model查询api
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9446917
 2022-07-22 22:46:01 +08:00
Yingda Chen
59879b687a [to #42322933]update develop.md 2022-07-19 14:07:47 +08:00
wenmeng.zwm
231f400133 [to #43112534] finetune support and first case 
co-contributed with 夕陌&雨泓

 * add torch epoch based trainer and dis utils
 * add hooks including optimizer, lrscheduler, logging, checkpoint, evaluation, time profiling
 * add torch mdoel base and test
 * add optimizer and lrscheduler module
 * add sbert for text classification example
 * add task_dataset for dataset-level processor

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9338412
 2022-07-14 16:25:55 +08:00
yingda.chen
d532d45559 [to #42322933]unified revision control 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9319772
 2022-07-08 20:20:42 +08:00
yingda.chen
8a65b0b1ff [to #42322933]update docs 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9316720
 2022-07-08 14:52:29 +08:00
wenmeng.zwm
9f1ad5da80 fix several small problems for v0.2 
* rename name of whl to modelscope
* auto install all requirements when running citest
* auto download dynamic lib for aec pipeline
* fix setup.py  audio extras not set
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9297825
 2022-07-07 16:40:11 +08:00
wenmeng.zwm
274cf6ffa9 [to #42362425] fix audio_requirement and refine quickstart, changelog doc 
* make audio requirements optional
 * add changelog for version v0.2
 * add numpy constraint for compatibility with tensorflow1.15
 * update faq
 * fix nlp requiring tensorflow
 * add torchvision to multimodal dependency
 * bump version from 0.2.1 to 0.2.2
 * add warning msg when tensorflow is not installed
 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9268278
 2022-07-05 21:44:33 +08:00
zhangzhicheng.zzc
cf194ef6cd [to #42322933] nlp preprocessor refactor 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9269314

    * init

* token to ids

* add model

* model forward ready

* add intent

* intent preprocessor ready

* intent success

* merge master

* test with model hub

* add flake8

* update

* update

* update

* Merge branch 'master' into nlp/space/gen

* delete file about gen

* init

* fix flake8 bug

* [to #42322933] init

* bug fix

* [to #42322933] init

* update pipeline registry info

* Merge remote-tracking branch 'origin/master' into feat/nli

* [to #42322933] init

* [to #42322933] init

* modify forward

* [to #42322933] init

* generation ready

* init

* Merge branch 'master' into feat/zero_shot_classification

# Conflicts:
#	modelscope/preprocessors/__init__.py

* [to #42322933] bugfix

* [to #42322933] pre commit fix

* fill mask

* registry multi models on model and pipeline

* add tests

* test level >= 0

* local gen ready

* merge with master

* dialog modeling ready

* fix comments: rename and refactor AliceMindMLM; adjust pipeline

* space intent and modeling(generation) are ready

* bug fix

* add dep

* add dep

* support dst data processor

* merge with nlp/space/dst

* merge with master

* Merge remote-tracking branch 'origin' into feat/fill_mask

Conflicts:
	modelscope/models/nlp/__init__.py
	modelscope/pipelines/builder.py
	modelscope/pipelines/outputs.py
	modelscope/preprocessors/nlp.py
	requirements/nlp.txt

* merge with master

* merge with master 2/2

* fix comments

* fix isort for pre-commit check

* allow params pass to pipeline's __call__ method

* Merge remote-tracking branch 'origin/master' into feat/zero_shot_classification

* merge with nli task

* merge with sentiment_classification

* merge with zero_shot_classfication

* merge with fill_mask

* merge with space

* merge with master head

* Merge remote-tracking branch 'origin' into feat/fill_mask

Conflicts:
	modelscope/utils/constant.py

* fix: pipeline module_name from model_type to 'fill_mask' & fix merge bug

* unfiinished change

* fix bug

* unfinished

* unfinished

* revise modelhub dependency

* Merge branch 'feat/nlp_refactor' of http://gitlab.alibaba-inc.com/Ali-MaaS/MaaS-lib into feat/nlp_refactor

* add eval() to pipeline call

* add test level

* ut run passed

* add default args

* tmp

* merge master

* all ut passed

* remove an useless enum

* revert a mis modification

* revert a mis modification

* Merge commit 'ace8af92465f7d772f035aebe98967726655f12c' into feat/nlp

* commit 'ace8af92465f7d772f035aebe98967726655f12c':
  [to #42322933] Add cv-action-recongnition-pipeline to maas lib
  [to #42463204]  support Pil.Image for image_captioning_pipeline
  [to #42670107] restore pydataset test
  [to #42322933] add create if not exist and add(back) create model example         Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9130661
  [to #41474818]fix: fix errors in task name definition

# Conflicts:
#	modelscope/pipelines/builder.py
#	modelscope/utils/constant.py

* Merge branch 'feat/nlp' into feat/nlp_refactor

* feat/nlp:
  [to #42322933] Add cv-action-recongnition-pipeline to maas lib
  [to #42463204]  support Pil.Image for image_captioning_pipeline
  [to #42670107] restore pydataset test
  [to #42322933] add create if not exist and add(back) create model example         Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9130661
  [to #41474818]fix: fix errors in task name definition

# Conflicts:
#	modelscope/pipelines/builder.py

* fix compile bug

* refactor space

* Merge branch 'feat/nlp_refactor' of http://gitlab.alibaba-inc.com/Ali-MaaS/MaaS-lib into feat/nlp_refactor

* Merge remote-tracking branch 'origin' into feat/fill_mask

* fix

* pre-commit lint

* lint file

* lint file

* lint file

* update modelhub dependency

* lint file

* ignore dst_processor temporary

* solve comment: 1. change MaskedLMModelBase to MaskedLanguageModelBase 2. remove a useless import

* recommit

* remove MaskedLanguageModel from __all__

* Merge commit '1a0d4af55a2eee69d89633874890f50eda8f8700' into feat/nlp_refactor

* commit '1a0d4af55a2eee69d89633874890f50eda8f8700':
  [to #42322933] test level check         Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9143809
  [to #42322933] update nlp models name in metainfo         Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9134657

# Conflicts:
#	modelscope/metainfo.py

* update

* revert pipeline params update

* remove zeroshot

* update sequence classfication outpus

* merge with fill mask

* Merge remote-tracking branch 'origin' into feat/fill_mask

* fix

* fix flake8 warning of dst

* Merge remote-tracking branch 'origin/feat/fill_mask' into feat/nlp

* merge with master

* remove useless test.py

* Merge remote-tracking branch 'origin/master' into feat/nlp

* remove unformatted space trainer

* revise based on comment except chinease comment

* skip ci blocking

* translation pipeline

* csanmt model for translation pipeline

* update

* update

* update builder.py

* change Chinese notes of space3.0 into English

* translate chinese comment to english

* add space to metainfo

* update casnmt_translation

* update csanmt transformer

* merge with master

* update csanmt translation

* update lint

* update metainfo.py

* Update translation_pipeline.py

* Update builder.py

* fix: 1. make csanmt derived from Model 2. add kwargs to prevent from call error

* pre-commit check

* temp exclue flake8

* temp ignore translation files

* fix bug

* pre-commit passed

* fixbug

* fixbug

* revert pre commit ignorance

* pre-commit passed

* fix bug

* merge with master

* add missing setting

* merge with master

* add outputs

* modify test level

* modify chinese comment

* remove useless doc

* space outputs normalization

* Merge remote-tracking branch 'origin/master' into nlp/translation

* update translation_pipeline.py

* Merge remote-tracking branch 'origin/master' into feat/nlp

* Merge remote-tracking branch 'origin/master' into nlp/translation

* add new __init__ method

* add new __init__ method

* update output format

* Merge remote-tracking branch 'origin/master' into feat/nlp

* update output format

* merge with master

* merge with nlp/translate

* update the translation comment

* update the translation comment

* Merge branch 'nlp/translation' into feat/nlp

* Merge remote-tracking branch 'origin/master' into feat/nlp

* Merge remote-tracking branch 'origin/master' into feat/nlp

* nlp preprocessor refactor

* add get_model_type in util.hub

* update the default preprocessor args

* update the fill mask preprocessor

* bug typo fixed
 2022-07-05 20:40:48 +08:00
yingda.chen
a60d675d25 [to #42322933]formalize test data storage 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9251871
 2022-07-04 12:11:12 +08:00
yingda.chen
0260b89431 [to #42322933] unified output keys 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9240440
 2022-07-01 18:29:34 +08:00
huangjun.hj
31b203acfb * [to #42322933] refine quick_start.md and pipeline.md 2022-07-01 17:35:56 +08:00
wenmeng.zwm
8e51a073a6 [to #42966122] requirements enchanment and self-host repo support 
* add self-hosted repo:
* add extra requirements for different field and reduce necessary requirements
* update docker file with so required by audio
* add requirements checker which will be used later when implement lazy import
* remove repeated requirements and replace opencv-python-headless with opencv-python

example usage:
```shell
pip install model_scope[all] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[cv] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[nlp] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[audio] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[multi-modal] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html

```
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9211383
 2022-07-01 16:38:06 +08:00
yingda.chen
6702b29e21 [to #42794773]rename pydataset to msdataset 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9165402
 2022-06-27 11:09:38 +08:00
Yingda Chen
b6e3fd80b0 Revert "[to #42794773] rename pydataset to msdataset" 
This reverts commit c8e2e6de0e.
 2022-06-25 08:50:28 +08:00
Yingda Chen
c8e2e6de0e [to #42794773] rename pydataset to msdataset 2022-06-25 08:36:48 +08:00
jiaqi.sjq
b1490bfd7f [to #9061073] feat: merge tts to master 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9061073
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9061073

    * [to #41669377] docs and tools refinement and release 

1. add build_doc linter script
2. add sphinx-docs support
3. add development doc and api doc
4. change version to 0.1.0 for the first internal release version

Link: https://code.aone.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8775307

* [to #41669377] add pipeline tutorial and fix bugs 

1. add pipleine tutorial
2. fix bugs when using pipeline with certain model and preprocessor

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8814301

* refine doc

* refine doc

* merge remote release/0.1 and fix conflict

* Merge branch 'release/0.1' into 'nls/tts'

Release/0.1



See merge request !1700968

* [Add] add tts preprocessor without requirements. finish requirements build later

* [Add] add requirements and frd submodule

* [Fix] remove models submodule

* [Add] add am module

* [Update] update am and vocoder

* [Update] remove submodule

* [Update] add models

* [Fix] fix init error

* [Fix] fix bugs with tts pipeline

* merge master

* [Update] merge from master

* remove frd subdmoule and using wheel from oss

* change scripts

* [Fix] fix bugs in am and vocoder

* [Merge] merge from master

* Merge branch 'master' into nls/tts

* [Fix] fix bugs

* [Fix] fix pep8

* Merge branch 'master' into nls/tts

* [Update] remove hparams and import configuration from kwargs

* Merge branch 'master' into nls/tts

* upgrade tf113 to tf115

* Merge branch 'nls/tts' of gitlab.alibaba-inc.com:Ali-MaaS/MaaS-lib into nls/tts

* add multiple versions of ttsfrd

* merge master

* [Fix] fix cr comments

* Merge branch 'master' into nls/tts

* [Fix] fix cr comments 0617

* Merge branch 'master' into nls/tts

* [Fix] remove comment out codes

* [Merge] merge from master

* [Fix] fix crash for incompatible tf and pytorch version, and frd using zip file resource

* Merge branch 'master' into nls/tts

* [Add] add cuda support
 2022-06-20 17:23:11 +08:00
wenmeng.zwm
201922d33d [to #42461396] add git-lfs support and mv test data to git-lfs 2022-06-17 19:33:15 +08:00
wenmeng.zwm
c59833c7ee [to #42461396] feat: test_level support 
* add test level support
* update develop doc
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9021354
 2022-06-15 14:53:49 +08:00
yingda.chen
d7112be056 [to #42510875]use sphinx book theme 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9042850
 2022-06-15 14:07:57 +08:00
wenmeng.zwm
67086e26f9 [to #42362932] feat: docker_support 
* add dockerfile
* uninstall opencv-python-headless
* update develop doc
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9020302
 2022-06-14 16:29:12 +08:00
wenmeng.zwm
1f6b376599 [to #42373878] refactor maaslib to modelscope 
1.  refactor maaslib to modelscope
2.  fix UT error
3.  support pipeline which does not register default model

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8988388
 2022-06-09 20:16:26 +08:00
wenmeng.zwm
dd00195814 [to #42362853] add default model support and fix circular import 
1. add default model support
2. fix circular import
3. temporarily skip ofa and palm test which costs too much time

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8981076
 2022-06-09 16:57:33 +08:00
yingda.chen
0d840d519c [to #42339763] move pydataset into maas_lib 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8974892
 2022-06-09 10:14:48 +08:00
feiwu.yfw
235880f300 [to #42339763] merge pydataset into maas-lib 
* merge pydataset to the repo
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8955999
 2022-06-08 18:29:39 +08:00
wenmeng.zwm
d395e9abc2 [to #42281043] support kwargs in pipeline 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8949062
* support kwargs in pipeline

* update develop doc with CR instruction
 2022-06-07 14:49:57 +08:00
huangjun.hj
cdacbfb476 refine pipeline doc 2022-06-01 11:55:14 +08:00
yingda.chen
f8eb699f7f refine tests and examples 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8898823
 2022-06-01 10:20:53 +08:00
wenmeng.zwm
c4bfd6cced [to #41999503] refine doc and requirements for linux and mac 
1. refine quick start and pipeline doc
2. remove tf pytorch easynlp from requirements
3. lazy import for torch and tensorflow
4. test successfully on linux and mac intel cpu
5. update api doc

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8882373
 2022-05-31 11:49:46 +08:00