Commit Graph

66 Commits

Author SHA1 Message Date
yingda.chen
0acbfe1663 [to #42322933] interface refine with doc
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9159678
2022-06-24 23:54:10 +08:00
mulin.lyh
6991620f59 [to #42698276]fix: git repo operations supports, gitlab token certification support. 2022-06-24 16:43:32 +08:00
jiaqi.sjq
4a3f22259f * Relax version requirements in audio.txt
* Fix bugs in ttsfrd which may cause text-to-speech break disappear and upgrade it to version 0.0.2
* other fix to make ut pass
2022-06-24 12:05:01 +08:00
yingda.chen
e7571a566f [to #42322933] skip dataset test for now
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9155405
2022-06-24 11:47:28 +08:00
suluyan.sly
0286dd45cc [to #42322933] Add nlp-structbert/veco-fill-mask-pipeline to maas lib
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9069107
2022-06-24 10:12:16 +08:00
yingda.chen
1a0d4af55a [to #42322933] test level check
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9143809
2022-06-23 16:55:48 +08:00
zhangzhicheng.zzc
371530ed1d [to #42322933] update nlp models name in metainfo
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9134657
2022-06-23 11:58:22 +08:00
yongfei.zyf
ace8af9246 [to #42322933] Add cv-action-recongnition-pipeline to maas lib
达摩行为识别合入maas lib
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9134444
2022-06-23 10:22:43 +08:00
lingcai.wl
2019315c54 [to #42463204] support Pil.Image for image_captioning_pipeline 2022-06-22 23:54:38 +08:00
feiwu.yfw
b8ba35bd27 [to #42670107] restore pydataset test
* pydataset unitest
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9132896
2022-06-22 22:01:25 +08:00
yingda.chen
8400ca1fc7 [to #42322933] add create if not exist and add(back) create model example
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9130661
2022-06-22 18:39:08 +08:00
luoyiyun.lyy
849410e107 [to #41474818]fix: fix errors in task name definition
修改constant.py中task name定义的错误
- sentence-similarity重复定义
- fill-mask多加了一个空格
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9125956

    * [to #41474818]fix: fix errors in task name definition
2022-06-22 15:49:58 +08:00
wenmeng.zwm
e288cf076e [to #42362853] refactor pipeline and standardize module_name
* using get_model to validate hub path 
* support reading pipeline info from configuration file
* add metainfo const
* update model type and pipeline type and fix UT
* relax requimrent for protobuf
* skip two dataset tests due to temporal failure
 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9118154
2022-06-22 14:15:32 +08:00
mulin.lyh
76c6ff6329 [to #42675838]merge model hub code
合并model hub 代码
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9096493
2022-06-21 20:04:25 +08:00
zhangzhicheng.zzc
7a175ee9b3 [to #42322933]move tts dependency to local
requirements/audio.txt
中依赖存在两个问题:
1. torch/tensorflow版本写死
2. 两个whl包是linux环境下才可安装,进而whl包安装的package要在mac上忽略的话,代码中必须把相应package放到local中引用

上面两个问题在linux环境中的测试用例是可以通过的。
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9103904

    * move tts dependency to local
2022-06-21 14:06:09 +08:00
xixing.tj
968e660235 [to #42322933] create ocr_detection task
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8967432

* create ocr_detection task

* replace c++ nms with python version

* replace c++ decoder with python version

* add requirements for ocr_detection
2022-06-21 11:56:52 +08:00
yichang.zyc
a2cf3d619e [to #42322933]update ofa caption model
将caption pipeline的实现从pipeline下沉到model,并拆解preprocessor
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9081211

    * [to #41669377] docs and tools refinement and release 

1. add build_doc linter script
2. add sphinx-docs support
3. add development doc and api doc
4. change version to 0.1.0 for the first internal release version

Link: https://code.aone.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8775307

* [to #41669377] add pipeline tutorial and fix bugs 

1. add pipleine tutorial
2. fix bugs when using pipeline with certain model and preprocessor

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8814301

* refine doc

* refine doc

* upload ofa for caption(with source code but not whl)

* remove data in gitignore

* append uncommitted data dir in ofa

* remove ofa_dir , use ofa.whl instead.

* update BPE

* rollback changes used in debugging.

* Merge branch 'master' into ofa/image_caption

# Conflicts:
#	docs/README.md
#	docs/source/conf.py
#	docs/source/index.rst
#	docs/source/tutorials/pipeline.md
#	maas_lib/models/nlp/sequence_classification_model.py
#	maas_lib/pipelines/builder.py
#	maas_lib/version.py
#	setup.py
#	tests/pipelines/test_text_classification.py

* 1. fix a bug in pipelines/builder.py.
2. modify model_path to model in image_captioning.py.

* 1. rename test_image_captioning.py.

* format all files using pre-commit.

* add fairseq in requirements.txt

* add fairseq in requirements.txt

* change fairseq path to git repo to a whl on oss in ofa.txt.

* change module_name to 'ofa'

* Merge remote-tracking branch 'origin/master' into ofa/image_caption

# Conflicts:
#	maas_lib/pipelines/builder.py

* optim requirements for ofa / refine image_captioning.py

* uncommited change.

* feat: Fix confilct, auto commit by WebIDE

* Merge remote-tracking branch 'origin/master' into ofa/image_caption

# Conflicts:
#	maas_lib/pipelines/multi_modal/__init__.py
#	modelscope/pipelines/multi_modal/image_captioning.py
#	tests/pipelines/test_image_captioning.py

* merge master

* merge master

* merge master

* rename

* Merge remote-tracking branch 'origin/master' into ofa/nlu

* add caption model

* Merge remote-tracking branch 'origin/master' into ofa/nlu

* update ofa caption model

* fix some typo, update unittest

* use local test image

* use local test image

* refactor, ofa -> multi_model

* merge master

* 删除 image_caption_pipeline.py
2022-06-21 11:48:49 +08:00
feiwu.yfw
c7238a470b [to #42670107]pydataset fetch data from datahub
* pydataset fetch data from datahub
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9060856
2022-06-21 11:10:28 +08:00
bin.xue
97a0087976 [to #42322933]语音使用local import避免其他任务加载torchaudio
feat: local import torchaudio
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9097950

    * feat: local import torchaudio
2022-06-20 20:30:27 +08:00
jiaqi.sjq
b1490bfd7f [to #9061073] feat: merge tts to master
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9061073
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9061073

    * [to #41669377] docs and tools refinement and release 

1. add build_doc linter script
2. add sphinx-docs support
3. add development doc and api doc
4. change version to 0.1.0 for the first internal release version

Link: https://code.aone.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8775307

* [to #41669377] add pipeline tutorial and fix bugs 

1. add pipleine tutorial
2. fix bugs when using pipeline with certain model and preprocessor

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8814301

* refine doc

* refine doc

* merge remote release/0.1 and fix conflict

* Merge branch 'release/0.1' into 'nls/tts'

Release/0.1



See merge request !1700968

* [Add] add tts preprocessor without requirements. finish requirements build later

* [Add] add requirements and frd submodule

* [Fix] remove models submodule

* [Add] add am module

* [Update] update am and vocoder

* [Update] remove submodule

* [Update] add models

* [Fix] fix init error

* [Fix] fix bugs with tts pipeline

* merge master

* [Update] merge from master

* remove frd subdmoule and using wheel from oss

* change scripts

* [Fix] fix bugs in am and vocoder

* [Merge] merge from master

* Merge branch 'master' into nls/tts

* [Fix] fix bugs

* [Fix] fix pep8

* Merge branch 'master' into nls/tts

* [Update] remove hparams and import configuration from kwargs

* Merge branch 'master' into nls/tts

* upgrade tf113 to tf115

* Merge branch 'nls/tts' of gitlab.alibaba-inc.com:Ali-MaaS/MaaS-lib into nls/tts

* add multiple versions of ttsfrd

* merge master

* [Fix] fix cr comments

* Merge branch 'master' into nls/tts

* [Fix] fix cr comments 0617

* Merge branch 'master' into nls/tts

* [Fix] remove comment out codes

* [Merge] merge from master

* [Fix] fix crash for incompatible tf and pytorch version, and frd using zip file resource

* Merge branch 'master' into nls/tts

* [Add] add cuda support
2022-06-20 17:23:11 +08:00
hemu.zp
99fb503695 [to #42322933] Add Palm2.0 model.
接入支持中英文的 Palm2.0 模型,复用 text-generation-pipeline

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9066550
2022-06-20 16:00:31 +08:00
yingda.chen
c4b6a23bc9 [to #42322933] unify naming for model and pipeline files
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9083378
2022-06-20 10:54:00 +08:00
bin.xue
31498c1d6a [to #41669377] add speech AEC pipeline
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8973072

    * [to #41669377] docs and tools refinement and release 

1. add build_doc linter script
2. add sphinx-docs support
3. add development doc and api doc
4. change version to 0.1.0 for the first internal release version

Link: https://code.aone.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8775307

* [to #41669377] add pipeline tutorial and fix bugs 

1. add pipleine tutorial
2. fix bugs when using pipeline with certain model and preprocessor

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8814301

* refine doc

* feat: add audio aec pipeline and preprocessor

* feat: add audio aec model classes

* feat: add audio aec loss functions

* refactor:delete no longer used loss function

* [to #42281043] support kwargs in pipeline 
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8949062
* support kwargs in pipeline

* update develop doc with CR instruction

* Merge branch 'release/0.1' into dev/aec

* style: reformat code by pre-commit tools

* feat:support maas_lib pipeline auto downloading model

* test:add aec test case as sample code

* feat:aec pipeline use config from maashub

* feat:aec pipeline use feature parameters from maashub

* update setup.cfg to disable PEP8 rule W503 in flake8 and yapf

* format:fix double quoted strings, indent issues and optimize import

* refactor:extract some constant in aec pipeline

* refactor: delete no longer used  __main__ statement

* chore:change all Chinese comments to English

* fix: change file name style to lower case

* refactor: rename model name

* feat:load C++ .so from LD_LIBRARY_PATH

* feat:register PROPROCESSOR for LinearAECAndFbank

* refactory:move aec process from postprocess() to forward() and update comments

* refactory:add more readable error message when audio sample rate is not 16000

* fix: package maas_lib renamed to modelscope in import statement

* feat: optimize the error message of audio layer classes

* format: delete empty lines

* refactor: rename audio preprocessor and optimize error message

* refactor: change aec model id to damo/speech_dfsmn_aec_psm_16k

* refactor: change sample audio file url to public oss

* Merge branch 'master' into dev/aec

* feat: add output info for aec pipeline

* fix: normalize output audio data to [-1.0, 1.0]

* refactor:use constant from ModelFile

* feat: AEC pipeline can use c++ lib in current working directory and the test will download it

* fix: c++ downloading should work wherever test is triggerd
2022-06-17 19:56:11 +08:00
wenmeng.zwm
201922d33d [to #42461396] add git-lfs support and mv test data to git-lfs 2022-06-17 19:33:15 +08:00
zhangzhicheng.zzc
eb3209a79a [to #42322933]中文分词
chinese word segmentation
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9051491

    * add word segmentation

* Merge branch 'master' of http://gitlab.alibaba-inc.com/Ali-MaaS/MaaS-lib

* test with model hub

* merge with master

* update some description and test levels

* adding purge logic in test

* merge with master

* update variables definition

* generic word segmentation model as token classification model

* add output check
2022-06-17 14:00:31 +08:00
yingda.chen
ad8e080e37 [to #42322933] refactor model name
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9068994
2022-06-17 10:25:54 +08:00
wenmeng.zwm
4f7928bb6e [to #42362853] formalize the output of pipeline and make pipeline reusable
* format pipeline output and check it
* fix UT
* add docstr to clarify the difference between model.postprocess and pipeline.postprocess

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9051405
2022-06-16 11:15:09 +08:00
zhangzhicheng.zzc
ba471d4492 [to #42322933]sentence-similarity
Adding the new task of sentence_similarity, in which the model is the sofa version of structbert
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9016402

    * sbert-sentence-similarity

* [to #42322933] pip8

* merge with master for file dirs  update

* add test cases

* pre-commit lint check

* remove useless file

* download models again~

* skip time consuming test case

* update for pr reviews

* merge with master

* add test level

* reset test level to env level

* [to #42322933] init

* [to #42322933] init

* adding purge logic in test

* merge with head

* change test level

* using sequence classification processor for similarity
2022-06-15 23:35:12 +08:00
lingcai.wl
d983bdfc8e [to #42463204] support Pil.Image for image_captioning
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9049211
2022-06-15 18:37:40 +08:00
wenmeng.zwm
c59833c7ee [to #42461396] feat: test_level support
* add test level support
* update develop doc
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9021354
2022-06-15 14:53:49 +08:00
yingda.chen
d7112be056 [to #42510875]use sphinx book theme
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9042850
2022-06-15 14:07:57 +08:00
yingda.chen
5786b9a0a1 [to #42322933]formalize image matting
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9019685
2022-06-15 14:06:53 +08:00
wenmeng.zwm
b4fc38e1b9 [to #42461396] add Pillow version constraint and update hub version
1.  use Pillow >= 6.2.0
2. change skip test msg for image caption
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9041194
2022-06-15 11:43:52 +08:00
wenmeng.zwm
67086e26f9 [to #42362932] feat: docker_support
* add dockerfile
* uninstall opencv-python-headless
* update develop doc
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9020302
2022-06-14 16:29:12 +08:00
wenmeng.zwm
8a030ead72 [to #42362853] feat: rename config to configuration and remove repeated task fileds
1. rename maas_config to configuration
2. remove task field image and video, using cv instead

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9010802
2022-06-13 19:44:34 +08:00
wenmeng.zwm
3a78e32eb2 Revert "[to #42322933]formalize image matting"
This reverts commit de3ea0db54.
2022-06-13 18:30:29 +08:00
Yingda Chen
de3ea0db54 [to #42322933]formalize image matting 2022-06-13 15:19:30 +08:00
yingda.chen
b31c86aa0e [to #42409340] add hub specifier
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9005038
2022-06-13 14:15:54 +08:00
wenmeng.zwm
76bc51eadd [to #42362853] support multimodel in pipeline
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8997599
2022-06-10 16:22:28 +08:00
yingda.chen
3c1ec035fd [to #42322933] refine cartoon model and add model op utitlity
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8993758
2022-06-10 12:56:44 +08:00
wenmeng.zwm
1f6b376599 [to #42373878] refactor maaslib to modelscope
1.  refactor maaslib to modelscope
2.  fix UT error
3.  support pipeline which does not register default model

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8988388
2022-06-09 20:16:26 +08:00
wenmeng.zwm
dd00195814 [to #42362853] add default model support and fix circular import
1. add default model support
2. fix circular import
3. temporarily skip ofa and palm test which costs too much time

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8981076
2022-06-09 16:57:33 +08:00
myf272609
0f5b214ce0 [to #42322933] Add cv-person-image-cartoon-pipeline to maas lib
mass lib 接入人像卡通化算法
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8925552

    * update

* fix style issues

* fix style issues

* fix style issues

* fix style issues

* add requirements

* fix bug

* adapt class init

* fix

* fix tf2 issue

* feat: Fix confilct, auto commit by WebIDE

* fix commit issues

* fix
2022-06-09 15:18:38 +08:00
yingda.chen
0d840d519c [to #42339763] move pydataset into maas_lib
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8974892
2022-06-09 10:14:48 +08:00
yingda.chen
e3b8ec3bf1 [to #42339559] support multiple models
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8972440

    * [to #42339559] support multiple models
2022-06-08 21:27:14 +08:00
menrui.mr
68e64a90d4 [to #42322933] add image_caption_pipeline with OFA
1. add OFA whl for image caption pipeline
2. fix a bug in pipelines/builder.py
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8930942

    * [to #41669377] docs and tools refinement and release 

1. add build_doc linter script
2. add sphinx-docs support
3. add development doc and api doc
4. change version to 0.1.0 for the first internal release version

Link: https://code.aone.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8775307

* [to #41669377] add pipeline tutorial and fix bugs 

1. add pipleine tutorial
2. fix bugs when using pipeline with certain model and preprocessor

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8814301

* refine doc

* refine doc

* upload ofa for caption(with source code but not whl)

* remove data in gitignore

* append uncommitted data dir in ofa

* remove ofa_dir , use ofa.whl instead.

* update BPE

* rollback changes used in debugging.

* Merge branch 'master' into ofa/image_caption

# Conflicts:
#	docs/README.md
#	docs/source/conf.py
#	docs/source/index.rst
#	docs/source/tutorials/pipeline.md
#	maas_lib/models/nlp/sequence_classification_model.py
#	maas_lib/pipelines/builder.py
#	maas_lib/version.py
#	setup.py
#	tests/pipelines/test_text_classification.py

* 1. fix a bug in pipelines/builder.py.
2. modify model_path to model in image_captioning.py.

* 1. rename test_image_captioning.py.

* format all files using pre-commit.

* add fairseq in requirements.txt

* add fairseq in requirements.txt

* change fairseq path to git repo to a whl on oss in ofa.txt.

* change module_name to 'ofa'

* Merge remote-tracking branch 'origin/master' into ofa/image_caption

# Conflicts:
#	maas_lib/pipelines/builder.py

* optim requirements for ofa / refine image_captioning.py

* uncommited change.

* feat: Fix confilct, auto commit by WebIDE
2022-06-08 18:38:19 +08:00
feiwu.yfw
235880f300 [to #42339763] merge pydataset into maas-lib
* merge pydataset to the repo
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8955999
2022-06-08 18:29:39 +08:00
hemu.zp
8a76f40754 [to #42322933]Add text-generation-pipeline with Palm model.
将 Palm 中文模型接入 MaaS,添加了文本生成 pipeline
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8934393

    * add text_generation model and pipeline

* fix bug

* fix bug

* add TextGenerator in pipeline

* fix bug

* update checkpoint and test inputs

* remove magic number..

* fix bug

* adjust code with AutoModel

* clear comments and tidy up the code

* move model.eval() into generator

* update master interface and lint code

* replace 'palm-text-generation' with 'palm'

* add text_generation model and pipeline

* fix bug

* fix bug

* add TextGenerator in pipeline

* fix bug

* fix conflict of pipeline.txt

* remove magic number..

* fix bug

* adjust code with AutoModel

* clear comments and tidy up the code

* move model.eval() into generator

* fix conflict

* replace 'palm-text-generation' with 'palm'

* fix conflict

* add test_run_modelhub

* update sofa version

* modify sofa version

* add test_run_with_model_name

* fix bug
2022-06-08 17:11:04 +08:00
yingda.chen
d6868ddffe [to #42323743] retain local cached model files by default
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8963687
2022-06-08 14:22:23 +08:00
yingda.chen
e075ad2245 [to #42322515]support plain pipeline for bert
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8945177

    * support plain pipeline for bert
2022-06-08 11:29:25 +08:00