Commit Graph

262 Commits

Author SHA1 Message Date
wenmeng.zwm
d55525bfb6 [to #43112771] requirements check and lazy import support 2022-07-27 17:29:16 +08:00
shichen.fsc
5ac448f5c7 [to #42322933] simplify asr inference code, and remove disk-write behavior
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9410174
2022-07-27 14:49:24 +08:00
mulin.lyh
a7ffa7f6ce [to #43588165]fix: nlp case github connection broken
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9528871

    * [to #43588165]fix: nlp case github connection broken
2022-07-27 11:00:40 +08:00
mulin.lyh
fc90bf0d1a [to #43554786]fix: test error is not detected in gate test, protobuf version to (3, 3.21.0) for tensorflow
限制protobuf版本,修复单元测试有error返回值为0问题
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9510263

    * fix test error is not detected in gate test, protobuf version to (3, 3.21.0)
2022-07-26 10:57:16 +08:00
shichen.fsc
b3b950e616 [to #42322933] simplify kws code, and remove disk-write behavior
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9491681
2022-07-25 22:37:15 +08:00
Yingda Chen
badc91aafa Revert "[to #43259593]chore: update cv requirements (更新cv侧统计的依赖项) "
This reverts commit bab28ae3fa.
2022-07-25 13:50:18 +08:00
liangting.zl
bab28ae3fa [to #43259593]chore: update cv requirements (更新cv侧统计的依赖项)
[依赖收集及列表](https://yuque.antfin.com/docs/share/59d1f42c-e061-4e73-903f-0af053a03627#XXbRB)
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9456258
2022-07-25 13:31:36 +08:00
hemu.zp
ab04ceafc5 [to #42322933] set rouge_score version
2022/7/22 更新的 rouge_socre 0.0.7 版本存在 import 报错的问题,暂时将 rouge_score 库版本设置为 <= 0.0.4
2022-07-22 18:22:43 +08:00
yingda.chen
dbb450932d [to #42322933] clean requirment
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9458816
2022-07-20 20:43:42 +08:00
hemu.zp
0b7b964226 [to #42322933] Add palm finetuning
Palm 模型支持 finetuning
2022-07-20 16:36:11 +08:00
mulin.lyh
69047b99ae [to #43387011]feat: ci test to new host and running in docker
ci 测试迁移新的机器,并且在容器中运行,减小互相干扰的可能
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9427096

    * add docker ci script
2022-07-20 11:49:05 +08:00
ly119399
40b1a9566d [to #42322933] remove dependency en_core_web_sm from nlp.txt
将nlp.txt中的en_core_web_sm依赖删除,在代码中下载响应的依赖
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9432253
2022-07-19 22:39:54 +08:00
wenmeng.zwm
e62cd756df [to #42322933] relax requirements
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9407594
2022-07-18 17:52:14 +08:00
jiaqi.sjq
a17f29ce54 [to #42322933] Update tts task inputs
Refactor tts task inputs
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9412937
2022-07-18 17:50:59 +08:00
shichen.fsc
d7c780069f [to #42322933] add asr inference with pytorch(espnet framework)
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9273537
2022-07-11 16:48:47 +08:00
jiaqi.sjq
d313c440c4 [to #9303837] Merge frontend am and vocoder into one model card
Merge frontend, am and vocoder model card into one model card.
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9303837
2022-07-08 14:26:18 +08:00
wenmeng.zwm
9f1ad5da80 fix several small problems for v0.2
* rename name of whl to modelscope
* auto install all requirements when running citest
* auto download dynamic lib for aec pipeline
* fix setup.py  audio extras not set
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9297825
2022-07-07 16:40:11 +08:00
wenmeng.zwm
274cf6ffa9 [to #42362425] fix audio_requirement and refine quickstart, changelog doc
* make audio requirements optional
 * add changelog for version v0.2
 * add numpy constraint for compatibility with tensorflow1.15
 * update faq
 * fix nlp requiring tensorflow
 * add torchvision to multimodal dependency
 * bump version from 0.2.1 to 0.2.2
 * add warning msg when tensorflow is not installed
 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9268278
2022-07-05 21:44:33 +08:00
智丞
e5f99bc6d0 update requirements/nlp 2022-07-03 10:39:17 +08:00
智丞
54122b2dab merge with master 2022-07-02 21:58:42 +08:00
zhangzhicheng.zzc
5b98cc1513 [to #42322933]新增:nli,sentiment_classification,dialog_intent,dialog_modeling
添加了,nli,sentiment_classification, dialog_intent, dialog_modeling几个pipeline。同时加入了nlp里面sequence classification一些简单的抽象。
去掉了zero_shot_classification
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9159089
2022-07-01 23:13:03 +08:00
wenmeng.zwm
8e51a073a6 [to #42966122] requirements enchanment and self-host repo support
* add self-hosted repo:
* add extra requirements for different field and reduce necessary requirements
* update docker file with so required by audio
* add requirements checker which will be used later when implement lazy import
* remove repeated requirements and replace opencv-python-headless with opencv-python

example usage:
```shell
pip install model_scope[all] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[cv] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[nlp] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[audio] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[multi-modal] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html

```
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9211383
2022-07-01 16:38:06 +08:00
ly119399
90c5e18183 update requirement 2022-06-30 19:28:33 +08:00
hemu.zp
a03106172f [to #42322933] Update sofa version 1.0.4.2
更新 sofa 到 1.0.4.2 版本,解决 palm 模型无法并行的 bug
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9217921
2022-06-30 15:35:59 +08:00
ly119399
4cfc4b43a0 merge feat/nlp 2022-06-29 22:27:52 +08:00
hemu.zp
fabea5604e [to #42322933] Add MPLUG model
添加 MPLUG 模型的 visual question answering 任务 pipeline

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9182119
2022-06-29 11:08:34 +08:00
bin.xue
04b7eba285 [to #42322933] Merge ANS pipeline into master
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9178339

    * refactor: move aec models to audio/aec

* refactor: move aec models to audio/aec

* refactor: move aec models to audio/aec

* refactor: move aec models to audio/aec

* feat: add unittest for ANS pipeline

* Merge branch 'master' into dev/ans

* add new SoundFile to audio dependency

* Merge branch 'master' into dev/ans

* use ANS pipeline name from metainfo

* Merge branch 'master' into dev/ans

* chore: update docstring of ANS module

* Merge branch 'master' into dev/ans

* refactor: use names from metainfo

* refactor: enable ans unittest

* refactor: add more log message in unittest
2022-06-28 14:41:08 +08:00
ly119399
80461600f7 merge nlp 2022-06-25 13:44:31 +08:00
智丞
6741bc5afd merge with master 2022-06-24 15:12:39 +08:00
jiaqi.sjq
4a3f22259f * Relax version requirements in audio.txt
* Fix bugs in ttsfrd which may cause text-to-speech break disappear and upgrade it to version 0.0.2
* other fix to make ut pass
2022-06-24 12:05:01 +08:00
suluyan.sly
0286dd45cc [to #42322933] Add nlp-structbert/veco-fill-mask-pipeline to maas lib
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9069107
2022-06-24 10:12:16 +08:00
雨泓
c74a252b6c Merge commit 'ace8af92465f7d772f035aebe98967726655f12c' into feat/nlp
* commit 'ace8af92465f7d772f035aebe98967726655f12c':
  [to #42322933] Add cv-action-recongnition-pipeline to maas lib
  [to #42463204]  support Pil.Image for image_captioning_pipeline
  [to #42670107] restore pydataset test
  [to #42322933] add create if not exist and add(back) create model example         Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9130661
  [to #41474818]fix: fix errors in task name definition

# Conflicts:
#	modelscope/pipelines/builder.py
#	modelscope/utils/constant.py
2022-06-23 11:26:44 +08:00
yongfei.zyf
ace8af9246 [to #42322933] Add cv-action-recongnition-pipeline to maas lib
达摩行为识别合入maas lib
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9134444
2022-06-23 10:22:43 +08:00
智丞
5fdfb9e5b7 merge with master head 2022-06-22 14:23:07 +08:00
wenmeng.zwm
e288cf076e [to #42362853] refactor pipeline and standardize module_name
* using get_model to validate hub path 
* support reading pipeline info from configuration file
* add metainfo const
* update model type and pipeline type and fix UT
* relax requimrent for protobuf
* skip two dataset tests due to temporal failure
 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9118154
2022-06-22 14:15:32 +08:00
智丞
4dfab005ca merge with space 2022-06-22 11:32:56 +08:00
智丞
17e6a3d41d merge with fill_mask 2022-06-22 11:30:03 +08:00
mulin.lyh
76c6ff6329 [to #42675838]merge model hub code
合并model hub 代码
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9096493
2022-06-21 20:04:25 +08:00
suluyan
cf75f2aeec Merge remote-tracking branch 'origin' into feat/fill_mask
Conflicts:
	modelscope/models/nlp/__init__.py
	modelscope/pipelines/builder.py
	modelscope/pipelines/outputs.py
	modelscope/preprocessors/nlp.py
	requirements/nlp.txt
2022-06-21 15:33:24 +08:00
xixing.tj
968e660235 [to #42322933] create ocr_detection task
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8967432

* create ocr_detection task

* replace c++ nms with python version

* replace c++ decoder with python version

* add requirements for ocr_detection
2022-06-21 11:56:52 +08:00
智丞
05ac2b15d1 merge with master 2022-06-20 21:39:52 +08:00
ly119399
c6cf0d20c5 add dep 2022-06-20 17:35:53 +08:00
jiaqi.sjq
b1490bfd7f [to #9061073] feat: merge tts to master
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9061073
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9061073

    * [to #41669377] docs and tools refinement and release 

1. add build_doc linter script
2. add sphinx-docs support
3. add development doc and api doc
4. change version to 0.1.0 for the first internal release version

Link: https://code.aone.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8775307

* [to #41669377] add pipeline tutorial and fix bugs 

1. add pipleine tutorial
2. fix bugs when using pipeline with certain model and preprocessor

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8814301

* refine doc

* refine doc

* merge remote release/0.1 and fix conflict

* Merge branch 'release/0.1' into 'nls/tts'

Release/0.1



See merge request !1700968

* [Add] add tts preprocessor without requirements. finish requirements build later

* [Add] add requirements and frd submodule

* [Fix] remove models submodule

* [Add] add am module

* [Update] update am and vocoder

* [Update] remove submodule

* [Update] add models

* [Fix] fix init error

* [Fix] fix bugs with tts pipeline

* merge master

* [Update] merge from master

* remove frd subdmoule and using wheel from oss

* change scripts

* [Fix] fix bugs in am and vocoder

* [Merge] merge from master

* Merge branch 'master' into nls/tts

* [Fix] fix bugs

* [Fix] fix pep8

* Merge branch 'master' into nls/tts

* [Update] remove hparams and import configuration from kwargs

* Merge branch 'master' into nls/tts

* upgrade tf113 to tf115

* Merge branch 'nls/tts' of gitlab.alibaba-inc.com:Ali-MaaS/MaaS-lib into nls/tts

* add multiple versions of ttsfrd

* merge master

* [Fix] fix cr comments

* Merge branch 'master' into nls/tts

* [Fix] fix cr comments 0617

* Merge branch 'master' into nls/tts

* [Fix] remove comment out codes

* [Merge] merge from master

* [Fix] fix crash for incompatible tf and pytorch version, and frd using zip file resource

* Merge branch 'master' into nls/tts

* [Add] add cuda support
2022-06-20 17:23:11 +08:00
ly119399
b812cb78c9 add dep 2022-06-20 17:10:54 +08:00
ly119399
d0b33eade8 space intent and modeling(generation) are ready 2022-06-20 16:39:21 +08:00
hemu.zp
99fb503695 [to #42322933] Add Palm2.0 model.
接入支持中英文的 Palm2.0 模型,复用 text-generation-pipeline

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9066550
2022-06-20 16:00:31 +08:00
bin.xue
31498c1d6a [to #41669377] add speech AEC pipeline
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8973072

    * [to #41669377] docs and tools refinement and release 

1. add build_doc linter script
2. add sphinx-docs support
3. add development doc and api doc
4. change version to 0.1.0 for the first internal release version

Link: https://code.aone.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8775307

* [to #41669377] add pipeline tutorial and fix bugs 

1. add pipleine tutorial
2. fix bugs when using pipeline with certain model and preprocessor

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8814301

* refine doc

* feat: add audio aec pipeline and preprocessor

* feat: add audio aec model classes

* feat: add audio aec loss functions

* refactor:delete no longer used loss function

* [to #42281043] support kwargs in pipeline 
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8949062
* support kwargs in pipeline

* update develop doc with CR instruction

* Merge branch 'release/0.1' into dev/aec

* style: reformat code by pre-commit tools

* feat:support maas_lib pipeline auto downloading model

* test:add aec test case as sample code

* feat:aec pipeline use config from maashub

* feat:aec pipeline use feature parameters from maashub

* update setup.cfg to disable PEP8 rule W503 in flake8 and yapf

* format:fix double quoted strings, indent issues and optimize import

* refactor:extract some constant in aec pipeline

* refactor: delete no longer used  __main__ statement

* chore:change all Chinese comments to English

* fix: change file name style to lower case

* refactor: rename model name

* feat:load C++ .so from LD_LIBRARY_PATH

* feat:register PROPROCESSOR for LinearAECAndFbank

* refactory:move aec process from postprocess() to forward() and update comments

* refactory:add more readable error message when audio sample rate is not 16000

* fix: package maas_lib renamed to modelscope in import statement

* feat: optimize the error message of audio layer classes

* format: delete empty lines

* refactor: rename audio preprocessor and optimize error message

* refactor: change aec model id to damo/speech_dfsmn_aec_psm_16k

* refactor: change sample audio file url to public oss

* Merge branch 'master' into dev/aec

* feat: add output info for aec pipeline

* fix: normalize output audio data to [-1.0, 1.0]

* refactor:use constant from ModelFile

* feat: AEC pipeline can use c++ lib in current working directory and the test will download it

* fix: c++ downloading should work wherever test is triggerd
2022-06-17 19:56:11 +08:00
suluyan
3aa1a70ac8 add tests 2022-06-17 00:36:25 +08:00
yingda.chen
d7112be056 [to #42510875]use sphinx book theme
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9042850
2022-06-15 14:07:57 +08:00
wenmeng.zwm
b4fc38e1b9 [to #42461396] add Pillow version constraint and update hub version
1.  use Pillow >= 6.2.0
2. change skip test msg for image caption
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9041194
2022-06-15 11:43:52 +08:00