dataset_refactor/requirements/framework.txt

addict
attrs
datasets>=2.18.0
einops
oss2
Pillow
python-dateutil>=2.1
scipy
# latest version has some compatible issue.
setuptools==69.5.1
simplejson>=3.3.0
sortedcontainers>=1.5.9
transformers
urllib3>=1.26
[to #41402703] add basic modules * add constant * add logger module * add registry and builder module * add fileio module * add requirements and setup.cfg * add config module and tests * add citest script Link: https://code.aone.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8718998 2022-05-17 10:15:00 +08:00			`addict`
[to #45631658]feat support eas deploy 服务端文档链接(可能需要登录)： https://test.modelscope.cn/api/v1/deployer/docs Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10478609 2022-10-24 15:12:48 +08:00			`attrs`
Upgrade datasets (#921) * del _datasets_server import in hf_dataset_util * fix streaming for youku-mplug and adopt latest datasets * fix download config copy * update ut * add youku in test_general_datasets * update UT for general dataset * adapt to datasets version: 2.19.0 or later * add assert for youku data UT * fix disable_tqdm in some functions for 2.19.0 or later * update get_module_with_script * set trust_remote_code is True in load_dataset_with_ctx * update print info * update requirements for datasets version restriction * fix _dataset_info * add pillow * update comments * update comment * reuse _download function in DataDownloadManager * remove unused code * update test_run_modelhub in Human3DAnimationTest * set datasets>=2.18.0 2024-07-23 22:26:12 +08:00			`datasets>=2.18.0`
[to #42966122] requirements enchanment and self-host repo support * add self-hosted repo: * add extra requirements for different field and reduce necessary requirements * update docker file with so required by audio * add requirements checker which will be used later when implement lazy import * remove repeated requirements and replace opencv-python-headless with opencv-python example usage: ```shell pip install model_scope[all] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html pip install model_scope[cv] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html pip install model_scope[nlp] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html pip install model_scope[audio] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html pip install model_scope[multi-modal] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html ``` Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9211383 2022-07-01 16:38:06 +08:00			`einops`
[to #43660556] msdataset数据集加载 Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9552632 * load csv dataset from modelscoop 2022-07-29 12:22:48 +08:00			`oss2`
Release/1.16 merge to master (#922) * update version * merge master * fix fairseq issue * fix pip compatible issue * fix bug * version to 1.16.1 * xformers<0.0.27 * fix matplotlib.cm compatible issue * remove transformers from frameowrk * vllm to 0.5.1 * not upgrade pip * add Pillow to framework * Update version.py --------- Co-authored-by: xingjun.wang <xingjun.wxj@alibaba-inc.com> Co-authored-by: mulin.lyh <mulin.lyh@taobao.com> 2024-07-22 20:58:32 +08:00			`Pillow`
Add requirements for jsonplus Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12331254 2023-04-14 14:05:36 +08:00			`python-dateutil>=2.1`
[to #41669377] add speech AEC pipeline Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8973072 * [to #41669377] docs and tools refinement and release 1. add build_doc linter script 2. add sphinx-docs support 3. add development doc and api doc 4. change version to 0.1.0 for the first internal release version Link: https://code.aone.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8775307 * [to #41669377] add pipeline tutorial and fix bugs 1. add pipleine tutorial 2. fix bugs when using pipeline with certain model and preprocessor Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8814301 * refine doc * feat: add audio aec pipeline and preprocessor * feat: add audio aec model classes * feat: add audio aec loss functions * refactor:delete no longer used loss function * [to #42281043] support kwargs in pipeline Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8949062 * support kwargs in pipeline * update develop doc with CR instruction * Merge branch 'release/0.1' into dev/aec * style: reformat code by pre-commit tools * feat:support maas_lib pipeline auto downloading model * test:add aec test case as sample code * feat:aec pipeline use config from maashub * feat:aec pipeline use feature parameters from maashub * update setup.cfg to disable PEP8 rule W503 in flake8 and yapf * format:fix double quoted strings, indent issues and optimize import * refactor:extract some constant in aec pipeline * refactor: delete no longer used __main__ statement * chore:change all Chinese comments to English * fix: change file name style to lower case * refactor: rename model name * feat:load C++ .so from LD_LIBRARY_PATH * feat:register PROPROCESSOR for LinearAECAndFbank * refactory:move aec process from postprocess() to forward() and update comments * refactory:add more readable error message when audio sample rate is not 16000 * fix: package maas_lib renamed to modelscope in import statement * feat: optimize the error message of audio layer classes * format: delete empty lines * refactor: rename audio preprocessor and optimize error message * refactor: change aec model id to damo/speech_dfsmn_aec_psm_16k * refactor: change sample audio file url to public oss * Merge branch 'master' into dev/aec * feat: add output info for aec pipeline * fix: normalize output audio data to [-1.0, 1.0] * refactor:use constant from ModelFile * feat: AEC pipeline can use c++ lib in current working directory and the test will download it * fix: c++ downloading should work wherever test is triggerd 2022-06-17 19:56:11 +08:00			`scipy`
add dataset download (#906) * add dataset download * fix cr issue * fix cv matplotlib issue * refactor code * fix ut issue * remove debug code * remove unused import * fix import issue * sleep 65s before start docker avoid kill and run failed --------- Co-authored-by: mulin.lyh <mulin.lyh@taobao.com> 2024-07-12 19:06:44 +08:00			`# latest version has some compatible issue.`
			`setuptools==69.5.1`
Add requirements for jsonplus Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12331254 2023-04-14 14:05:36 +08:00			`simplejson>=3.3.0`
Add sortedcontainers to requirements Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12310786 2023-04-12 22:26:56 +08:00			`sortedcontainers>=1.5.9`
fix dataset page bug, framework add transformers dependency (#927) * fix dataset page bug, framework add transformers dependency Co-authored-by: mulin.lyh <mulin.lyh@taobao.com> 2024-07-26 19:03:39 +08:00			`transformers`
[to #50607174]feat: support parallel download large model file Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13113235 * [to #50607174]feat: support parallel download large model file 2023-06-30 15:30:00 +08:00			`urllib3>=1.26`