Yunnglin
bb3768cabb
update setup file
2025-11-11 13:06:22 +08:00
Yunlin Mao
b034267345
[Feature] Add scan_cache_dir function ( #1348 )
...
* init cache manager
* add scan cache
* fix lint
* add scan cache unit test
* add download first for scan cache test
2025-05-28 09:59:03 +08:00
Yingda Chen
83f1e20e80
add repo existence check hub-api ( #1060 )
...
* add repo existence check api
* update
---------
Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com >
2024-11-02 08:04:40 +08:00
liuyhwangyh
f7a32e48e3
add dataset download ( #906 )
...
* add dataset download
* fix cr issue
* fix cv matplotlib issue
* refactor code
* fix ut issue
* remove debug code
* remove unused import
* fix import issue
* sleep 65s before start docker avoid kill and run failed
---------
Co-authored-by: mulin.lyh <mulin.lyh@taobao.com >
2024-07-12 19:06:44 +08:00
liuyhwangyh
ff77845772
add command line usage, optimize command line log, optimize packaging… ( #888 )
...
* add command line usage, optimize command line log, optimize packaging version compatible
* Update command.md
* fix logging dup
* Refactor install dependencies (#889 )
* refactor install dependencies, default only depencies of hub and datasets
* move pandas import to function
* update hub deps
* test
* remove generate ast file
* remove gast dependency
* replace gast with ast
* add dependency version
* remove lap for compile error
* fix comments issue
* add install ollama
---------
Co-authored-by: mulin.lyh <mulin.lyh@taobao.com >
---------
Co-authored-by: mulin.lyh <mulin.lyh@taobao.com >
2024-06-30 20:15:03 +08:00
liuyhwangyh
17da5e2264
fix error report ( #868 )
...
Co-authored-by: mulin.lyh <mulin.lyh@taobao.com >
2024-05-28 14:38:19 +08:00
mulin.lyh
5ba9fd2307
modify auto gptq and vllm env
...
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/14790283
* upgrade to python3.10
* modify auto gptq and vllm env
* fix lint issue
* Merge remote-tracking branch 'origin/master' into python10_support
* python310 support
* build from repo
* add commit id force install modelscope every build
* add commit id force install modelscope every build
* fix cpu build issue
* fix datahub error message
* Merge branch 'python10_support' of gitlab.alibaba-inc.com:Ali-MaaS/MaaS-lib into python10_support
* add --no-cache-dir install auto_gptq
2023-11-27 20:21:00 +08:00
mulin.lyh
8c80b0c3f5
support master branch version and add http request id
...
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/14072105
* support master branch version and add http request id
* modify no revision use master
* add specified revision="master"
* error message add request id
2023-09-20 19:29:30 +08:00
yuze.zyz
f856c04785
Ignore http error
...
忽略log打印，防止在check_model_is_latest时打出信息使用户不明所以。
将log改为Error栈
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13463005
2023-07-31 17:14:12 +08:00
mulin.lyh
ba646ef1d2
display hub error message
...
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12581373
2023-05-10 20:58:44 +08:00
xingjun.wxj
43edddd31f
[to #42322933 ] msdataset module refactor and add 1230 features
...
1. 优化本地数据集加载链路
2. local与remote解耦，无网络环境下也可以使用SDK
3. 升级hf datasets及其相关依赖到最新版(2.7.0+)
4. 解决元数据感知不到数据文件变更的问题
5. 系统分层设计
6. 本地缓存管理问题
7. 优化error log输出信息
8. 支持streaming load
* a. 支持数据文件为zip格式的streaming
* b. 支持Image/Text/Audio/Biodata等格式数据集的iter
* c. 兼容训练数据在meta中的历史数据集的streaming load
* d. 支持数据文件为文件夹格式的streaming load
9. finetune任务串接进一步规范
* a. 避免出现to_hf_dataset这种使用，将常用的tf相关的func封装起来
* b. 去掉了跟hf混用的一些逻辑，统一包装到MsDataset里面
10. 超大数据集场景优化
* a. list oss objects: 直接拉取meta中的csv mapping，不需要做 list_oss_objects的api调用（前述提交已实现）
* b. 优化sts过期加载问题（前述提交已实现）
11. 支持dataset_name格式为：namespace/dataset_name的输入方式
参考Aone链接： https://aone.alibaba-inc.com/v2/project/1162242/task/46262894
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11264406
2023-01-10 07:01:34 +08:00
mulin.lyh
ab07dc5b5a
google style docs and selected file generator
...
ref: https://yuque.alibaba-inc.com/pai/rwqgvl/go8sc8tqzeqqfmsz
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11150212
* google style docs and selected file generator
2023-01-03 16:27:29 +08:00
mulin.lyh
3b75623be4
[to #45773874 ]fix: get_model revision=None bug, and hub case occasionally delete test model failed
...
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10549680
2022-10-27 17:06:18 +08:00
mulin.lyh
384377b8f5
* [to #45486649 ]feat: modelscope model version use model repo tag, unsupport branch or commit it, client user-agent header unified
2022-10-26 13:55:51 +08:00
mulin.lyh
7257f6c6fb
[to #45631658 ]feat support eas deploy
...
服务端文档链接(可能需要登录)：
https://test.modelscope.cn/api/v1/deployer/docs
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10478609
2022-10-24 15:12:48 +08:00
mulin.lyh
ac07b719e9
[to #45546922 ]feat: add fasttext package
...
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10431169
* [to #45546922 ]feat: add fasttext package
2022-10-17 20:51:58 +08:00
mulin.lyh
5bdb8fb78b
[to #45451935 ]fix: add create model detail log for create failed.
...
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10382795
2022-10-13 18:30:06 +08:00
wenmeng.zwm
6808e9a301
[to #44902099 ] add license for framework files
...
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10189613
2022-09-20 17:49:31 +08:00
xingjun.wxj
7cb72cc46e
[to #42322933 ]MsDataset upload bugfix for 0830 version.
...
CR link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10109035#tab=changes&file=8348e8153b2f4a6dbd52e471b4980542355408ed
Please refer to aone links:
1. https://aone.alibaba-inc.com/v2/project/1162242/bug#viewIdentifier=b622c099e2199bc034401fbe&openWorkitemIdentifier=44889184
2. https://aone.alibaba-inc.com/v2/project/1162242/bug#viewIdentifier=b622c099e2199bc034401fbe&openWorkitemIdentifier=44858810
3. https://aone.alibaba-inc.com/v2/project/1162242/bug#viewIdentifier=b622c099e2199bc034401fbe&openWorkitemIdentifier=44857728
4. https://aone.alibaba-inc.com/v2/project/1162242/bug#viewIdentifier=b622c099e2199bc034401fbe&openWorkitemIdentifier=44658972
2022-09-14 19:24:48 +08:00
tanfan.zjh
dc45fce542
[to #42322933 ]新增FAQ问答模型
...
Maas新增FAQ问答模型
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9797053
2022-08-26 13:06:41 +08:00
mulin.lyh
6cf0a56ade
[to #43913168 ]fix: add file download integrity check
...
添加文件下载完整性验证
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9700279
* [to #43913168 ]fix: add file download integrity check
2022-08-11 10:45:04 +08:00
mulin.lyh
47dda0a5f9
[to #43698011 ]feat: login with access token
2022-08-02 18:16:28 +08:00
mulin.lyh
69047b99ae
[to #43387011 ]feat: ci test to new host and running in docker
...
ci 测试迁移新的机器，并且在容器中运行，减小互相干扰的可能
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9427096
* add docker ci script
2022-07-20 11:49:05 +08:00
mulin.lyh
3eed264bad
[to #43040150 ] fix: login warn refactor, error message when exception, ut case to new user
...
未登录warn提示信息重构，只有遇到请求失败，exception时，如果cookie为空，提示用户login，单元测试用户修改成单独
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9283579
* [to #42322933 ] add space dialog-state tracking pipeline
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9227018
* init
* token to ids
* add model
* model forward ready
* add intent
* intent preprocessor ready
* intent success
* merge master
* test with model hub
* add flake8
* update
* update
* update
* Merge branch 'master' into nlp/space/gen
* delete file about gen
* init
* fix flake8 bug
* [to #42322933 ] init
* bug fix
* [to #42322933 ] init
* update pipeline registry info
* Merge remote-tracking branch 'origin/master' into feat/nli
* [to #42322933 ] init
* [to #42322933 ] init
* modify forward
* [to #42322933 ] init
* generation ready
* init
* Merge branch 'master' into feat/zero_shot_classification
# Conflicts:
# modelscope/preprocessors/__init__.py
* [to #42322933 ] bugfix
* [to #42322933 ] pre commit fix
* fill mask
* registry multi models on model and pipeline
* add tests
* test level >= 0
* local gen ready
* merge with master
* dialog modeling ready
* fix comments: rename and refactor AliceMindMLM; adjust pipeline
* space intent and modeling(generation) are ready
* bug fix
* add dep
* add dep
* support dst data processor
* merge with nlp/space/dst
* merge with master
* Merge remote-tracking branch 'origin' into feat/fill_mask
Conflicts:
modelscope/models/nlp/__init__.py
modelscope/pipelines/builder.py
modelscope/pipelines/outputs.py
modelscope/preprocessors/nlp.py
requirements/nlp.txt
* merge with master
* merge with master 2/2
* fix comments
* fix isort for pre-commit check
* allow params pass to pipeline's __call__ method
* Merge remote-tracking branch 'origin/master' into feat/zero_shot_classification
* merge with nli task
* merge with sentiment_classification
* merge with zero_shot_classfication
* merge with fill_mask
* merge with space
* merge with master head
* Merge remote-tracking branch 'origin' into feat/fill_mask
Conflicts:
modelscope/utils/constant.py
* fix: pipeline module_name from model_type to 'fill_mask' & fix merge bug
* unfiinished change
* fix bug
* unfinished
* unfinished
* revise modelhub dependency
* Merge branch 'feat/nlp_refactor' of http://gitlab.alibaba-inc.com/Ali-MaaS/MaaS-lib into feat/nlp_refactor
* add eval() to pipeline call
* add test level
* ut run passed
* add default args
* tmp
* merge master
* all ut passed
* remove an useless enum
* revert a mis modification
* revert a mis modification
* Merge commit 'ace8af92465f7d772f035aebe98967726655f12c' into feat/nlp
* commit 'ace8af92465f7d772f035aebe98967726655f12c':
[to #42322933 ] Add cv-action-recongnition-pipeline to maas lib
[to #42463204 ] support Pil.Image for image_captioning_pipeline
[to #42670107 ] restore pydataset test
[to #42322933 ] add create if not exist and add(back) create model example Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9130661
[to #41474818 ]fix: fix errors in task name definition
# Conflicts:
# modelscope/pipelines/builder.py
# modelscope/utils/constant.py
* Merge branch 'feat/nlp' into feat/nlp_refactor
* feat/nlp:
[to #42322933 ] Add cv-action-recongnition-pipeline to maas lib
[to #42463204 ] support Pil.Image for image_captioning_pipeline
[to #42670107 ] restore pydataset test
[to #42322933 ] add create if not exist and add(back) create model example Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9130661
[to #41474818 ]fix: fix errors in task name definition
# Conflicts:
# modelscope/pipelines/builder.py
* fix compile bug
* refactor space
* Merge branch 'feat/nlp_refactor' of http://gitlab.alibaba-inc.com/Ali-MaaS/MaaS-lib into feat/nlp_refactor
* Merge remote-tracking branch 'origin' into feat/fill_mask
* fix
* pre-commit lint
* lint file
* lint file
* lint file
* update modelhub dependency
* lint file
* ignore dst_processor temporary
* solve comment: 1. change MaskedLMModelBase to MaskedLanguageModelBase 2. remove a useless import
* recommit
* remove MaskedLanguageModel from __all__
* Merge commit '1a0d4af55a2eee69d89633874890f50eda8f8700' into feat/nlp_refactor
* commit '1a0d4af55a2eee69d89633874890f50eda8f8700':
[to #42322933 ] test level check Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9143809
[to #42322933 ] update nlp models name in metainfo Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9134657
# Conflicts:
# modelscope/metainfo.py
* update
* revert pipeline params update
* remove zeroshot
* update sequence classfication outpus
* merge with fill mask
* Merge remote-tracking branch 'origin' into feat/fill_mask
* fix
* init dialog state tracking
* fix flake8 warning of dst
* Merge remote-tracking branch 'origin/feat/fill_mask' into feat/nlp
* merge with master
* remove useless test.py
* add init
* merge nlp
* Merge remote-tracking branch 'origin/master' into feat/nlp
* remove unformatted space trainer
* Merge branch 'feat/nlp' into nlp/space/dst
* revise based on comment except chinease comment
* skip ci blocking
* change Chinese notes of space3.0 into English
* translate chinese comment to english
* add space to metainfo
* space dst pipeline is ready, but model's result is wrong
* merge feat/nlp
* merge with master
* change processor
* change example
* test case ready
* dst loacl ready
* update dst conf
* merge feat/nlp
* inform revise
* inherit bug fix
* init
* add 2 complete examples
* fix bug
* add test case
* merge with master
* modify model name
* add missing setting
* add outputs
* modify test level
* modify chinese comment
* remove useless doc
* merge feat/nlp
* Merge remote-tracking branch 'origin' into nlp/space/dst
* Merge branch 'feat/nlp' into nlp/space/dst
* dst test ready
* merge feat nlp
* space outputs normalization
* update dst
* merge feat nlp
* Merge branch 'master' into nlp/space/dst
* update requirement
* merge with master
* Merge remote-tracking branch 'origin/master' into nlp/space/dst
* formating output
* update requirements/nlp
* merge with master
* add test cases
* Merge remote-tracking branch 'origin/master' into nlp/space/dst
* merge with master
* login warn refactor, error message when exception, ut case to new user
* login warn refactor, error message when exception, ut case to new user
2022-07-08 20:03:23 +08:00
mulin.lyh
0d17eb5b39
[to #42849800 #42822853 #42822836 #42822791 #42822717 #42820011 ]fix: bug test bugs
...
修复测试bug
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9186775
* [to #42849800 #42822853 #42822836 #42822791 #42822717 #42820011 ]fix: test bugs
2022-06-28 21:12:15 +08:00
feiwu.yfw
5da470fd5d
[to #42791465 , #42779255 , #42777959 , #42757844 , #42756050 , #42746916 , #42743595 , #42791863 ] fix: fix msdataset
...
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9174075
* fix msdataset
2022-06-28 20:40:57 +08:00
mulin.lyh
6991620f59
[to #42698276 ]fix: git repo operations supports, gitlab token certification support.
2022-06-24 16:43:32 +08:00
mulin.lyh
76c6ff6329
[to #42675838 ]merge model hub code
...
合并model hub 代码
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9096493
2022-06-21 20:04:25 +08:00
