Mirrors/modelscope
1
0
Fork 0
mirror of https://github.com/modelscope/modelscope.git synced 2026-02-24 20:19:51 +01:00
Code Issues Packages Projects Releases Wiki Activity
1,842 Commits 267 Branches 107 Tags
dev/dataset_fix_version
Commit Graph

31 Commits

Author SHA1 Message Date
xingjun.wang
0bef1100a3 modify datasets version 2023-07-25 21:05:41 +08:00
xingjun.wang
49cd9a41d5 modify datasets version 2023-07-25 21:04:28 +08:00
mulin.lyh
3683a4386f [to #50607174]feat: support parallel download large model file 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13113235
* [to #50607174]feat: support parallel download large model file
 2023-06-30 15:30:00 +08:00
xingjun.wxj
0af8752024 Unfreeze datasets version setting 
1. Unfreeze datasets version setting
2. Fix some bugs in class NativeIterableDataset
3. Remove auto-converting (image/audio) object in col:FILE
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13080190
 2023-06-28 15:50:57 +08:00
mulin.lyh
25a7398aea docker file py38 and py37 compatible merge 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12722824
* debug

* add missing deps

* add stanza

* fix numpy issue

* numpy version

* tf case

* numpy <= 1.22.0

* fix tf 1.15.5 case

* add python38 and python37 compatible docker file

* cv add fvcore

* add easycv

* remove debug code

* fix style issue

* fix python3.7 python3.8 compatible issue

* comment

* modify run config

* scipy<=1.7.3

* revert scipy<=1.7.3

* fix compatible bug

* fix compatible bug

* add easycv

* update version: 1.6.1rc0
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12754942
fix python3.8 and python3.8 compatible issue

* numpy version change

* numpy version

* numpy version

* fix numpy version

* fix numpy version

* update version: 1.6.1rc0
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12754942
fix python3.8 and python3.8 compatible issue

* restore setup.py

* restore setup.py

* add build base image

* fix style

* numpy version fix

* optimize build image, split base and modelscope image

* fix style issue

* modify example path

* modify examples folder

* add SETUPTOOLS_USE_DISTUTILS=stdlib comments
 2023-06-08 10:01:58 +08:00
xingjun.wxj
5b99bb24fe update version: 1.6.1rc0 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12754942
 2023-05-25 20:28:36 +08:00
mulin.lyh
6abdce929d fix py38 issue 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12660340
 2023-05-17 22:11:23 +08:00
hemu.zp
1d28c5b730 mv mmdet to cv 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12614719
 2023-05-15 11:48:29 +08:00
hemu.zp
ca85447363 remove pai-easycv requirement 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12564234
 2023-05-11 14:31:18 +08:00
mulin.lyh
d772d96cf5 [to #49147498]feat: support python3.8 2023-05-06 18:01:02 +08:00
hemu.zp
b3e7eca817 lmit mmdet version 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12339472
 2023-04-15 17:08:55 +08:00
yuze.zyz
2650d37094 Fix the log level and return value of push_to_hub 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12338218
 2023-04-15 16:42:35 +08:00
hemu.zp
a4bafd8b25 Add requirements for jsonplus 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12331254
 2023-04-14 14:05:36 +08:00
hemu.zp
4e78f611e6 Add sortedcontainers to requirements 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12310786
 2023-04-12 22:26:56 +08:00
jiangnana.jnn
6f9d59f469 fix jsonplus, support ndarray 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12056076

* fix jsonplus

* add unittest

* fix json dump np float
 2023-03-21 18:10:10 +08:00
yuze.zyz
4a0cb67e66 fix numpy dependency 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11978446
 2023-03-14 13:31:22 +08:00
mulin.lyh
e54694690f [to #46993990]feat: run ci cases base on code diff to reduct ci test time 2023-02-06 08:00:19 +00:00
Yingda Chen
af62b3e9ad limit pyarrow version 2023-01-31 11:35:09 +08:00
yingda.chen
2e029a1eda update requirment and fix some typos 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11479462
 2023-01-31 01:23:56 +00:00
xingjun.wxj
5a01eca834 [to #42322933] fix datasets version incompatible issue 
最近release的datasets==2.9.0版本在IterableDataset等类和方法上，与老版本(<=2.8.0)不兼容；修改requirements将其版本限制在2.7.0~2.8.0

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11476999

* fix datasets==2.8.0 and datasets==2.9.0 incompatible issue
 2023-01-29 11:13:09 +00:00
xingjun.wxj
43edddd31f [to #42322933] msdataset module refactor and add 1230 features 
1. 优化本地数据集加载链路  
2. local与remote解耦，无网络环境下也可以使用SDK  
3. 升级hf datasets及其相关依赖到最新版(2.7.0+)
4. 解决元数据感知不到数据文件变更的问题  
5. 系统分层设计
6. 本地缓存管理问题  
7. 优化error log输出信息  
8. 支持streaming load	
* a. 支持数据文件为zip格式的streaming
* b. 支持Image/Text/Audio/Biodata等格式数据集的iter
* c. 兼容训练数据在meta中的历史数据集的streaming load
* d. 支持数据文件为文件夹格式的streaming load

9. finetune任务串接进一步规范
* a. 避免出现to_hf_dataset这种使用，将常用的tf相关的func封装起来  
* b. 去掉了跟hf混用的一些逻辑，统一包装到MsDataset里面

10. 超大数据集场景优化
* a. list oss objects: 直接拉取meta中的csv mapping，不需要做 list_oss_objects的api调用（前述提交已实现）
* b. 优化sts过期加载问题（前述提交已实现）

11. 支持dataset_name格式为：namespace/dataset_name的输入方式

参考Aone链接： https://aone.alibaba-inc.com/v2/project/1162242/task/46262894
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11264406
 2023-01-10 07:01:34 +08:00
wenmeng.zwm
070ec00720 remove opencv-python from framework requirements and remove easynlp from nlp default requirements 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11119626
 2022-12-16 14:27:23 +08:00
wenmeng.zwm
4dd99b8f6e Revert "move opencv dependency from framwork to cv " 
This reverts commit e970a6eb43.
 2022-11-30 18:29:03 +08:00
yingda.chen
e970a6eb43 move opencv dependency from framwork to cv 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10708368
 2022-11-12 20:09:48 +08:00
班扬
79c44a6810 add event tracking 2022-11-01 15:41:01 +08:00
班扬
84032f90e3 add event tracking 2022-11-01 15:34:58 +08:00
Yingda Chen
3464324f6b [to #42322933] limit datasets version for now 2022-10-31 22:15:37 +08:00
mulin.lyh
7257f6c6fb [to #45631658]feat support eas deploy 
服务端文档链接(可能需要登录)：
https://test.modelscope.cn/api/v1/deployer/docs
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10478609
 2022-10-24 15:12:48 +08:00
wenmeng.zwm
275f8b4323 Revert "[to #45071449] fix setup error " 
This reverts commit a26e6e3869.
 2022-10-14 13:55:09 +08:00
wenmeng.zwm
a26e6e3869 [to #45071449] fix setup error 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10196007
 2022-10-12 17:33:03 +08:00
wenmeng.zwm
ba74cdf97e [to #43878347] Rename runtime.txt to framework.txt 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10000642

    * rename runtime.txt  to framework.txt
 2022-09-03 12:10:16 +08:00