Mirrors/modelscope
1
0
Fork 0
mirror of https://github.com/modelscope/modelscope.git synced 2025-12-16 08:17:45 +01:00
Code Issues Packages Projects Releases Wiki Activity
2,407 Commits 262 Branches 106 Tags
543a2b32d79dc0c87fbfe67e0fb87b2a6fcf88c8
Commit Graph

44 Commits

Author SHA1 Message Date
liuyhwangyh
4e2555c5a3 Release/1.16 merge to master (#922) 
* update version

* merge master

* fix fairseq issue

* fix pip compatible issue

* fix bug

* version to 1.16.1

* xformers<0.0.27

* fix matplotlib.cm compatible issue

* remove transformers from frameowrk

* vllm to 0.5.1

* not upgrade pip

* add Pillow to framework

* Update version.py

---------

Co-authored-by: xingjun.wang <xingjun.wxj@alibaba-inc.com>
Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>
 2024-07-22 20:58:32 +08:00
liuyhwangyh
f7a32e48e3 add dataset download (#906) 
* add dataset download

* fix cr issue

* fix cv matplotlib issue

* refactor code

* fix ut issue

* remove debug code

* remove unused import

* fix import issue

* sleep 65s before start docker avoid kill and run failed

---------

Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>
 2024-07-12 19:06:44 +08:00
liuyhwangyh
ff77845772 add command line usage, optimize command line log, optimize packaging… (#888) 
* add command line usage, optimize command line log, optimize packaging version compatible

* Update command.md

* fix logging dup

* Refactor install dependencies (#889)

* refactor install dependencies, default only depencies of hub and datasets

* move pandas import to function

* update hub deps

* test

* remove generate ast file

* remove gast dependency

* replace gast with ast

* add dependency version

* remove lap for compile error

* fix comments issue

* add install ollama

---------

Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>

---------

Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>
 2024-06-30 20:15:03 +08:00
mulin.lyh
909e54fa33 modify build script 2024-06-05 19:08:46 +08:00
liuyhwangyh
5912cf3132 upgrade image build (#773) 
* upgrade image build

* add timm upgrade

---------

Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>
 2024-02-23 16:24:54 +08:00
liuyhwangyh
672c32e7bd fix ci compatible issues，fix llmpipeline lazy import issue (#725) 
* fix ci issue

* fix case issue

* modify lint to python3.10

* fix case issue

---------

Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>
 2024-01-17 22:19:05 +08:00
ly119399
26984de221 fix bug of branch release/1.11 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/15342815
* remove DOCKER_BUILDKIT=0 for cpu build issue

* force upgrade vllm

* add install sudo

* update transformers to 4.36.2

* fix mmcv-full issue

* add tiny_cuda_nn build

* install tinycudann

* reset
 2024-01-10 22:24:19 +08:00
mulin.lyh
228ef8afa3 add instal fastapi for server support 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/15248194
* add instal fastapi for server support
 2024-01-04 14:36:07 +08:00
mulin.lyh
385486daf2 upgrade cuda to 12.1.0 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/15183132
* upgrade cuda to 12.1.0

* remove tb-nightly, remove dup compile megatron util

* uninstall tb-nightly reinstall tensorboard
 2023-12-28 17:47:58 +08:00
mulin.lyh
fe8bfa9219 when build force install funasr pai-eacv etc 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/14812168
* when build force install funasr pai-eacv etc
 2023-11-29 17:40:09 +08:00
mulin.lyh
5ba9fd2307 modify auto gptq and vllm env 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/14790283
* upgrade to python3.10

* modify auto gptq and vllm env

* fix lint issue

* Merge remote-tracking branch 'origin/master' into python10_support

* python310 support

* build from repo

* add commit id force install modelscope every build

* add commit id force install modelscope every build

* fix cpu build issue

* fix datahub error message

* Merge branch 'python10_support' of gitlab.alibaba-inc.com:Ali-MaaS/MaaS-lib into python10_support

* add --no-cache-dir install auto_gptq
 2023-11-27 20:21:00 +08:00
mulin.lyh
087cb4e463 upgrade flash attention to 2.32.2 2023-10-16 16:13:28 +08:00
mulin.lyh
74532181d0 add jupyterlab install to docker 2023-09-28 17:30:38 +08:00
mulin.lyh
e7ecb55e00 remove onnxruntime version 2023-09-28 14:30:55 +08:00
mulin.lyh
f6d4107b0a fix onnxruntime issue 2023-09-28 10:49:51 +08:00
mulin.lyh
31d04f45b7 force reinstall 1.9.2 2023-09-28 01:06:32 +08:00
mulin.lyh
0075684692 add use gpu arg 2023-09-27 13:51:11 +08:00
mulin.lyh
6944d41734 skip install detection2 in cpu 2023-09-27 11:47:23 +08:00
mulin.lyh
dd9fa8b69c add install swift from oss 2023-09-26 16:40:26 +08:00
mulin.lyh
b02f941a02 modify version number 2023-09-15 15:10:29 +08:00
mulin.lyh
f05707ff33 fix transformers compatible issue of position_ids 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13974608
* fix transformers compatible issue of position_ids
 2023-09-11 13:37:10 +08:00
mulin.lyh
c11bbb03fb fix docker file bug 2023-09-01 18:05:11 +08:00
mulin.lyh
ad1081b08e fix pipeline and image build issue 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13884178
* fix pipeline and image build issue
 2023-09-01 18:02:50 +08:00
mulin.lyh
cba4e40bc1 fix numpy pandas compatible issue 
明确受影响的模型(damo)：  
ONE-PEACE-4B	ModuleNotFoundError: MyCustomPipeline: MyCustomModel: No module named 'one_peace'，缺少依赖。
cv_resnet50_face-reconstruction	 不兼容tf2  
nlp_automatic_post_editing_for_translation_en2de	tf2.0兼容性问题，tf1.x需要  
cv_resnet18_ocr-detection-word-level_damo	tf2.x兼容性问题  
cv_resnet18_ocr-detection-line-level_damo	tf兼容性问题  
cv_resnet101_detection_fewshot-defrcn	模型限制必须detection0.3+torch1.11.0"  
speech_dfsmn_ans_psm_48k_causal	"librosa， numpy兼容性问题  
cv_mdm_motion-generation	"依赖numpy版本兼容性问题：   File ""/opt/conda/lib/python3.8/site-packages/smplx/body_models.py"",  
cv_resnet50_ocr-detection-vlpt	numpy兼容性问题  
cv_clip-it_video-summarization_language-guided_en	tf兼容性问题

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13744636
* numpy and pandas no version

* modify compatible issue

* fix numpy compatible issue

* modify ci

* fix lint issue

* replace Image.ANTIALIAS to Image.Resampling.LANCZOS pillow compatible

* skip uncompatible cases

* fix numpy compatible issue, skip cases that can not compatbile numpy or tensorflow2.x

* skip compatible cases

* fix clip model issue

* fix body 3d keypoints compatible issue
 2023-08-22 23:04:31 +08:00
mulin.lyh
c1df82acd8 init pip to aliyuncs 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13487845

* init pip to aliyuncs
 2023-08-01 16:55:33 +08:00
mulin.lyh
da2badaacd [to #51274424]fix: easycv extension and tinycudann conflict 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13471150
* fix easycv extension and tinycudann conflict
 2023-07-31 18:08:53 +08:00
mulin.lyh
a780935317 torch1.11 and torch2.0.1 build script refactor 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13408751

* [to #51122016]feat: build image support torch1.x 2.x cuda11.3.0 11.7.1

* remove base  version

* env cudatookit-version

* build script merge

* torch 1.11.0 fix
 2023-07-25 21:15:00 +08:00
mulin.lyh
25a7398aea docker file py38 and py37 compatible merge 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12722824
* debug

* add missing deps

* add stanza

* fix numpy issue

* numpy version

* tf case

* numpy <= 1.22.0

* fix tf 1.15.5 case

* add python38 and python37 compatible docker file

* cv add fvcore

* add easycv

* remove debug code

* fix style issue

* fix python3.7 python3.8 compatible issue

* comment

* modify run config

* scipy<=1.7.3

* revert scipy<=1.7.3

* fix compatible bug

* fix compatible bug

* add easycv

* update version: 1.6.1rc0
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12754942
fix python3.8 and python3.8 compatible issue

* numpy version change

* numpy version

* numpy version

* fix numpy version

* fix numpy version

* update version: 1.6.1rc0
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12754942
fix python3.8 and python3.8 compatible issue

* restore setup.py

* restore setup.py

* add build base image

* fix style

* numpy version fix

* optimize build image, split base and modelscope image

* fix style issue

* modify example path

* modify examples folder

* add SETUPTOOLS_USE_DISTUTILS=stdlib comments
 2023-06-08 10:01:58 +08:00
mulin.lyh
86128cd3a8 [to #47612577]feat: image supoort 3d models, python3.8 test version 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11542009


* mmcv-full<=1.7.0 for mmdet3d

* fix code style issue

* add speechbrain==0.5.7

* add compatible comments
 2023-02-07 02:28:20 +00:00
wenmeng.zwm
ed859e5274 Title: merge master-github and fix conflict 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11370549
 2023-01-10 14:03:08 +08:00
mulin.lyh
04b9704b48 [to #47190649]fix: merge v1.1 docker file 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11371422

    * v1.1 docker file modify
 2023-01-10 14:02:15 +08:00
wenmeng.zwm
9ce750f4a9 merge master-github and fix conflict 2023-01-10 11:12:37 +08:00
xingjun.wxj
43edddd31f [to #42322933] msdataset module refactor and add 1230 features 
1. 优化本地数据集加载链路  
2. local与remote解耦，无网络环境下也可以使用SDK  
3. 升级hf datasets及其相关依赖到最新版(2.7.0+)
4. 解决元数据感知不到数据文件变更的问题  
5. 系统分层设计
6. 本地缓存管理问题  
7. 优化error log输出信息  
8. 支持streaming load	
* a. 支持数据文件为zip格式的streaming
* b. 支持Image/Text/Audio/Biodata等格式数据集的iter
* c. 兼容训练数据在meta中的历史数据集的streaming load
* d. 支持数据文件为文件夹格式的streaming load

9. finetune任务串接进一步规范
* a. 避免出现to_hf_dataset这种使用，将常用的tf相关的func封装起来  
* b. 去掉了跟hf混用的一些逻辑，统一包装到MsDataset里面

10. 超大数据集场景优化
* a. list oss objects: 直接拉取meta中的csv mapping，不需要做 list_oss_objects的api调用（前述提交已实现）
* b. 优化sts过期加载问题（前述提交已实现）

11. 支持dataset_name格式为：namespace/dataset_name的输入方式

参考Aone链接： https://aone.alibaba-inc.com/v2/project/1162242/task/46262894
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11264406
 2023-01-10 07:01:34 +08:00
chenxujun
d510683475 Update modelscope_env_init.sh 
Fix typo
 2022-12-03 06:59:31 +08:00
mulin.lyh
eb37921ad6 [to #45915958]feat: v1.0.2 docker file update, add deepspeed, apex, uncore 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10617428
 2022-11-11 11:03:00 +08:00
zfm104435
08327d5af9 在modelscope 基础镜像中添加 jupyter lab 插件，支持静默关闭功能 
在modelscope 基础镜像中添加 jupyter lab 插件，用于收集用户在notebook 中的鼠标、键盘事件，来支持静默关闭功能。
主要改变如下：
1. 新增jupyterlab_active_log 插件的源码和编译后的文件
2. 修改Dockerfile.ubuntu 文件，将上述源码文件拷贝的jupyter的插件目录中。
3. 修改pip和conda 的镜像地址为阿里云的
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10690152
 2022-11-10 22:10:13 +08:00
mulin.lyh
ac07b719e9 [to #45546922]feat: add fasttext package 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10431169

    * [to #45546922]feat: add fasttext package
 2022-10-17 20:51:58 +08:00
mulin.lyh
c51b74c2ea [to #45220645]fix: fix ffmpeg mp4 encoder bug 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10284398

    * [to #45220645]fix: fix ffmpeg mp4 encoder bug
 2022-09-28 13:36:09 +08:00
mulin.lyh
ad6bb1e7d9 [to #44790143]fix: add ipythonkernel to image for dsw 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10049527

    * add ipykernel to image for dsw
 2022-09-07 20:51:15 +08:00
wenmeng.zwm
ba74cdf97e [to #43878347] Rename runtime.txt to framework.txt 
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10000642

    * rename runtime.txt  to framework.txt
 2022-09-03 12:10:16 +08:00
mulin.lyh
01bb751425 [to #43653669]feat: auto build docker images 
auto build docker images
aone 任务： https://test.aone.alibaba-inc.com/jobs/1824567?buildId=143470479
修改任务分支
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9566518

    * [to #43653669]feat: auto build docker images
 2022-08-22 16:01:42 +08:00
wenmeng.zwm
8e51a073a6 [to #42966122] requirements enchanment and self-host repo support 
* add self-hosted repo:
* add extra requirements for different field and reduce necessary requirements
* update docker file with so required by audio
* add requirements checker which will be used later when implement lazy import
* remove repeated requirements and replace opencv-python-headless with opencv-python

example usage:
```shell
pip install model_scope[all] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[cv] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[nlp] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[audio] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html
pip install model_scope[multi-modal] -f https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/release/maas/repo.html

```
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9211383
 2022-07-01 16:38:06 +08:00
wenmeng.zwm
4f7928bb6e [to #42362853] formalize the output of pipeline and make pipeline reusable 
* format pipeline output and check it
* fix UT
* add docstr to clarify the difference between model.postprocess and pipeline.postprocess

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9051405
 2022-06-16 11:15:09 +08:00
wenmeng.zwm
67086e26f9 [to #42362932] feat: docker_support 
* add dockerfile
* uninstall opencv-python-headless
* update develop doc
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9020302
 2022-06-14 16:29:12 +08:00