Commit Graph

76 Commits

Author SHA1 Message Date
Yunnglin
52e227c48b fix(docker): force numpy>=2.0 after evalscope install
ms-opencompass pulls numpy<2.0, downgrading numpy from 2.x to 1.26.4.
Force reinstall numpy>=2.0 after pip install .[eval] to restore it.
2026-06-15 17:43:30 +08:00
Jintao Huang
f599019206 fix cpu image 2026-06-11 15:15:28 +08:00
Jintao Huang
faae851237 update docker 2026-06-11 15:15:23 +08:00
Yunnglin
1a53418643 Add ipywidgets dependency to Docker images 2026-06-10 14:16:53 +08:00
Xingjun.Wang
680c3638e0 [Feat & Refactor] Refactor hub and CLI modules (#1732)
* refactor(hub): shim layer delegating to modelscope-hub

- Replace hub/api.py (4674→250 lines) with shim inheriting LegacyHubApi
- Replace hub/snapshot_download.py, callback.py with thin shims
- Partial shim hub/file_download.py (retain http_get_file)
- Shim hub/constants.py and errors.py with legacy aliases
- Shim hub/git.py, repository.py, cache_manager.py, upload_*.py
- Migrate CLI entry to modelscope_hub.cli.main:run_cmd
- Adapt 6 CLI commands as modelscope_hub.cli_plugins
- Delete redundant CLI files (download/upload/login/create/etc)
- Add modelscope-hub>=0.2.0 dependency, Python>=3.10
- Add __getattr__ proxy for forward-compatible method access
- Propagate timeout/max_retries to internal LegacyClient
- Bridge MODELSCOPE_CREDENTIALS_PATH env var to HubConfig

* fix lint: isort/yapf formatting + exclude hub/api.py from hooks

* set modelscope-hub>=0.0.5

* remove unused code

* refactor(hub): standardize token naming — git_token vs token

Disambiguate git token and SDK/API token naming across the hub layer:
- ModelScopeConfig: get_token/save_token → get_git_token/save_git_token
  (old names kept as deprecated aliases with DeprecationWarning)
- GitCommandWrapper: rename token params to git_token in clone/push/config
- Repository/DatasetRepository: auth_token → git_token (deprecated compat kept)
- data_loader.py: update caller to use get_git_token()

SDK token references (HubApi(token=...), get_cookies(access_token=...),
commit_scheduler.token) remain unchanged as they correctly use `token` naming.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* remove(msdatasets): remove all Virgo-related implementation

Remove the entire Virgo dataset subsystem which is no longer needed:
- Remove VirgoDataset class and VirgoDownloader
- Remove VirgoAuthConfig and VirgoDatasetConfig
- Remove Hubs.virgo enum value
- Remove fetch_virgo_meta from DataMetaManager
- Remove download_virgo_files from DatasetContextConfig
- Remove test_virgo_dataset.py test file
- Clean up unused imports (pandas, MaxComputeUtil, valid_url, etc.)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(hub): add OSS dataset operations and meta-file download to HubApi

Add methods that msdatasets depends on but don't belong in modelscope_hub:
- _legacy_request: internal helper combining legacy HTTP transport with
  application-level envelope validation (Code/Data/Message)
- list_oss_dataset_objects: list OSS storage objects for a dataset
- delete_oss_dataset_object / delete_oss_dataset_dir: delete OSS objects
- fetch_meta_files_from_url: download and cache meta CSV/JSONL files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix imports issue

* fix: address PR review feedback

- cli/plugins.py: change --yes and --all flags to action='store_true'
- hub/git.py: replace os.linesep with .splitlines() for cross-platform safety
- hub/__init__.py: use is_file() with fallback for robust credentials path detection

* fix lint

* update ms hub version

* fix(ci): add PyPI official as fallback index for pip

Aliyun mirror may lag behind PyPI for newly published packages,
causing dependency resolution failures (e.g. modelscope-hub>=0.0.6).
Add pypi.org/simple as extra-index-url so new versions are immediately
available while keeping the Aliyun mirror as the primary source.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix UTs

* remove unused UTs

* fix ut

* update modelscope-hub installation for source code

* fix UT

* fix uts

* fix ut

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-09 22:23:14 +08:00
Seas0
204caba8b2 fix: quote version of timm specs in Dockerfile.ubuntu (#1728)
Co-authored-by: Seas0 <liusichen.seas0@bytedance.com>
2026-06-09 20:00:52 +08:00
Yunnglin
b87b344f47 Upgrade numpy to 2.x for 1.38 Docker images
- Replace deprecated numpy aliases (np.math.ceil → math.ceil, np.Inf → np.inf)
- Upgrade Docker constraints: numpy>=2.0, cython>=3.0, remove scipy upper bound
2026-06-08 14:52:00 +08:00
tastelikefeet
a420e68953 Merge branch 'release/1.34'
# Conflicts:
#	docker/build_image.py
2026-03-13 11:36:25 +08:00
tastelikefeet
2fedc9bb9e Feat/update docker file (#1640) 2026-03-09 14:21:38 +08:00
hjh0119
ae1e83febb remove ollama 2026-01-15 20:53:05 +08:00
hjh0119
c137ed08c2 docker zstd 2026-01-15 20:33:27 +08:00
suluyana
09bc99c8c3 fix: Remove nvidia.cn from APT sources to fix apt failures in certain environments (#1575)
Co-authored-by: suluyan <suluyan.sly@aliabab-inc.com>
2026-01-09 16:09:23 +08:00
Yunlin Mao
9cb7ce7029 Update build file and workflow (#1551)
* update build file

* update setup file

* update fairseq dep

* fix error log
2025-12-02 10:50:39 +08:00
Jintao
2f3ac0fcd2 [docker] update swift docker (#1502) 2025-09-16 12:49:29 +08:00
tastelikefeet
6b70c8b577 Merge 1.29 to master (#1469) 2025-08-18 13:51:00 +08:00
tastelikefeet
3bad15b779 add ms-agent into dockerfile (#1436) 2025-08-04 17:08:48 +08:00
suluyan
7230561cbd image deps upgrade: transformers, huggingface_hub, node.js 2025-07-03 15:59:20 +08:00
Yunlin Mao
093ae87111 Update 1.26.0 docker image dependency (#1339) 2025-05-28 17:36:07 +08:00
Jintao
03dbc7bd11 Merge release1.25 to master (#1319) 2025-04-23 13:40:54 +08:00
Jintao
32aefd7428 Merge release1.24 to master (#1298) 2025-04-07 13:32:09 +08:00
Jintao
3bfff5cb53 Add swift docker (#1299) 2025-04-07 13:24:18 +08:00
tastelikefeet
e733458746 Update omegaconf (#1234) 2025-02-19 18:11:23 +08:00
tastelikefeet
7246d0b359 Merge 1.23 hotfix to master (#1227) 2025-02-17 10:30:24 +08:00
tastelikefeet
b5bb6d7bb0 Use legacy cache (#1215) 2025-02-07 17:31:32 +08:00
Jintao
b1e1f66fad Merge release 1.20 docker (#1109) 2024-11-29 13:44:36 +08:00
Jintao
e961f1671d Update docker from release/1.20 (#1077) 2024-11-11 14:56:36 +08:00
Jintao
61c2fd97e4 Update llm docker (#1076) 2024-11-08 18:09:05 +08:00
Jintao
d02d887918 update docker (#1073) 2024-11-08 10:56:39 +08:00
Jintao
125f44fb20 update docker evalscope version (#1071) 2024-11-07 18:56:17 +08:00
Jintao
a51f25abdf fix docker numpy version (#1069) 2024-11-07 13:58:08 +08:00
tastelikefeet
33308cedd4 fix numpy build error (#1068) 2024-11-06 20:40:58 +08:00
tastelikefeet
6856b157ed try to reduce the image size of llm (#1067) 2024-11-06 15:52:08 +08:00
tastelikefeet
43536064d5 default install tf-keras (#1064) 2024-11-04 20:15:18 +08:00
tastelikefeet
328bbc0494 Fix some bugs (#1056)
* fix some bugs

* remove install tf-kersa

* fix

* use bin bash as default
2024-10-30 15:49:38 +08:00
tastelikefeet
3762ba3318 fix tests (#1053) 2024-10-28 19:50:33 +08:00
tastelikefeet
62ffc04215 Fix the slow downloading (#1051)
* fix slow download

* fix

* fix audio installation

* fix build

* fix

* test

* test

* test

* test

* fix

* fix

* fix

* fix
2024-10-25 13:37:01 +08:00
tastelikefeet
acc60bab1c Fix pypi mirror (#1048)
* update flow name

* fix

* fix

* update docker builder

* lint

* fix build

* fix cpu build
2024-10-24 10:35:10 +08:00
tastelikefeet
7a57ee418c Refine dockerfile (#1042) 2024-10-23 09:59:51 +08:00
tastelikefeet
320095144b Refactor Dockerfile (#1036) 2024-10-21 18:26:23 +08:00
Xingjun.Wang
e320b40a0d Fix/dockerfile (#1034)
* update version

* update build_image for swift

* add outlines in build_image.sh

* set outlines to 0.0.46 to avoid failure of building docker

* set vllm==0.5.2 in Dockerfile.ubuntu to fix CI issue

* update outlines version: <0.1

* update ms-swift installation in build_image.sh

* merge diff from release/1.19
2024-10-21 11:25:02 +08:00
liuyhwangyh
4e2555c5a3 Release/1.16 merge to master (#922)
* update version

* merge master

* fix fairseq issue

* fix pip compatible issue

* fix bug

* version to 1.16.1

* xformers<0.0.27

* fix matplotlib.cm compatible issue

* remove transformers from frameowrk

* vllm to 0.5.1

* not upgrade pip

* add Pillow to framework

* Update version.py

---------

Co-authored-by: xingjun.wang <xingjun.wxj@alibaba-inc.com>
Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>
2024-07-22 20:58:32 +08:00
liuyhwangyh
f7a32e48e3 add dataset download (#906)
* add dataset download

* fix cr issue

* fix cv matplotlib issue

* refactor code

* fix ut issue

* remove debug code

* remove unused import

* fix import issue

* sleep 65s before start docker avoid kill and run failed

---------

Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>
2024-07-12 19:06:44 +08:00
liuyhwangyh
ff77845772 add command line usage, optimize command line log, optimize packaging… (#888)
* add command line usage, optimize command line log, optimize packaging version compatible

* Update command.md

* fix logging dup

* Refactor install dependencies (#889)

* refactor install dependencies, default only depencies of hub and datasets

* move pandas import to function

* update hub deps

* test

* remove generate ast file

* remove gast dependency

* replace gast with ast

* add dependency version

* remove lap for compile error

* fix comments issue

* add install ollama

---------

Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>

---------

Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>
2024-06-30 20:15:03 +08:00
mulin.lyh
909e54fa33 modify build script 2024-06-05 19:08:46 +08:00
liuyhwangyh
5912cf3132 upgrade image build (#773)
* upgrade image build

* add timm upgrade

---------

Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>
2024-02-23 16:24:54 +08:00
liuyhwangyh
672c32e7bd fix ci compatible issues,fix llmpipeline lazy import issue (#725)
* fix ci issue

* fix case issue

* modify lint to python3.10

* fix case issue

---------

Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>
2024-01-17 22:19:05 +08:00
ly119399
26984de221 fix bug of branch release/1.11
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/15342815
* remove DOCKER_BUILDKIT=0 for cpu build issue

* force upgrade vllm

* add install sudo

* update transformers to 4.36.2

* fix mmcv-full issue

* add tiny_cuda_nn build

* install tinycudann

* reset
2024-01-10 22:24:19 +08:00
mulin.lyh
228ef8afa3 add instal fastapi for server support
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/15248194
* add instal fastapi for server support
2024-01-04 14:36:07 +08:00
mulin.lyh
385486daf2 upgrade cuda to 12.1.0
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/15183132
* upgrade cuda to 12.1.0

* remove tb-nightly, remove dup compile megatron util

* uninstall tb-nightly reinstall tensorboard
2023-12-28 17:47:58 +08:00
mulin.lyh
fe8bfa9219 when build force install funasr pai-eacv etc
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/14812168
* when build force install funasr pai-eacv etc
2023-11-29 17:40:09 +08:00