108 Commits

Author SHA1 Message Date
Yunlin Mao
020cad8229 [Backport] Docker fixes from release/1.38 (#1735)
* Upgrade numpy to 2.x for 1.38 Docker images

- Replace deprecated numpy aliases (np.math.ceil → math.ceil, np.Inf → np.inf)
- Upgrade Docker constraints: numpy>=2.0, cython>=3.0, remove scipy upper bound

* Add ipywidgets dependency to Docker images

* update docker

* fix

* fix

* fix

* fix cpu image

* fix(docker): force numpy>=2.0 after evalscope install

ms-opencompass pulls numpy<2.0, downgrading numpy from 2.x to 1.26.4.
Force reinstall numpy>=2.0 after pip install .[eval] to restore it.

* add metax dockerfile and its requirements for ms-swift 4.2.x (#1734)

* Update npu dockerfile (#1736)

* fix(docker): remove unrelated added docker files

---------

Co-authored-by: Jintao Huang <huangjintao.hjt@alibaba-inc.com>
Co-authored-by: dwd <dwd1044898101@gmail.com>
Co-authored-by: addsubmuldiv <zyh13227@163.com>
2026-06-26 17:10:49 +08:00
addsubmuldiv
c1d88e6ab7 Update npu dockerfile (#1736) 2026-06-15 11:07:56 +08:00
dwd
37eb369e9b add metax dockerfile and its requirements for ms-swift 4.2.x (#1734) 2026-06-15 10:04:01 +08:00
Xingjun.Wang
50f8d37bc9 [Feat & Refactor] Refactor hub and CLI modules (#1732)
* refactor(hub): shim layer delegating to modelscope-hub

- Replace hub/api.py (4674→250 lines) with shim inheriting LegacyHubApi
- Replace hub/snapshot_download.py, callback.py with thin shims
- Partial shim hub/file_download.py (retain http_get_file)
- Shim hub/constants.py and errors.py with legacy aliases
- Shim hub/git.py, repository.py, cache_manager.py, upload_*.py
- Migrate CLI entry to modelscope_hub.cli.main:run_cmd
- Adapt 6 CLI commands as modelscope_hub.cli_plugins
- Delete redundant CLI files (download/upload/login/create/etc)
- Add modelscope-hub>=0.2.0 dependency, Python>=3.10
- Add __getattr__ proxy for forward-compatible method access
- Propagate timeout/max_retries to internal LegacyClient
- Bridge MODELSCOPE_CREDENTIALS_PATH env var to HubConfig

* fix lint: isort/yapf formatting + exclude hub/api.py from hooks

* set modelscope-hub>=0.0.5

* remove unused code

* refactor(hub): standardize token naming — git_token vs token

Disambiguate git token and SDK/API token naming across the hub layer:
- ModelScopeConfig: get_token/save_token → get_git_token/save_git_token
  (old names kept as deprecated aliases with DeprecationWarning)
- GitCommandWrapper: rename token params to git_token in clone/push/config
- Repository/DatasetRepository: auth_token → git_token (deprecated compat kept)
- data_loader.py: update caller to use get_git_token()

SDK token references (HubApi(token=...), get_cookies(access_token=...),
commit_scheduler.token) remain unchanged as they correctly use `token` naming.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* remove(msdatasets): remove all Virgo-related implementation

Remove the entire Virgo dataset subsystem which is no longer needed:
- Remove VirgoDataset class and VirgoDownloader
- Remove VirgoAuthConfig and VirgoDatasetConfig
- Remove Hubs.virgo enum value
- Remove fetch_virgo_meta from DataMetaManager
- Remove download_virgo_files from DatasetContextConfig
- Remove test_virgo_dataset.py test file
- Clean up unused imports (pandas, MaxComputeUtil, valid_url, etc.)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(hub): add OSS dataset operations and meta-file download to HubApi

Add methods that msdatasets depends on but don't belong in modelscope_hub:
- _legacy_request: internal helper combining legacy HTTP transport with
  application-level envelope validation (Code/Data/Message)
- list_oss_dataset_objects: list OSS storage objects for a dataset
- delete_oss_dataset_object / delete_oss_dataset_dir: delete OSS objects
- fetch_meta_files_from_url: download and cache meta CSV/JSONL files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix imports issue

* fix: address PR review feedback

- cli/plugins.py: change --yes and --all flags to action='store_true'
- hub/git.py: replace os.linesep with .splitlines() for cross-platform safety
- hub/__init__.py: use is_file() with fallback for robust credentials path detection

* fix lint

* update ms hub version

* fix(ci): add PyPI official as fallback index for pip

Aliyun mirror may lag behind PyPI for newly published packages,
causing dependency resolution failures (e.g. modelscope-hub>=0.0.6).
Add pypi.org/simple as extra-index-url so new versions are immediately
available while keeping the Aliyun mirror as the primary source.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix UTs

* remove unused UTs

* fix ut

* update modelscope-hub installation for source code

* fix UT

* fix uts

* fix ut

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-09 20:00:20 +08:00
Seas0
491e3df224 fix: quote version of timm specs in Dockerfile.ubuntu (#1728)
Co-authored-by: Seas0 <liusichen.seas0@bytedance.com>
2026-06-03 18:50:04 +08:00
xvxuopop
6f347a203c update npu dockerfile (#1716) 2026-05-14 10:36:31 +08:00
dwd
2f5f52fc3c feat(docker/Metax): add metax dockerfile and its requirements for ms-swift 4.1.x (#1689) 2026-05-06 20:57:17 +08:00
addsubmuldiv
c18f11ccbd update ascend dockerfile (#1687) 2026-04-29 16:06:57 +08:00
Xingjun.Wang
862afeea7b [Refactor] Refactor the modelscope download module (#1683) 2026-04-21 16:00:56 +08:00
addsubmuldiv
55af6ce8e4 update ascend dockerfile (#1679) 2026-04-17 17:04:26 +08:00
Xingjun.Wang
9cc43a3140 Fix GPG KEY (#1678)
* refactor hf datasets utils and fix issue

* update datasets version

* fix download cli

* add engine=python for stream loading csv

* fix dataset builder for csv engine=python

* fix datasets csv OverflowError

* update UT commit scheduler test level

* fix python gpg key
2026-04-14 15:57:33 +08:00
tastelikefeet
ce6a902d76 merge 1.35.1 to master (#1653) 2026-03-19 15:01:51 +08:00
Xingjun.Wang
832d88cb02 [Fix] lazy import oss2 (#1649)
* lazy import oss2

* fix lint
2026-03-17 16:03:42 +08:00
tastelikefeet
a420e68953 Merge branch 'release/1.34'
# Conflicts:
#	docker/build_image.py
2026-03-13 11:36:25 +08:00
vx120
467a2206e3 add metax dockerfile and its requirements for ms-swift (#1643) 2026-03-10 16:11:49 +08:00
tastelikefeet
2fedc9bb9e Feat/update docker file (#1640) 2026-03-09 14:21:38 +08:00
hjh0119
ae1e83febb remove ollama 2026-01-15 20:53:05 +08:00
hjh0119
c137ed08c2 docker zstd 2026-01-15 20:33:27 +08:00
hjh0119
de3399e8de pin optimum 2026-01-14 10:35:49 +08:00
hjh0119
e6ef4af1a1 pin optimum 2.0.0 for gpu/llm image 2026-01-13 18:18:15 +08:00
suluyana
09bc99c8c3 fix: Remove nvidia.cn from APT sources to fix apt failures in certain environments (#1575)
Co-authored-by: suluyan <suluyan.sly@aliabab-inc.com>
2026-01-09 16:09:23 +08:00
Xingjun.Wang
be6874ea8e Feat: add set_repo_visibility in hub api (#1564)
* add set_repo_visibility

* fix cr

* fix lint
2025-12-05 15:37:45 +08:00
Yunlin Mao
9cb7ce7029 Update build file and workflow (#1551)
* update build file

* update setup file

* update fairseq dep

* fix error log
2025-12-02 10:50:39 +08:00
baymax591
6f3200dfc2 [docker] Add Ascend NPU dockerfile (#1550)
Co-authored-by: ji-huazhong <hzji210@gmail.com>
Co-authored-by: vx120 <893600387@qq.com>
2025-11-26 09:56:47 +08:00
Jintao
2f3ac0fcd2 [docker] update swift docker (#1502) 2025-09-16 12:49:29 +08:00
tastelikefeet
6b70c8b577 Merge 1.29 to master (#1469) 2025-08-18 13:51:00 +08:00
tastelikefeet
169f523082 fix tensorflow cannot load gpu (#1456) 2025-08-11 11:04:17 +08:00
tastelikefeet
3bad15b779 add ms-agent into dockerfile (#1436) 2025-08-04 17:08:48 +08:00
jinghanhu
b537efebed Merge release 1.28 (#1401) 2025-08-04 17:00:14 +08:00
suluyan
7230561cbd image deps upgrade: transformers, huggingface_hub, node.js 2025-07-03 15:59:20 +08:00
suluyan
6a895360da fix boost.tar.gz not in gzip format 2025-07-03 15:59:12 +08:00
hjh0119
dee7d0576d fix pgp fail 2025-07-03 15:58:10 +08:00
Yunlin Mao
093ae87111 Update 1.26.0 docker image dependency (#1339) 2025-05-28 17:36:07 +08:00
Jintao
03dbc7bd11 Merge release1.25 to master (#1319) 2025-04-23 13:40:54 +08:00
Jintao
32aefd7428 Merge release1.24 to master (#1298) 2025-04-07 13:32:09 +08:00
Jintao
3bfff5cb53 Add swift docker (#1299) 2025-04-07 13:24:18 +08:00
tastelikefeet
e733458746 Update omegaconf (#1234) 2025-02-19 18:11:23 +08:00
tastelikefeet
7246d0b359 Merge 1.23 hotfix to master (#1227) 2025-02-17 10:30:24 +08:00
tastelikefeet
b5bb6d7bb0 Use legacy cache (#1215) 2025-02-07 17:31:32 +08:00
Xingjun.Wang
77d7262c22 fix lint (#1188) 2025-01-15 14:25:09 +08:00
Xingjun.Wang
648187626a Merge release 1.22 (#1187)
* bump version 1.22.0

* fix confict between lmdeploy & vllm

* flash-attn version

* fix version building image

* fix https://www.modelscope.cn/models/iic/nlp_structbert_address-parsing_chinese_base/feedback/issueDetail/20431 (#1170)

* fix path contatenation to be windows compatabile (#1176)

* fix path contatenation to be windows compatabile

* support dataset too

---------

Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>

* logger.warning when using remote code (#1171)

* logger warning when using remote code

Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>

* feat: all other ollama models (#1174)

* add cases

* new models

---------

Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>

* Unify datasets cache dir (#1178)

* fix cache

* fix lint

* fix dataset cache

* fix lint

* remove

* Add repo_id and repo_type in snapshot_download (#1172)

* add repo_id and repo_type in snapshot_download

* fix positional args

* update

* Fix/text gen (#1177)

* fix text-gen: read pipeline type from configuration.json first

---------

Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>

* update doc with llama_index (#1180)

* update version to 1.22.1

* merge release/1.22

---------

Co-authored-by: suluyana <suluyan_sly@163.com>
Co-authored-by: tastelikefeet <58414341+tastelikefeet@users.noreply.github.com>
Co-authored-by: Yingda Chen <yingdachen@apache.org>
Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
Co-authored-by: suluyana <110878454+suluyana@users.noreply.github.com>
Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>
Co-authored-by: Yunlin Mao <mao.looper@qq.com>
2025-01-15 14:08:52 +08:00
tastelikefeet
46c4ce9bdd support tag ci_image (#1112) 2024-12-01 15:17:25 +08:00
Jintao
b1e1f66fad Merge release 1.20 docker (#1109) 2024-11-29 13:44:36 +08:00
Jintao
e961f1671d Update docker from release/1.20 (#1077) 2024-11-11 14:56:36 +08:00
Jintao
61c2fd97e4 Update llm docker (#1076) 2024-11-08 18:09:05 +08:00
Jintao
a393b6ef24 update docker (#1075) 2024-11-08 15:49:50 +08:00
Jintao
d02d887918 update docker (#1073) 2024-11-08 10:56:39 +08:00
Jintao
125f44fb20 update docker evalscope version (#1071) 2024-11-07 18:56:17 +08:00
Jintao
a51f25abdf fix docker numpy version (#1069) 2024-11-07 13:58:08 +08:00
tastelikefeet
33308cedd4 fix numpy build error (#1068) 2024-11-06 20:40:58 +08:00