Reuse flat/hub legacy caches in download paths. For docker builds, pass
PIP_EXTRA_INDEX_URL via ARG and RUN export so PyPI is available during
install without persisting it as an image ENV default.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(msdatasets): make HfFileSystem monkey-patch idempotent for repeated streaming loads
When streaming=True the patches on HfFileSystem._open and HfFileSystem.__init__
are intentionally kept alive after load_dataset_with_ctx exits. A second call to
load_dataset_with_ctx then snaps the already-patched wrappers as 'originals',
causing _hf_fs_init_with_cookie / _hf_fs_open to call themselves recursively
until RecursionError: maximum recursion depth exceeded.
Fix: detect whether HfFileSystem is already patched before overwriting
_hf_fs_{open,init}_original, and skip re-applying the patch when it is already
in place. The finally-block restores only patches that were applied in the
current invocation, leaving pre-existing patches intact.
Adds: test_hf_filesystem_patch_idempotent_for_repeated_streaming_loads
* fix(msdatasets): restore patches when streaming load fails; hermetic tests
Addresses both Gemini Code Assist review comments on PR #1754:
1. Bug fix – patch leak on failed streaming loads:
Replace guard with flag.
The flag is set True only after load_dataset() succeeds with streaming=True.
Any exception in load_dataset (network, auth, invalid dataset, etc.) leaves
_streaming_dataset_returned=False, so the finally block always restores all
monkey-patches, preventing permanent global-state corruption.
2. Test improvement – hermetic test environment:
Add _reset_hf_filesystem_patch() helper that strips pre-existing patches
before each unit test, ensuring tests are independent of execution order.
Add test_hf_filesystem_patch_restored_when_streaming_load_fails to cover
the bug scenario introduced in fix 1.
* update requirements for hub
* feat(hub): restore progress_callbacks on snapshot_download shim (issue #1757)
The progress_callbacks parameter was dropped from snapshot_download after v1.38, breaking legacy code and GUI progress reporting. Re-expose it on modelscope.hub.snapshot_download (positioned before token per the v1.34 signature) and forward it to modelscope_hub.compat. Requires modelscope-hub >= 0.1.8.
Add network-free forwarding tests that patch the compat delegate so the real shim executes.
* fix lint
* feat(hub): add legacy-cache capability guard to snapshot_download shim
Warn once (thread-safe) when the installed modelscope-hub lacks pre-1.38 legacy cache auto-detection (DownloadManager._find_legacy_repo_dir, added in modelscope-hub>=0.1.7), so programmatic snapshot_download/dataset_snapshot_download callers are not silently downloading into the new layout while an old cache exists. Capability is probed, not reimplemented (kept single-sourced in modelscope-hub). Adds network-free tests for present/absent capability and fire-once behavior.
* fix: resolve cross-repo auto_map when cache paths contain --
modelscope_hub 0.1.x layout embeds -- in local dirs; rejoining that
path into class_reference broke transformers' split("--"). Pass the
local snapshot as pretrained_model_name_or_path with a bare class name
instead, and stop mutating the args tuple in place.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: reject empty model name in trusted owner cache paths
check_model_from_owner_group treated paths like iic--/snapshots/v1 as
trusted because split('--') still yields two parts. Require both owner
and name segments to be non-empty.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: cover remote pretrained args tuple rebuild in dynamic module patcher
The TypeError from mutating *args was already fixed with the auto_map
double-dash change; add an explicit regression test for that branch.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: derive valid module names from snapshot revision paths
Path(model_dir).stem on revisions like v1.0.4 produced v1.0, so
importlib treated it as package v1 and raised ModuleNotFoundError.
Use the owner--name (+ revision) segment and sanitize to an identifier.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: cover module naming for dotted snapshot revisions
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: handle pretrained_model_name_or_path passed via kwargs
When the arg is only in kwargs, downloading and then forcing a
positional overwrite caused a duplicate-keyword TypeError. Resolve and
update kwargs or args consistently for both the download and cross-repo
branches.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix lint
* fix(docker): catch JSONDecodeError when querying Docker Hub tags
Non-JSON Hub responses (block pages, 5xx HTML) would bypass URLError
handling and crash json.load; surface them as RuntimeError instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: suluyan <suluyan.sly@aliabab-inc.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Temporarily pin for the 1.39.0 release: TE 2.17 has no prebuilt
transformer_engine_torch wheels and fails to compile from source.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add image_type=amd that auto-resolves the newest concrete Docker Hub
tag, probes rocm/python/torch without GPU, installs modelscope from the
build branch, and tags as ubuntu*-rocm*-py*-torch*-{version}.
Co-authored-by: Cursor <cursoragent@cursor.com>
Keep ipywidgets, jupyter_core, nbconvert, and nbclient inside the
INSTALL_MS_DEPS gate so latest and old image builds are unaffected.
Co-authored-by: Cursor <cursoragent@cursor.com>
Install ipywidgets, ipykernel, jupyter_core, nbconvert, and nbclient
outside the INSTALL_MS_DEPS gate so latest GPU images also support
python -m jupyter nbconvert for agent notebook debugging.
Co-authored-by: Cursor <cursoragent@cursor.com>
Align LLM image pip config with GPU images by dropping the persistent
pypi.org extra-index-url that was added for modelscope-hub fallback.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(hub): shim layer delegating to modelscope-hub
- Replace hub/api.py (4674→250 lines) with shim inheriting LegacyHubApi
- Replace hub/snapshot_download.py, callback.py with thin shims
- Partial shim hub/file_download.py (retain http_get_file)
- Shim hub/constants.py and errors.py with legacy aliases
- Shim hub/git.py, repository.py, cache_manager.py, upload_*.py
- Migrate CLI entry to modelscope_hub.cli.main:run_cmd
- Adapt 6 CLI commands as modelscope_hub.cli_plugins
- Delete redundant CLI files (download/upload/login/create/etc)
- Add modelscope-hub>=0.2.0 dependency, Python>=3.10
- Add __getattr__ proxy for forward-compatible method access
- Propagate timeout/max_retries to internal LegacyClient
- Bridge MODELSCOPE_CREDENTIALS_PATH env var to HubConfig
* fix lint: isort/yapf formatting + exclude hub/api.py from hooks
* set modelscope-hub>=0.0.5
* remove unused code
* refactor(hub): standardize token naming — git_token vs token
Disambiguate git token and SDK/API token naming across the hub layer:
- ModelScopeConfig: get_token/save_token → get_git_token/save_git_token
(old names kept as deprecated aliases with DeprecationWarning)
- GitCommandWrapper: rename token params to git_token in clone/push/config
- Repository/DatasetRepository: auth_token → git_token (deprecated compat kept)
- data_loader.py: update caller to use get_git_token()
SDK token references (HubApi(token=...), get_cookies(access_token=...),
commit_scheduler.token) remain unchanged as they correctly use `token` naming.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* remove(msdatasets): remove all Virgo-related implementation
Remove the entire Virgo dataset subsystem which is no longer needed:
- Remove VirgoDataset class and VirgoDownloader
- Remove VirgoAuthConfig and VirgoDatasetConfig
- Remove Hubs.virgo enum value
- Remove fetch_virgo_meta from DataMetaManager
- Remove download_virgo_files from DatasetContextConfig
- Remove test_virgo_dataset.py test file
- Clean up unused imports (pandas, MaxComputeUtil, valid_url, etc.)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(hub): add OSS dataset operations and meta-file download to HubApi
Add methods that msdatasets depends on but don't belong in modelscope_hub:
- _legacy_request: internal helper combining legacy HTTP transport with
application-level envelope validation (Code/Data/Message)
- list_oss_dataset_objects: list OSS storage objects for a dataset
- delete_oss_dataset_object / delete_oss_dataset_dir: delete OSS objects
- fetch_meta_files_from_url: download and cache meta CSV/JSONL files
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix imports issue
* fix: address PR review feedback
- cli/plugins.py: change --yes and --all flags to action='store_true'
- hub/git.py: replace os.linesep with .splitlines() for cross-platform safety
- hub/__init__.py: use is_file() with fallback for robust credentials path detection
* fix lint
* update ms hub version
* fix(ci): add PyPI official as fallback index for pip
Aliyun mirror may lag behind PyPI for newly published packages,
causing dependency resolution failures (e.g. modelscope-hub>=0.0.6).
Add pypi.org/simple as extra-index-url so new versions are immediately
available while keeping the Aliyun mirror as the primary source.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix UTs
* remove unused UTs
* fix ut
* update modelscope-hub installation for source code
* fix UT
* fix uts
* fix ut
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* 🐛 fix(clearcache): fix single model cache path
- Add 'models' subdirectory under 'hub' for single model cache paths
- Align dataset and model path structures for consistency
* 🐛 fix(clearcache): use cache root helpers to handle MODELSCOPE_CACHE correctly
Unconditionally prepending `hub/` broke `$MODELSCOPE_CACHE`-set users,
since the cache root has no `hub/` segment in that case. Reuse
get_model_cache_root() / get_dataset_cache_root() from file_utils,
which already handle both default and env-set layouts. Add tests
covering both scenarios.
---------
Co-authored-by: Yunnglin <mao.looper@qq.com>
