Files
modelscope/modelscope/hub/__init__.py
Xingjun.Wang 50f8d37bc9 [Feat & Refactor] Refactor hub and CLI modules (#1732)
* refactor(hub): shim layer delegating to modelscope-hub

- Replace hub/api.py (4674→250 lines) with shim inheriting LegacyHubApi
- Replace hub/snapshot_download.py, callback.py with thin shims
- Partial shim hub/file_download.py (retain http_get_file)
- Shim hub/constants.py and errors.py with legacy aliases
- Shim hub/git.py, repository.py, cache_manager.py, upload_*.py
- Migrate CLI entry to modelscope_hub.cli.main:run_cmd
- Adapt 6 CLI commands as modelscope_hub.cli_plugins
- Delete redundant CLI files (download/upload/login/create/etc)
- Add modelscope-hub>=0.2.0 dependency, Python>=3.10
- Add __getattr__ proxy for forward-compatible method access
- Propagate timeout/max_retries to internal LegacyClient
- Bridge MODELSCOPE_CREDENTIALS_PATH env var to HubConfig

* fix lint: isort/yapf formatting + exclude hub/api.py from hooks

* set modelscope-hub>=0.0.5

* remove unused code

* refactor(hub): standardize token naming — git_token vs token

Disambiguate git token and SDK/API token naming across the hub layer:
- ModelScopeConfig: get_token/save_token → get_git_token/save_git_token
  (old names kept as deprecated aliases with DeprecationWarning)
- GitCommandWrapper: rename token params to git_token in clone/push/config
- Repository/DatasetRepository: auth_token → git_token (deprecated compat kept)
- data_loader.py: update caller to use get_git_token()

SDK token references (HubApi(token=...), get_cookies(access_token=...),
commit_scheduler.token) remain unchanged as they correctly use `token` naming.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* remove(msdatasets): remove all Virgo-related implementation

Remove the entire Virgo dataset subsystem which is no longer needed:
- Remove VirgoDataset class and VirgoDownloader
- Remove VirgoAuthConfig and VirgoDatasetConfig
- Remove Hubs.virgo enum value
- Remove fetch_virgo_meta from DataMetaManager
- Remove download_virgo_files from DatasetContextConfig
- Remove test_virgo_dataset.py test file
- Clean up unused imports (pandas, MaxComputeUtil, valid_url, etc.)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(hub): add OSS dataset operations and meta-file download to HubApi

Add methods that msdatasets depends on but don't belong in modelscope_hub:
- _legacy_request: internal helper combining legacy HTTP transport with
  application-level envelope validation (Code/Data/Message)
- list_oss_dataset_objects: list OSS storage objects for a dataset
- delete_oss_dataset_object / delete_oss_dataset_dir: delete OSS objects
- fetch_meta_files_from_url: download and cache meta CSV/JSONL files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix imports issue

* fix: address PR review feedback

- cli/plugins.py: change --yes and --all flags to action='store_true'
- hub/git.py: replace os.linesep with .splitlines() for cross-platform safety
- hub/__init__.py: use is_file() with fallback for robust credentials path detection

* fix lint

* update ms hub version

* fix(ci): add PyPI official as fallback index for pip

Aliyun mirror may lag behind PyPI for newly published packages,
causing dependency resolution failures (e.g. modelscope-hub>=0.0.6).
Add pypi.org/simple as extra-index-url so new versions are immediately
available while keeping the Aliyun mirror as the primary source.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix UTs

* remove unused UTs

* fix ut

* update modelscope-hub installation for source code

* fix UT

* fix uts

* fix ut

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-09 20:00:20 +08:00

50 lines
1.9 KiB
Python

"""modelscope.hub — shim layer delegating to modelscope_hub."""
import os as _os
from pathlib import Path as _Path
from modelscope_hub import HubConfig as _HubConfig
from modelscope_hub import get_default_config as _get_default_config
from modelscope_hub import set_default_config as _set_default_config
from .callback import ProgressCallback, TqdmCallback
from .commit_scheduler import CommitScheduler
from .snapshot_download import snapshot_download
def _sync_config() -> None:
"""Bridge legacy env vars that modelscope_hub does not natively recognize."""
# MODELSCOPE_CACHE is already handled by HubConfig; only sync
# if a non-standard alias is set.
legacy_cache = _os.environ.get('MS_CACHE_HOME')
if legacy_cache and not _os.environ.get('MODELSCOPE_CACHE'):
_set_default_config(_HubConfig(cache_dir=legacy_cache))
# Bridge MODELSCOPE_CREDENTIALS_PATH → HubConfig.config_dir so credential
# lookup honours the legacy override.
creds_path = _os.environ.get('MODELSCOPE_CREDENTIALS_PATH')
if creds_path:
resolved = _Path(creds_path).expanduser().resolve()
# Legacy convention may point at either the credentials directory
# (e.g. ``~/.modelscope``) or at a credentials file inside it; the
# new HubConfig always expects the directory. Treat the path as a
# file when it exists as one, falling back to the legacy
# ``credentials`` filename heuristic for paths that do not yet
# exist on disk.
is_file = resolved.is_file() or (not resolved.exists()
and resolved.name == 'credentials')
config_dir = resolved.parent if is_file else resolved
cfg = _get_default_config()
if cfg.config_dir != config_dir:
cfg.config_dir = config_dir
_sync_config()
__all__ = [
'CommitScheduler',
'ProgressCallback',
'TqdmCallback',
'snapshot_download',
]