20 Commits

Author SHA1 Message Date
Xingjun.Wang
680c3638e0 [Feat & Refactor] Refactor hub and CLI modules (#1732)
* refactor(hub): shim layer delegating to modelscope-hub

- Replace hub/api.py (4674→250 lines) with shim inheriting LegacyHubApi
- Replace hub/snapshot_download.py, callback.py with thin shims
- Partial shim hub/file_download.py (retain http_get_file)
- Shim hub/constants.py and errors.py with legacy aliases
- Shim hub/git.py, repository.py, cache_manager.py, upload_*.py
- Migrate CLI entry to modelscope_hub.cli.main:run_cmd
- Adapt 6 CLI commands as modelscope_hub.cli_plugins
- Delete redundant CLI files (download/upload/login/create/etc)
- Add modelscope-hub>=0.2.0 dependency, Python>=3.10
- Add __getattr__ proxy for forward-compatible method access
- Propagate timeout/max_retries to internal LegacyClient
- Bridge MODELSCOPE_CREDENTIALS_PATH env var to HubConfig

* fix lint: isort/yapf formatting + exclude hub/api.py from hooks

* set modelscope-hub>=0.0.5

* remove unused code

* refactor(hub): standardize token naming — git_token vs token

Disambiguate git token and SDK/API token naming across the hub layer:
- ModelScopeConfig: get_token/save_token → get_git_token/save_git_token
  (old names kept as deprecated aliases with DeprecationWarning)
- GitCommandWrapper: rename token params to git_token in clone/push/config
- Repository/DatasetRepository: auth_token → git_token (deprecated compat kept)
- data_loader.py: update caller to use get_git_token()

SDK token references (HubApi(token=...), get_cookies(access_token=...),
commit_scheduler.token) remain unchanged as they correctly use `token` naming.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* remove(msdatasets): remove all Virgo-related implementation

Remove the entire Virgo dataset subsystem which is no longer needed:
- Remove VirgoDataset class and VirgoDownloader
- Remove VirgoAuthConfig and VirgoDatasetConfig
- Remove Hubs.virgo enum value
- Remove fetch_virgo_meta from DataMetaManager
- Remove download_virgo_files from DatasetContextConfig
- Remove test_virgo_dataset.py test file
- Clean up unused imports (pandas, MaxComputeUtil, valid_url, etc.)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(hub): add OSS dataset operations and meta-file download to HubApi

Add methods that msdatasets depends on but don't belong in modelscope_hub:
- _legacy_request: internal helper combining legacy HTTP transport with
  application-level envelope validation (Code/Data/Message)
- list_oss_dataset_objects: list OSS storage objects for a dataset
- delete_oss_dataset_object / delete_oss_dataset_dir: delete OSS objects
- fetch_meta_files_from_url: download and cache meta CSV/JSONL files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix imports issue

* fix: address PR review feedback

- cli/plugins.py: change --yes and --all flags to action='store_true'
- hub/git.py: replace os.linesep with .splitlines() for cross-platform safety
- hub/__init__.py: use is_file() with fallback for robust credentials path detection

* fix lint

* update ms hub version

* fix(ci): add PyPI official as fallback index for pip

Aliyun mirror may lag behind PyPI for newly published packages,
causing dependency resolution failures (e.g. modelscope-hub>=0.0.6).
Add pypi.org/simple as extra-index-url so new versions are immediately
available while keeping the Aliyun mirror as the primary source.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix UTs

* remove unused UTs

* fix ut

* update modelscope-hub installation for source code

* fix UT

* fix uts

* fix ut

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-09 22:23:14 +08:00
AAAkater
fbc3be49b2 🐛 fix(clearcache): fix single model cache path (#1724)
* 🐛 fix(clearcache): fix single model cache path

- Add 'models' subdirectory under 'hub' for single model cache paths
- Align dataset and model path structures for consistency

* 🐛 fix(clearcache): use cache root helpers to handle MODELSCOPE_CACHE correctly

Unconditionally prepending `hub/` broke `$MODELSCOPE_CACHE`-set users,
since the cache root has no `hub/` segment in that case. Reuse
get_model_cache_root() / get_dataset_cache_root() from file_utils,
which already handle both default and env-set layouts. Add tests
covering both scenarios.

---------

Co-authored-by: Yunnglin <mao.looper@qq.com>
2026-06-09 20:00:51 +08:00
Xingjun.Wang
32d7c7062c [Fix] fix token auth for downloading (#1674) 2026-04-11 14:07:46 +08:00
tastelikefeet
918a808dc2 compat with tf5.0 (#1618) 2026-03-07 22:40:43 +08:00
Xingjun.Wang
4f867bfe80 Fix: deprecate delete_repo, delete_model and delete_dataset due to token a… (#1588) 2026-01-12 11:51:46 +08:00
Xingjun.Wang
ee05e12d75 Add create repo cli (#1445) 2025-08-05 21:42:41 +08:00
Emmanuel Ferdman
7d71b45fa7 fix: migrate to unittest.assertNotEqual (#1350)
Signed-off-by: Emmanuel Ferdman <emmanuelferdman@gmail.com>
2025-06-05 14:54:45 +08:00
Yunlin Mao
b034267345 [Feature] Add scan_cache_dir function (#1348)
* init cache manager

* add scan cache

* fix lint

* add scan cache unit test

* add download first for scan cache test
2025-05-28 09:59:03 +08:00
Yingda Chen
e32a9361e3 fix accuracy case sensitiveness (#1118)
Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
2024-12-03 21:25:18 +08:00
Yingda Chen
63febc58a6 add llamafile support to command line (#1087)
* add llamafile support to command line


Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
2024-11-20 09:24:58 +08:00
liuyhwangyh
96dfccbd75 fix compatible of origin command line (#944)
Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>
2024-08-07 18:31:45 +08:00
liuyhwangyh
bf7c717e17 fix file download off by one bug (#943)
* fix file download off by one bug, command line default download model, add repo-type parameter
---------

Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>
2024-08-07 10:40:25 +08:00
liuyhwangyh
f93a184d88 add donwload command line and local_dir parameter (#866)
* add donwload command line and local_dir parameter

Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>
2024-05-25 14:21:55 +08:00
liuyhwangyh
672c32e7bd fix ci compatible issues,fix llmpipeline lazy import issue (#725)
* fix ci issue

* fix case issue

* modify lint to python3.10

* fix case issue

---------

Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>
2024-01-17 22:19:05 +08:00
mulin.lyh
eb0f0216c6 fix torch 2.x compatible issue
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13045011

* fix torch 2.x compatible issue

* fix torch 2.x compatible issue

* fix complex-valued input tensor matching the output from stft with return_complex=True.

* skip plugin test temporarily for modify torch version

* fix test_speech_signal_process.py compatible issue

* fix lint issue

* upgrade funasr to 0.6.5
2023-06-27 14:40:51 +08:00
mulin.lyh
8ea82e6f6d * add lfs append 2023-04-18 20:32:11 +08:00
mulin.lyh
4bae2cf823 temporarily skip hub case 2023-04-04 11:08:19 +08:00
zhangzhicheng.zzc
8a19e9645d [to #47860410]plugin with cli tool
1. 支持 plugin方式接入外部 repo、github repo,本地repo,并进行外部插件管理
2. 支持allow_remote方式接入modelhub repo,该类型属于model 范畴不做额外插件管理
3. 支持cli 安装plugin相关

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11775456
2023-03-09 23:07:13 +08:00
liugao.lg
c28fd09d42 add more cli tools
提供独立repo库快速接入modelscope的小工具,模板创建,模型上传,一键导入、版本管理等命令

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11757517
2023-03-09 00:17:11 +08:00
wenmeng.zwm
285208912b add command line tool
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11661608

* add command line tool

* add unittest

* change absolute import to relative import for test case

* mv test_util to package
2023-02-20 21:21:54 +08:00