Commit Graph

28 Commits

Author SHA1 Message Date
Xingjun.Wang
680c3638e0 [Feat & Refactor] Refactor hub and CLI modules (#1732)
* refactor(hub): shim layer delegating to modelscope-hub

- Replace hub/api.py (4674→250 lines) with shim inheriting LegacyHubApi
- Replace hub/snapshot_download.py, callback.py with thin shims
- Partial shim hub/file_download.py (retain http_get_file)
- Shim hub/constants.py and errors.py with legacy aliases
- Shim hub/git.py, repository.py, cache_manager.py, upload_*.py
- Migrate CLI entry to modelscope_hub.cli.main:run_cmd
- Adapt 6 CLI commands as modelscope_hub.cli_plugins
- Delete redundant CLI files (download/upload/login/create/etc)
- Add modelscope-hub>=0.2.0 dependency, Python>=3.10
- Add __getattr__ proxy for forward-compatible method access
- Propagate timeout/max_retries to internal LegacyClient
- Bridge MODELSCOPE_CREDENTIALS_PATH env var to HubConfig

* fix lint: isort/yapf formatting + exclude hub/api.py from hooks

* set modelscope-hub>=0.0.5

* remove unused code

* refactor(hub): standardize token naming — git_token vs token

Disambiguate git token and SDK/API token naming across the hub layer:
- ModelScopeConfig: get_token/save_token → get_git_token/save_git_token
  (old names kept as deprecated aliases with DeprecationWarning)
- GitCommandWrapper: rename token params to git_token in clone/push/config
- Repository/DatasetRepository: auth_token → git_token (deprecated compat kept)
- data_loader.py: update caller to use get_git_token()

SDK token references (HubApi(token=...), get_cookies(access_token=...),
commit_scheduler.token) remain unchanged as they correctly use `token` naming.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* remove(msdatasets): remove all Virgo-related implementation

Remove the entire Virgo dataset subsystem which is no longer needed:
- Remove VirgoDataset class and VirgoDownloader
- Remove VirgoAuthConfig and VirgoDatasetConfig
- Remove Hubs.virgo enum value
- Remove fetch_virgo_meta from DataMetaManager
- Remove download_virgo_files from DatasetContextConfig
- Remove test_virgo_dataset.py test file
- Clean up unused imports (pandas, MaxComputeUtil, valid_url, etc.)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(hub): add OSS dataset operations and meta-file download to HubApi

Add methods that msdatasets depends on but don't belong in modelscope_hub:
- _legacy_request: internal helper combining legacy HTTP transport with
  application-level envelope validation (Code/Data/Message)
- list_oss_dataset_objects: list OSS storage objects for a dataset
- delete_oss_dataset_object / delete_oss_dataset_dir: delete OSS objects
- fetch_meta_files_from_url: download and cache meta CSV/JSONL files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix imports issue

* fix: address PR review feedback

- cli/plugins.py: change --yes and --all flags to action='store_true'
- hub/git.py: replace os.linesep with .splitlines() for cross-platform safety
- hub/__init__.py: use is_file() with fallback for robust credentials path detection

* fix lint

* update ms hub version

* fix(ci): add PyPI official as fallback index for pip

Aliyun mirror may lag behind PyPI for newly published packages,
causing dependency resolution failures (e.g. modelscope-hub>=0.0.6).
Add pypi.org/simple as extra-index-url so new versions are immediately
available while keeping the Aliyun mirror as the primary source.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix UTs

* remove unused UTs

* fix ut

* update modelscope-hub installation for source code

* fix UT

* fix uts

* fix ut

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-09 22:23:14 +08:00
tastelikefeet
918a808dc2 compat with tf5.0 (#1618) 2026-03-07 22:40:43 +08:00
Xingjun.Wang
7d11b77112 Fix trust_remote_code (#1462)
1. Set `trust_remote_code` to `True` by default in datasets module
2. Set `trust_remote_code` to `True` by default in PolyLM pipeline
2025-08-14 10:56:16 +08:00
Xingjun.Wang
055496c597 Fix CI 2025-08-07 19:26:32 +08:00
suluyana
299ece8b8a support ms-swift 3.0.0 (#1166) 2025-01-02 17:20:32 +08:00
suluyana
60780769b1 Fix/daily (#1155)
* fix(llm ppl): 1. cache position; 2. stream_gready_search; 3. swift_mapping

* fix punkt

---------

Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>
2024-12-23 09:55:12 +08:00
suluyana
1fe211ffe5 fix pipeline builder when model is not supported (#1125)
* fix pipeline builder when model is not supported

* fix ci & skip
---------

Co-authored-by: suluyan.sly@alibaba-inc.com <suluyan.sly@alibaba-inc.com>
2024-12-12 19:24:38 +08:00
mulin.lyh
247ae5f637 default aliyuncs pip
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13320919
* default aliyuncs pip

* pip to aliyun source
2023-07-18 19:59:23 +08:00
mulin.lyh
bb9163f71a remove hard code tsinghua
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13314592
* remove hard code tsinghua
2023-07-18 14:51:18 +08:00
mulin.lyh
fa7562fd96 [to #50384694]fix: ci hang when build torch extension
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12976382
2023-06-19 17:56:04 +08:00
mulin.lyh
7378f94455 [to #47749867] modify audio install command 2023-02-10 12:04:08 +00:00
mulin.lyh
e54694690f [to #46993990]feat: run ci cases base on code diff to reduct ci test time 2023-02-06 08:00:19 +00:00
pengzhendong
b0cf09d7b0 [ci] chang pypi url to tsinghua 2022-11-24 22:12:58 +08:00
wenmeng.zwm
584fa8f586 merge internal master 2022-11-18 14:21:05 +08:00
mulin.lyh
90a5efa1c2 [to #46106568]feat: parallel run ci case 2022-11-17 08:51:23 +08:00
wenmeng.zwm
9f2de967c5 merge master and fix conflict 2022-11-16 18:10:39 +08:00
mulin.lyh
c6e575458a [to #45878116]feat: run linter test first
提高ci效率,先跑linter,linter错误先报漏出来,不用合并release
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10596907

    * [to #45878116]feat: run linter test first
2022-11-11 16:19:14 +08:00
wenmeng.zwm
c638acbeda fix merge conflict 2022-10-26 17:12:25 +08:00
mulin.lyh
384377b8f5 * [to #45486649]feat: modelscope model version use model repo tag, unsupport branch or commit it, client user-agent header unified 2022-10-26 13:55:51 +08:00
wenmeng.zwm
ca7189e77a add dummy git config 2022-10-10 17:38:03 +08:00
wenmeng.zwm
e7b37b09d4 do linter test only in alibaba internal env 2022-10-10 12:03:05 +08:00
mulin.lyh
83dbf71302 [to #44702084]fix: ci pip install domain in single commands, find with requirement install failed is complicated.
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10019738

    * [to #44702084]fix: ci pip install domain in single commands, find with requirement install failed is complicated.
2022-09-05 17:38:05 +08:00
mulin.lyh
042cff7d68 [to #44702084]fix: ci pip install domain in single commands, find with requirement install failed is complicated.
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10014958

    * [to #44702084]fix: ci pip install domain in single commands, find with requirement install failed is complicated.
2022-09-05 16:08:50 +08:00
mulin.lyh
9e14d6727b [to #44571845]fix: ci support multiple image
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9974293
2022-09-02 11:02:43 +08:00
mulin.lyh
12698b31a0 [to #44340132] fix: ci case run out of gpu memory 2022-08-30 17:59:15 +08:00
mulin.lyh
01bb751425 [to #43653669]feat: auto build docker images
auto build docker images
aone 任务: https://test.aone.alibaba-inc.com/jobs/1824567?buildId=143470479
修改任务分支
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9566518

    * [to #43653669]feat: auto build docker images
2022-08-22 16:01:42 +08:00
wenmeng.zwm
c539811deb [to #43115513] remove repeated cv task
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9617409
2022-08-03 11:49:12 +08:00
mulin.lyh
69047b99ae [to #43387011]feat: ci test to new host and running in docker
ci 测试迁移新的机器,并且在容器中运行,减小互相干扰的可能
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9427096

    * add docker ci script
2022-07-20 11:49:05 +08:00