Commit Graph

9 Commits

Author SHA1 Message Date
Xingjun.Wang
50f8d37bc9 [Feat & Refactor] Refactor hub and CLI modules (#1732)
* refactor(hub): shim layer delegating to modelscope-hub

- Replace hub/api.py (4674→250 lines) with shim inheriting LegacyHubApi
- Replace hub/snapshot_download.py, callback.py with thin shims
- Partial shim hub/file_download.py (retain http_get_file)
- Shim hub/constants.py and errors.py with legacy aliases
- Shim hub/git.py, repository.py, cache_manager.py, upload_*.py
- Migrate CLI entry to modelscope_hub.cli.main:run_cmd
- Adapt 6 CLI commands as modelscope_hub.cli_plugins
- Delete redundant CLI files (download/upload/login/create/etc)
- Add modelscope-hub>=0.2.0 dependency, Python>=3.10
- Add __getattr__ proxy for forward-compatible method access
- Propagate timeout/max_retries to internal LegacyClient
- Bridge MODELSCOPE_CREDENTIALS_PATH env var to HubConfig

* fix lint: isort/yapf formatting + exclude hub/api.py from hooks

* set modelscope-hub>=0.0.5

* remove unused code

* refactor(hub): standardize token naming — git_token vs token

Disambiguate git token and SDK/API token naming across the hub layer:
- ModelScopeConfig: get_token/save_token → get_git_token/save_git_token
  (old names kept as deprecated aliases with DeprecationWarning)
- GitCommandWrapper: rename token params to git_token in clone/push/config
- Repository/DatasetRepository: auth_token → git_token (deprecated compat kept)
- data_loader.py: update caller to use get_git_token()

SDK token references (HubApi(token=...), get_cookies(access_token=...),
commit_scheduler.token) remain unchanged as they correctly use `token` naming.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* remove(msdatasets): remove all Virgo-related implementation

Remove the entire Virgo dataset subsystem which is no longer needed:
- Remove VirgoDataset class and VirgoDownloader
- Remove VirgoAuthConfig and VirgoDatasetConfig
- Remove Hubs.virgo enum value
- Remove fetch_virgo_meta from DataMetaManager
- Remove download_virgo_files from DatasetContextConfig
- Remove test_virgo_dataset.py test file
- Clean up unused imports (pandas, MaxComputeUtil, valid_url, etc.)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(hub): add OSS dataset operations and meta-file download to HubApi

Add methods that msdatasets depends on but don't belong in modelscope_hub:
- _legacy_request: internal helper combining legacy HTTP transport with
  application-level envelope validation (Code/Data/Message)
- list_oss_dataset_objects: list OSS storage objects for a dataset
- delete_oss_dataset_object / delete_oss_dataset_dir: delete OSS objects
- fetch_meta_files_from_url: download and cache meta CSV/JSONL files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix imports issue

* fix: address PR review feedback

- cli/plugins.py: change --yes and --all flags to action='store_true'
- hub/git.py: replace os.linesep with .splitlines() for cross-platform safety
- hub/__init__.py: use is_file() with fallback for robust credentials path detection

* fix lint

* update ms hub version

* fix(ci): add PyPI official as fallback index for pip

Aliyun mirror may lag behind PyPI for newly published packages,
causing dependency resolution failures (e.g. modelscope-hub>=0.0.6).
Add pypi.org/simple as extra-index-url so new versions are immediately
available while keeping the Aliyun mirror as the primary source.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix UTs

* remove unused UTs

* fix ut

* update modelscope-hub installation for source code

* fix UT

* fix uts

* fix ut

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-09 20:00:20 +08:00
Xingjun.Wang
d4301d192c [Fix | Refactor] Fix preview loading and refactor hf datasets utils (#1654) 2026-03-27 15:28:57 +08:00
tastelikefeet
834db59952 template and ollama in modelscope (#995) 2024-09-29 19:13:50 +08:00
jiangnana.jnn
6f9d59f469 fix jsonplus, support ndarray
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/12056076

* fix jsonplus

* add unittest

* fix json dump np float
2023-03-21 18:10:10 +08:00
zhangzhicheng.zzc
42898badf7 [to #42322933] update ast_index logic 2023-01-11 10:43:56 +08:00
mulin.lyh
f1ef4d8d9f [to #46219277]fix pre-commit pycqa flake8 to github
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10778739

    * [to #46219277]fix pre-commit pycqa flake8 to github
2022-11-17 21:02:14 +08:00
mulin.lyh
5f937b20cf [to #45549080]fix: fix pre_commit dependency importlib-metadata 5.0.0 compability issue
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10443005
2022-10-18 17:47:23 +08:00
mulin.lyh
69047b99ae [to #43387011]feat: ci test to new host and running in docker
ci 测试迁移新的机器,并且在容器中运行,减小互相干扰的可能
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9427096

    * add docker ci script
2022-07-20 11:49:05 +08:00
wenmeng.zwm
02492e2bff [to #41401401] init project
* add license
* add folder structure

Link: https://code.aone.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/8682838
2022-05-12 16:05:23 +08:00