Commit Graph

1216 Commits

Author SHA1 Message Date
Xingjun.Wang
5db3b88e2a add property session in HubApi (#1748) 2026-07-03 10:38:49 +08:00
tastelikefeet
c7dba8de1e Fix some security issues (#1737) 2026-06-15 10:35:50 +08:00
Xingjun.Wang
50f8d37bc9 [Feat & Refactor] Refactor hub and CLI modules (#1732)
* refactor(hub): shim layer delegating to modelscope-hub

- Replace hub/api.py (4674→250 lines) with shim inheriting LegacyHubApi
- Replace hub/snapshot_download.py, callback.py with thin shims
- Partial shim hub/file_download.py (retain http_get_file)
- Shim hub/constants.py and errors.py with legacy aliases
- Shim hub/git.py, repository.py, cache_manager.py, upload_*.py
- Migrate CLI entry to modelscope_hub.cli.main:run_cmd
- Adapt 6 CLI commands as modelscope_hub.cli_plugins
- Delete redundant CLI files (download/upload/login/create/etc)
- Add modelscope-hub>=0.2.0 dependency, Python>=3.10
- Add __getattr__ proxy for forward-compatible method access
- Propagate timeout/max_retries to internal LegacyClient
- Bridge MODELSCOPE_CREDENTIALS_PATH env var to HubConfig

* fix lint: isort/yapf formatting + exclude hub/api.py from hooks

* set modelscope-hub>=0.0.5

* remove unused code

* refactor(hub): standardize token naming — git_token vs token

Disambiguate git token and SDK/API token naming across the hub layer:
- ModelScopeConfig: get_token/save_token → get_git_token/save_git_token
  (old names kept as deprecated aliases with DeprecationWarning)
- GitCommandWrapper: rename token params to git_token in clone/push/config
- Repository/DatasetRepository: auth_token → git_token (deprecated compat kept)
- data_loader.py: update caller to use get_git_token()

SDK token references (HubApi(token=...), get_cookies(access_token=...),
commit_scheduler.token) remain unchanged as they correctly use `token` naming.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* remove(msdatasets): remove all Virgo-related implementation

Remove the entire Virgo dataset subsystem which is no longer needed:
- Remove VirgoDataset class and VirgoDownloader
- Remove VirgoAuthConfig and VirgoDatasetConfig
- Remove Hubs.virgo enum value
- Remove fetch_virgo_meta from DataMetaManager
- Remove download_virgo_files from DatasetContextConfig
- Remove test_virgo_dataset.py test file
- Clean up unused imports (pandas, MaxComputeUtil, valid_url, etc.)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(hub): add OSS dataset operations and meta-file download to HubApi

Add methods that msdatasets depends on but don't belong in modelscope_hub:
- _legacy_request: internal helper combining legacy HTTP transport with
  application-level envelope validation (Code/Data/Message)
- list_oss_dataset_objects: list OSS storage objects for a dataset
- delete_oss_dataset_object / delete_oss_dataset_dir: delete OSS objects
- fetch_meta_files_from_url: download and cache meta CSV/JSONL files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix imports issue

* fix: address PR review feedback

- cli/plugins.py: change --yes and --all flags to action='store_true'
- hub/git.py: replace os.linesep with .splitlines() for cross-platform safety
- hub/__init__.py: use is_file() with fallback for robust credentials path detection

* fix lint

* update ms hub version

* fix(ci): add PyPI official as fallback index for pip

Aliyun mirror may lag behind PyPI for newly published packages,
causing dependency resolution failures (e.g. modelscope-hub>=0.0.6).
Add pypi.org/simple as extra-index-url so new versions are immediately
available while keeping the Aliyun mirror as the primary source.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix UTs

* remove unused UTs

* fix ut

* update modelscope-hub installation for source code

* fix UT

* fix uts

* fix ut

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-09 20:00:20 +08:00
Xingjun.Wang
1804ac6bf8 [Feature] Add studio module (#1727) 2026-06-05 14:33:25 +08:00
AAAkater
2719754beb 🐛 fix(clearcache): fix single model cache path (#1724)
* 🐛 fix(clearcache): fix single model cache path

- Add 'models' subdirectory under 'hub' for single model cache paths
- Align dataset and model path structures for consistency

* 🐛 fix(clearcache): use cache root helpers to handle MODELSCOPE_CACHE correctly

Unconditionally prepending `hub/` broke `$MODELSCOPE_CACHE`-set users,
since the cache root has no `hub/` segment in that case. Reuse
get_model_cache_root() / get_dataset_cache_root() from file_utils,
which already handle both default and env-set layouts. Add tests
covering both scenarios.

---------

Co-authored-by: Yunnglin <mao.looper@qq.com>
2026-06-03 18:39:39 +08:00
tastelikefeet
8bab4b9bc1 Fix bug: 1. patch_context cannot revert classmethod 2. kernels testcase miss trust_remote_code and revision (#1729) 2026-05-31 16:11:21 +08:00
tastelikefeet
b3808b7c20 Support kernels downloading (#1697) 2026-04-28 10:29:19 +08:00
Xingjun.Wang
1d81c4612a [Feature] Add openapi auth support (#1686) 2026-04-22 17:39:18 +08:00
Jintao
6059df3635 [bugfix] compat swift 4.0 (#1680) 2026-04-16 15:30:02 +08:00
Xingjun.Wang
fc0e2159a3 [Fix] Fix preview csv stream loading (#1665) 2026-04-14 14:17:23 +08:00
Xingjun.Wang
32d7c7062c [Fix] fix token auth for downloading (#1674) 2026-04-11 14:07:46 +08:00
Xingjun.Wang
f4dbe65110 [Fix] fix dataset util (#1645) 2026-03-13 10:36:11 +08:00
tastelikefeet
918a808dc2 compat with tf5.0 (#1618) 2026-03-07 22:40:43 +08:00
tastelikefeet
dbbf268e70 Fix downloading repos in automap (#1630) 2026-03-02 11:33:50 +08:00
tastelikefeet
6dd24e3e3c lint code (#1619) 2026-02-19 17:58:09 +08:00
tastelikefeet
c1d91a4454 fix downloading txt files (#1610) 2026-02-19 17:18:40 +08:00
Xingjun.Wang
f9dd6794c4 Fix #1316 (#1612) 2026-02-11 13:57:52 +08:00
Xingjun.Wang
4f867bfe80 Fix: deprecate delete_repo, delete_model and delete_dataset due to token a… (#1588) 2026-01-12 11:51:46 +08:00
Gongsheng Li
a31c44cb5a Feat/update default revision (#1553) 2025-12-05 15:24:42 +08:00
Yunlin Mao
dfb370b77d fix daily regression (#1545)
* update daily regression

* update test
2025-11-07 17:45:59 +08:00
xiaojianping.xjp
23f3b9a43e update ollama: granite4,deepseek-v3.1,gpt-oss (#1531)
Co-authored-by: xiaojiaping.xjp <xiaojiaping.xjp@alibaba-inc.com>
2025-11-07 17:33:25 +08:00
Yunlin Mao
723599ac48 Update list datasets to OpenAPI (#1532)
* update list datasets

* Update modelscope/hub/constants.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update modelscope/hub/api.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update modelscope/hub/api.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update modelscope/hub/api.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* fix lint

* update workflow

* fix lint

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-11-05 17:04:29 +08:00
Gongsheng Li
3fd95b1bd7 Feat/support commit scheduler (#1480) 2025-10-21 16:41:55 +08:00
Gongsheng Li
4733d5374b add support for siblings field in repo_info (#1490) 2025-09-08 17:22:29 +08:00
Gongsheng Li
2fc6d8e822 Feat/support repo info (#1483) 2025-08-29 17:23:46 +08:00
tastelikefeet
a5039281d3 skip aigc test (#1476) 2025-08-23 11:12:35 +08:00
Xingjun.Wang
7d11b77112 Fix trust_remote_code (#1462)
1. Set `trust_remote_code` to `True` by default in datasets module
2. Set `trust_remote_code` to `True` by default in PolyLM pipeline
2025-08-14 10:56:16 +08:00
Xingjun.Wang
055496c597 Fix CI 2025-08-07 19:26:32 +08:00
Xingjun.Wang
0e49156bbe fix TestTinynasDamoyoloTrainerSingleGPU (#1449) 2025-08-06 18:50:25 +08:00
Koko-ry
1d93f7e56d Feat: Add AIGC model creation support to create_model (#1431)
1.  Add create aigc model in `create_model` func
2. Add CLI for creating aigc model
2025-08-06 17:01:42 +08:00
suluyana
53ceca4df4 feat: sentence_embedding pipeline (#1435) 2025-08-06 15:43:36 +08:00
Xingjun.Wang
ee05e12d75 Add create repo cli (#1445) 2025-08-05 21:42:41 +08:00
Koko-ry
51e28f5a49 Feat: add atomic capabilities - MCPApi (#1426) 2025-08-05 21:41:26 +08:00
tastelikefeet
d8bb003681 fix bug (#1441) 2025-08-04 22:05:49 +08:00
Yunlin Mao
ff7551c494 fix large log in scan cache test (#1439) 2025-08-04 20:58:30 +08:00
suluyana
b3d24d18e6 ollama template: qwen3-coder, gemma3n, mistral-small3.2, magistral, devstral (#1434)
Co-authored-by: suluyan <suluyan.sly@aliabab-inc.com>
2025-08-04 17:07:06 +08:00
tastelikefeet
556aa76dc3 fix pyyaml according to: https://github.com/Anchor0221/CVE-2025-50460 (#1428) 2025-08-01 22:04:44 +08:00
Koko-ry
d6f11c1c22 Skip UT TextToSpeechSambertHifigan16kPipelineTest temporarily (#1420) 2025-07-17 18:57:44 +08:00
tastelikefeet
f01864a2f9 weak file lock (#1417) 2025-07-17 18:38:24 +08:00
Jintao
29ea57dfbc [hub] Support ProgressCallback (#1380)
1. Add ProgressCallback for file download
2025-06-26 14:02:05 +08:00
Emmanuel Ferdman
7d71b45fa7 fix: migrate to unittest.assertNotEqual (#1350)
Signed-off-by: Emmanuel Ferdman <emmanuelferdman@gmail.com>
2025-06-05 14:54:45 +08:00
co63oc
8323fc5185 Fix typos in multiple files (#1357) 2025-06-05 14:04:29 +08:00
suluyana
800543d1aa fix: ollama template mapping (#1354)
* feat ollama template qwen3

* add phi4, llama4, qwen2.5vl

* fix dpsk-r1-qwen3

* rm duplicate case

---------

Co-authored-by: suluyan <suluyan.sly@aliabab-inc.com>
2025-06-04 20:13:35 +08:00
Z-yq
f9b8d4b9d2 vc ssr (#1208)
* vc ssr

* Add more patches for hf (#1160)

* clone and lint #1205 (#1209)

* 更新格式

* fix cache path (#1211)

Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>

* “update"

* fix create_commit login (#1210)

* support multiple include/exclude filter patterns in command line (#1214)

Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>

* Use legacy cache (#1215)

* fix name (#1216)

Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>

* fix path name for log accuracy (#1217)


* change log msg

---------

Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>

* fix visibility (#1222)

Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>

* Merge 1.23 hotfix to master (#1227)

* 修复格式问题

* fix 路径问题

* Update test_speech_super_resolution.py

* Update test_voice_conversion.py

---------

Co-authored-by: tastelikefeet <58414341+tastelikefeet@users.noreply.github.com>
Co-authored-by: Yingda Chen <yingdachen@apache.org>
Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
Co-authored-by: zhongyuqi <zhongyuqi@microbt.com>
2025-06-04 16:08:20 +08:00
Yunlin Mao
b034267345 [Feature] Add scan_cache_dir function (#1348)
* init cache manager

* add scan cache

* fix lint

* add scan cache unit test

* add download first for scan cache test
2025-05-28 09:59:03 +08:00
tastelikefeet
75d54927e1 Support trust_remote_code for pipeline and model (#1333) 2025-05-13 22:52:57 +08:00
tastelikefeet
a91f19ea54 Support downloading exact file for hf wrapper (#1323) 2025-04-30 14:57:59 +08:00
NononoSugar
8f3e1845d2 feat: new ollama template (#1318)
Co-authored-by: jbh02127155 <jbh02127155@alibaba-inc.com>
2025-04-24 12:43:07 +08:00
Xingjun.Wang
6adc8614c9 Add create_repo and upload UTs (#1282)
* add delete_repo, delete_dataset, exists_ok in create_repo

* add UT for create_repo
2025-04-07 10:24:26 +08:00
xingjun.wxj
10aa2c6bd8 add login for HFUtilTest 2025-03-27 22:42:48 +08:00