Mirrors/modelscope
1
0
Fork 0
mirror of https://github.com/modelscope/modelscope.git synced 2025-12-16 00:07:42 +01:00
Code Issues Packages Projects Releases Wiki Activity
2,837 Commits 262 Branches 106 Tags
master
Commit Graph

2837 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
tastelikefeet
f74433f6b2 Add more patches for hf (#1160) 2025-02-06 11:09:37 +08:00
Xingjun.Wang
50726e01d5 Update upload func (#1204) 
* update upload

* update

* temp

* fix pr comments

* update api.py
 2025-02-05 20:09:18 +08:00
Xingjun.Wang
d53dc6d703 remove MODELSCOPE_SHOW_INDIVIDUAL_PROGRESS_THRESHOLD (#1206) 2025-02-05 18:47:58 +08:00
suluyan
8f44f396ed Merge branch 'yh2' into feat/hf_pipeline 2025-02-02 15:00:23 +08:00
suluyan
2a927d868e Merge branch 'feat/whoami' of https://github.com/tastelikefeet/modelscope into yh2 2025-02-02 14:36:27 +08:00
Xingjun.Wang
8048f3c6b2 update upload progress (#1203) 2025-01-31 14:37:19 +08:00
yingda
d16945ffc8 test 2025-01-31 11:17:28 +08:00
yuze.zyz
eaf4ea8f21 fix comments 2025-01-27 21:46:34 +08:00
suluyan
816efec5d1 tmp for hf wrapper 2025-01-27 10:19:35 +08:00
yuze.zyz
565e45370e add tests 2025-01-26 18:50:28 +08:00
yuze.zyz
cf6aa132de remove useless file 2025-01-26 18:35:45 +08:00
tastelikefeet
d0cccf64af fix 2025-01-26 18:31:41 +08:00
yuze.zyz
22b7b25f44 lint 2025-01-26 17:04:26 +08:00
yuze.zyz
1900b57450 fix 2025-01-26 16:53:50 +08:00
yuze.zyz
4723e5c0ff fix 2025-01-26 16:26:37 +08:00
Yingda Chen
65075191f7 clean directory structure (#1201) 
默认
模型：
~/.cache/modelscope/hub/models/{model_owner}/{model_name}
数据集：
~/.cache/modelscope/hub/datasets/{dataset_owner}/{dataset_name}

配置MODELSCOPE_CACHE环境变量：
模型：
$MODELSCOPE_CACHE/models/{model_owner}/{model_name}
数据集：
$MODELSCOPE_CACHE/datasets/{dataset_owner}/{dataset_name}

Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
 2025-01-23 17:31:49 +08:00
yuze.zyz
2eda829499 fix 2025-01-22 23:06:20 +08:00
Xingjun.Wang
2c9ec4ff2d Add file size check in upload_folder (#1198) 
* add file size check in upload_folder

* update size_threshold_to_enforce_lfs
 2025-01-21 20:29:18 +08:00
suluyana
dc7e22ebeb feat: new ollama template (#1199) 
* feat: deepseek-r1, olmo2, command-r7b, deepseek-v3

---------

Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>
 2025-01-21 17:49:54 +08:00
luyan
defb668def tmp 2025-01-21 10:32:59 +08:00
yuze.zyz
d90c27524e Merge commit '58c90dee2c8c2b065baa75923ea6f1cb3964f9cb' into feat/whoami 
* commit '58c90dee2c8c2b065baa75923ea6f1cb3964f9cb': (23 commits)
  fix model directory (#1197)
  Fix typo and unused imports (#1196)
  Fix file exists of sub folder (#1192)
  Fix cookies due to firewall token expired (#1193)
  fix lint (#1188)
  Merge release 1.22 (#1187)
  add file name (#1186)
  Support upload file and folder in the hub api (#1152)
  update doc with llama_index (#1180)
  Fix/text gen (#1177)
  Add repo_id and repo_type in snapshot_download (#1172)
  Unify datasets cache dir (#1178)
  feat: all other ollama models (#1174)
  logger.warning when using remote code (#1171)
  fix path contatenation to be windows compatabile (#1176)
  fix https://www.modelscope.cn/models/iic/nlp_structbert_address-parsing_chinese_base/feedback/issueDetail/20431 (#1170)
  support ms-swift 3.0.0 (#1166)
  support latest datasets version (#1163)
  fix lint (#1168)
  fix check model (#1134)
  ...
 2025-01-20 20:33:15 +08:00
Yingda Chen
58c90dee2c fix model directory (#1197) 
Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
 2025-01-20 15:19:15 +08:00
tastelikefeet
b1404cdc2e Fix typo and unused imports (#1196) 
* fix file exists of sub folder

* fix code doc

* add ci

* lint code

* fix unused imports and typo
 2025-01-18 10:22:22 +08:00
tastelikefeet
b892164d11 Fix file exists of sub folder (#1192) 
* fix file exists of sub folder

* fix code doc

* add ci

* lint code
 2025-01-18 09:59:14 +08:00
Xingjun.Wang
8387bd8e9c Fix cookies due to firewall token expired (#1193) 
* fix cookies due to firewall token expired

* update
 2025-01-18 00:57:29 +08:00
Xingjun.Wang
77d7262c22 fix lint (#1188) 2025-01-15 14:25:09 +08:00
Xingjun.Wang
648187626a Merge release 1.22 (#1187) 
* bump version 1.22.0

* fix confict between lmdeploy & vllm

* flash-attn version

* fix version building image

* fix https://www.modelscope.cn/models/iic/nlp_structbert_address-parsing_chinese_base/feedback/issueDetail/20431 (#1170)

* fix path contatenation to be windows compatabile (#1176)

* fix path contatenation to be windows compatabile

* support dataset too

---------

Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>

* logger.warning when using remote code (#1171)

* logger warning when using remote code

Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>

* feat: all other ollama models (#1174)

* add cases

* new models

---------

Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>

* Unify datasets cache dir (#1178)

* fix cache

* fix lint

* fix dataset cache

* fix lint

* remove

* Add repo_id and repo_type in snapshot_download (#1172)

* add repo_id and repo_type in snapshot_download

* fix positional args

* update

* Fix/text gen (#1177)

* fix text-gen: read pipeline type from configuration.json first

---------

Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>

* update doc with llama_index (#1180)

* update version to 1.22.1

* merge release/1.22

---------

Co-authored-by: suluyana <suluyan_sly@163.com>
Co-authored-by: tastelikefeet <58414341+tastelikefeet@users.noreply.github.com>
Co-authored-by: Yingda Chen <yingdachen@apache.org>
Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
Co-authored-by: suluyana <110878454+suluyana@users.noreply.github.com>
Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>
Co-authored-by: Yunlin Mao <mao.looper@qq.com>
 2025-01-15 14:08:52 +08:00
Yingda Chen
70ac4db097 add file name (#1186) 
* add file name

* address cr

---------

Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
 2025-01-15 13:45:16 +08:00
Xingjun.Wang
f77b51e3c9 Support upload file and folder in the hub api (#1152) 
* update features

* update api

* add upload_file and thread_executor

* update upload file

* update api.py

* add cli for uploading

* run lint

* lint in msdataset

* temp

* add tqdm_desc in thread_executor

* update

* refine upload_file and upload_folder

* add endpoint for cli

* add uploading checker

* add path_or_fileobj and path_in_repo check in upload_file func

* add size limit to lfs: 1MB by default

* update lfs limit size: 10MB

* 5MB lfs limit

* fix test issue

* add pbar for upload_blob; del size_to_chunk_mb; fix allow_patterns and ignore_patterns

* fix commit uploaded blobs

* add update action for folder

* fix issues

* add normal files check

* update

* update

* set normal file size limit to 500MB

* update tqdm
 2025-01-14 23:59:53 +08:00
Yunlin Mao
bceb7196d1 update doc with llama_index (#1180) 2025-01-13 21:03:59 +08:00
suluyana
845a0bd5fe Fix/text gen (#1177) 
* fix text-gen: read pipeline type from configuration.json first

---------

Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>
 2025-01-13 10:05:42 +08:00
Xingjun.Wang
8f3358fe50 Add repo_id and repo_type in snapshot_download (#1172) 
* add repo_id and repo_type in snapshot_download

* fix positional args

* update
 2025-01-10 23:38:46 +08:00
Yunlin Mao
26812dc028 Unify datasets cache dir (#1178) 
* fix cache

* fix lint

* fix dataset cache

* fix lint

* remove
 2025-01-10 17:56:05 +08:00
suluyana
9aa661118c feat: all other ollama models (#1174) 
* add cases

* new models

---------

Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>
 2025-01-10 15:42:22 +08:00
suluyana
cc6fbbab5f logger.warning when using remote code (#1171) 
* logger warning when using remote code

Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>
 2025-01-10 15:38:05 +08:00
Yingda Chen
768f953b9d fix path contatenation to be windows compatabile (#1176) 
* fix path contatenation to be windows compatabile

* support dataset too

---------

Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
 2025-01-10 13:48:44 +08:00
yuze.zyz
c8f958182d wip 2025-01-08 20:50:07 +08:00
tastelikefeet
ba15de012f fix https://www.modelscope.cn/models/iic/nlp_structbert_address-parsing_chinese_base/feedback/issueDetail/20431 (#1170) 2025-01-06 17:29:44 +08:00
suluyana
299ece8b8a support ms-swift 3.0.0 (#1166) 2025-01-02 17:20:32 +08:00
Xingjun.Wang
71a54c1a51 support latest datasets version (#1163) 2025-01-02 17:06:46 +08:00
tastelikefeet
d257fdeac6 fix lint (#1168) 2025-01-02 17:06:24 +08:00
tastelikefeet
a89bb73995 fix check model (#1134) 2025-01-02 16:08:10 +08:00
Xingjun.Wang
9ee1ff8cee Update downloading progress (#1167) 
* add MODELSCOPE_SHOW_PROGRESS_THRESHOLD

* lint

* update: MODELSCOPE_SHOW_INDIVIDUAL_PROGRESS_THRESHOLD
 2024-12-31 18:28:15 +08:00
Yingda Chen
391c6cdec9 handle resuming download as retry (#1147) 
Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
 2024-12-30 12:48:55 +08:00
yuze.zyz
640b3bd49b wip 2024-12-29 18:13:05 +08:00
yuze.zyz
c42a0aca1f wip 2024-12-29 14:41:36 +08:00
yuze.zyz
225d1058e3 lint 2024-12-29 13:32:23 +08:00
yuze.zyz
9f7484bb82 wip 2024-12-29 13:21:01 +08:00
yuze.zyz
560a21e2a5 wip 2024-12-28 23:08:23 +08:00
Xingjun.Wang
8f3cf81982 Improve&fix dataset download command line (#1158) 
* fix dataset download cli

* fix default revision

* pre-commit

* add --max-workers for cli

* pre-commit

* update default max_workers
 2024-12-27 14:30:19 +08:00