Mirrors/modelscope
1
0
Fork 0
mirror of https://github.com/modelscope/modelscope.git synced 2026-02-24 20:19:51 +01:00
Code Issues Packages Projects Releases Wiki Activity
2,567 Commits 267 Branches 107 Tags
87c542bb554a8d4f1cd00a34bd9bd5b599e5996e
Commit Graph

2567 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yingda Chen
87c542bb55 support dataset as well 2025-02-07 14:30:39 +08:00
Yingda Chen
57376f51b6 add ut 2025-02-07 13:49:58 +08:00
Yingda Chen
44b9ccd0cb allow multiple include/exclude patterns for command line 2025-02-07 12:51:38 +08:00
tastelikefeet
1cf7f4ff52 fix create_commit login (#1210) 2025-02-06 18:22:29 +08:00
Yingda Chen
369606cda8 fix cache path (#1211) 
Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
 2025-02-06 17:05:32 +08:00
tastelikefeet
3b4841054d clone and lint #1205 (#1209) 2025-02-06 11:36:11 +08:00
tastelikefeet
f74433f6b2 Add more patches for hf (#1160) 2025-02-06 11:09:37 +08:00
Xingjun.Wang
50726e01d5 Update upload func (#1204) 
* update upload

* update

* temp

* fix pr comments

* update api.py
 2025-02-05 20:09:18 +08:00
Xingjun.Wang
d53dc6d703 remove MODELSCOPE_SHOW_INDIVIDUAL_PROGRESS_THRESHOLD (#1206) 2025-02-05 18:47:58 +08:00
Xingjun.Wang
8048f3c6b2 update upload progress (#1203) 2025-01-31 14:37:19 +08:00
yingda
d16945ffc8 test 2025-01-31 11:17:28 +08:00
Yingda Chen
65075191f7 clean directory structure (#1201) 
默认
模型：
~/.cache/modelscope/hub/models/{model_owner}/{model_name}
数据集：
~/.cache/modelscope/hub/datasets/{dataset_owner}/{dataset_name}

配置MODELSCOPE_CACHE环境变量：
模型：
$MODELSCOPE_CACHE/models/{model_owner}/{model_name}
数据集：
$MODELSCOPE_CACHE/datasets/{dataset_owner}/{dataset_name}

Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
 2025-01-23 17:31:49 +08:00
Xingjun.Wang
2c9ec4ff2d Add file size check in upload_folder (#1198) 
* add file size check in upload_folder

* update size_threshold_to_enforce_lfs
 2025-01-21 20:29:18 +08:00
suluyana
dc7e22ebeb feat: new ollama template (#1199) 
* feat: deepseek-r1, olmo2, command-r7b, deepseek-v3

---------

Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>
 2025-01-21 17:49:54 +08:00
Yingda Chen
58c90dee2c fix model directory (#1197) 
Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
 2025-01-20 15:19:15 +08:00
tastelikefeet
b1404cdc2e Fix typo and unused imports (#1196) 
* fix file exists of sub folder

* fix code doc

* add ci

* lint code

* fix unused imports and typo
 2025-01-18 10:22:22 +08:00
tastelikefeet
b892164d11 Fix file exists of sub folder (#1192) 
* fix file exists of sub folder

* fix code doc

* add ci

* lint code
 2025-01-18 09:59:14 +08:00
Xingjun.Wang
8387bd8e9c Fix cookies due to firewall token expired (#1193) 
* fix cookies due to firewall token expired

* update
 2025-01-18 00:57:29 +08:00
Xingjun.Wang
77d7262c22 fix lint (#1188) 2025-01-15 14:25:09 +08:00
Xingjun.Wang
648187626a Merge release 1.22 (#1187) 
* bump version 1.22.0

* fix confict between lmdeploy & vllm

* flash-attn version

* fix version building image

* fix https://www.modelscope.cn/models/iic/nlp_structbert_address-parsing_chinese_base/feedback/issueDetail/20431 (#1170)

* fix path contatenation to be windows compatabile (#1176)

* fix path contatenation to be windows compatabile

* support dataset too

---------

Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>

* logger.warning when using remote code (#1171)

* logger warning when using remote code

Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>

* feat: all other ollama models (#1174)

* add cases

* new models

---------

Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>

* Unify datasets cache dir (#1178)

* fix cache

* fix lint

* fix dataset cache

* fix lint

* remove

* Add repo_id and repo_type in snapshot_download (#1172)

* add repo_id and repo_type in snapshot_download

* fix positional args

* update

* Fix/text gen (#1177)

* fix text-gen: read pipeline type from configuration.json first

---------

Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>

* update doc with llama_index (#1180)

* update version to 1.22.1

* merge release/1.22

---------

Co-authored-by: suluyana <suluyan_sly@163.com>
Co-authored-by: tastelikefeet <58414341+tastelikefeet@users.noreply.github.com>
Co-authored-by: Yingda Chen <yingdachen@apache.org>
Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
Co-authored-by: suluyana <110878454+suluyana@users.noreply.github.com>
Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>
Co-authored-by: Yunlin Mao <mao.looper@qq.com>
 2025-01-15 14:08:52 +08:00
Yingda Chen
70ac4db097 add file name (#1186) 
* add file name

* address cr

---------

Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
 2025-01-15 13:45:16 +08:00
Xingjun.Wang
f77b51e3c9 Support upload file and folder in the hub api (#1152) 
* update features

* update api

* add upload_file and thread_executor

* update upload file

* update api.py

* add cli for uploading

* run lint

* lint in msdataset

* temp

* add tqdm_desc in thread_executor

* update

* refine upload_file and upload_folder

* add endpoint for cli

* add uploading checker

* add path_or_fileobj and path_in_repo check in upload_file func

* add size limit to lfs: 1MB by default

* update lfs limit size: 10MB

* 5MB lfs limit

* fix test issue

* add pbar for upload_blob; del size_to_chunk_mb; fix allow_patterns and ignore_patterns

* fix commit uploaded blobs

* add update action for folder

* fix issues

* add normal files check

* update

* update

* set normal file size limit to 500MB

* update tqdm
 2025-01-14 23:59:53 +08:00
Yunlin Mao
bceb7196d1 update doc with llama_index (#1180) 2025-01-13 21:03:59 +08:00
suluyana
845a0bd5fe Fix/text gen (#1177) 
* fix text-gen: read pipeline type from configuration.json first

---------

Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>
 2025-01-13 10:05:42 +08:00
Xingjun.Wang
8f3358fe50 Add repo_id and repo_type in snapshot_download (#1172) 
* add repo_id and repo_type in snapshot_download

* fix positional args

* update
 2025-01-10 23:38:46 +08:00
Yunlin Mao
26812dc028 Unify datasets cache dir (#1178) 
* fix cache

* fix lint

* fix dataset cache

* fix lint

* remove
 2025-01-10 17:56:05 +08:00
suluyana
9aa661118c feat: all other ollama models (#1174) 
* add cases

* new models

---------

Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>
 2025-01-10 15:42:22 +08:00
suluyana
cc6fbbab5f logger.warning when using remote code (#1171) 
* logger warning when using remote code

Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>
 2025-01-10 15:38:05 +08:00
Yingda Chen
768f953b9d fix path contatenation to be windows compatabile (#1176) 
* fix path contatenation to be windows compatabile

* support dataset too

---------

Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
 2025-01-10 13:48:44 +08:00
tastelikefeet
ba15de012f fix https://www.modelscope.cn/models/iic/nlp_structbert_address-parsing_chinese_base/feedback/issueDetail/20431 (#1170) 2025-01-06 17:29:44 +08:00
suluyana
299ece8b8a support ms-swift 3.0.0 (#1166) 2025-01-02 17:20:32 +08:00
Xingjun.Wang
71a54c1a51 support latest datasets version (#1163) 2025-01-02 17:06:46 +08:00
tastelikefeet
d257fdeac6 fix lint (#1168) 2025-01-02 17:06:24 +08:00
tastelikefeet
a89bb73995 fix check model (#1134) 2025-01-02 16:08:10 +08:00
Xingjun.Wang
9ee1ff8cee Update downloading progress (#1167) 
* add MODELSCOPE_SHOW_PROGRESS_THRESHOLD

* lint

* update: MODELSCOPE_SHOW_INDIVIDUAL_PROGRESS_THRESHOLD
 2024-12-31 18:28:15 +08:00
Yingda Chen
391c6cdec9 handle resuming download as retry (#1147) 
Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
 2024-12-30 12:48:55 +08:00
Xingjun.Wang
8f3cf81982 Improve&fix dataset download command line (#1158) 
* fix dataset download cli

* fix default revision

* pre-commit

* add --max-workers for cli

* pre-commit

* update default max_workers
 2024-12-27 14:30:19 +08:00
suluyana
60780769b1 Fix/daily (#1155) 
* fix(llm ppl): 1. cache position; 2. stream_gready_search; 3. swift_mapping

* fix punkt

---------

Co-authored-by: suluyan <suluyan.sly@alibaba-inc.com>
 2024-12-23 09:55:12 +08:00
Yingda Chen
9304d40539 fix user-specified cache path (#1154) 
Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
 2024-12-18 09:01:41 +08:00
suluyana
1fe211ffe5 fix pipeline builder when model is not supported (#1125) 
* fix pipeline builder when model is not supported

* fix ci & skip
---------

Co-authored-by: suluyan.sly@alibaba-inc.com <suluyan.sly@alibaba-inc.com>
 2024-12-12 19:24:38 +08:00
Yunlin Mao
c3a9bcd803 Fix python push model no disk space (#1148) 
* fix python push model
 2024-12-12 16:46:00 +08:00
Yingda Chen
8ba3a185e5 use streaming hash validation (#1127) 
* support streaming hash
Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
 2024-12-11 18:59:42 +08:00
mushenL
b9f7b60d8f fix daily fail case 1210yk (#1142) 
* init commit

* init commit

* init commit

* add AutoImageProcessor and BatchFeature

* Modify format specifications

* fix daily fail case

* fix daily fail case
 2024-12-11 10:40:26 +08:00
tastelikefeet
7579fac6f3 fix ut (#1143) 2024-12-11 10:19:29 +08:00
suluyana
40b689795a Fix ollama template (#1141) 
* add marco o1

* add qwq

* feat: llama3.3

* fix space , add exaone3.5 , fix ministral
 2024-12-10 14:22:02 +08:00
Yunlin Mao
31bf308e44 downgrade moviepy to 1.0.3 (#1133) 2024-12-10 10:10:01 +08:00
tastelikefeet
d2ef1003ea Skip obsolete sd pipeline (#1131) 2024-12-05 20:07:32 +08:00
Yingda Chen
101b9249f8 fix missing import (#1126) 
Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
 2024-12-05 09:26:34 +08:00
Yingda Chen
8d66932e3f make hash validation optional (#1124) 
Co-authored-by: Yingda Chen <yingda.chen@alibaba-inc.com>
 2024-12-04 18:46:48 +08:00
Yingda Chen
c67ffd4cbc optional hash 2024-12-04 18:25:52 +08:00