xingjun.wang
4493786e04
remove unused code
2024-07-22 20:51:01 +08:00
xingjun.wang
c2e954245d
reuse _download function in DataDownloadManager
2024-07-22 20:49:33 +08:00
xingjun.wang
b2157f1476
update comment
2024-07-22 20:45:00 +08:00
xingjun.wang
c9e1591798
update comments
2024-07-22 03:08:27 +08:00
xingjun.wang
71e6b7d72e
add pillow
2024-07-22 03:00:21 +08:00
xingjun.wang
72d05d1f75
fix _dataset_info
2024-07-22 02:40:00 +08:00
xingjun.wang
271f2a1590
update requirements for datasets version restriction
2024-07-22 02:08:53 +08:00
xingjun.wang
2b0d79253c
update print info
2024-07-21 20:20:47 +08:00
xingjun.wang
829a27e9ea
set trust_remote_code is True in load_dataset_with_ctx
2024-07-21 20:02:21 +08:00
xingjun.wang
671504b410
update get_module_with_script
2024-07-21 19:37:27 +08:00
xingjun.wang
6445b8c6d1
fix disable_tqdm in some functions for 2.19.0 or later
2024-07-21 19:29:28 +08:00
xingjun.wang
32ceec8f74
add assert for youku data UT
2024-07-21 19:13:37 +08:00
xingjun.wang
de98268558
adapt to datasets version: 2.19.0 or later
2024-07-21 18:51:53 +08:00
xingjun.wang
f3f9480654
update UT for general dataset
2024-07-21 18:37:12 +08:00
xingjun.wang
dd5ef32b5c
add youku in test_general_datasets
2024-07-21 03:05:17 +08:00
xingjun.wang
f8285e1fd7
update ut
2024-07-21 02:52:37 +08:00
xingjun.wang
db6add8e1c
fix download config copy
2024-07-20 19:39:35 +08:00
xingjun.wang
b4b7e29b28
fix streaming for youku-mplug and adopt latest datasets
2024-07-20 18:59:19 +08:00
xingjun.wang
ef5f7d6543
del _datasets_server import in hf_dataset_util
2024-07-19 23:17:23 +08:00
Yingda Chen
710fdaf081
Merge pull request #910 from SNESNya/patch-1
...
将 "library" 改为 "Library"
2024-07-15 08:42:23 +08:00
LAGSNES
c069e1cbf4
将 "library" 改为 "Library"
2024-07-15 00:23:04 +08:00
wenmeng zhou
09d7a231de
update discord in readme ( #908 )
2024-07-13 19:50:54 +08:00
wenmeng zhou
5c9e5a6283
Update discord in README.md
2024-07-13 19:29:23 +08:00
liuyhwangyh
f7a32e48e3
add dataset download ( #906 )
...
* add dataset download
* fix cr issue
* fix cv matplotlib issue
* refactor code
* fix ut issue
* remove debug code
* remove unused import
* fix import issue
* sleep 65s before start docker avoid kill and run failed
---------
Co-authored-by: mulin.lyh <mulin.lyh@taobao.com >
2024-07-12 19:06:44 +08:00
liuyhwangyh
53acf79d04
fix logger issue ( #899 )
...
Co-authored-by: mulin.lyh <mulin.lyh@taobao.com >
2024-07-03 20:03:46 +08:00
liuyhwangyh
083aa80d09
refactor ast index file generator ( #897 )
...
* refactor ast index file generator
* fix lint issue
---------
Co-authored-by: mulin.lyh <mulin.lyh@taobao.com >
2024-07-03 18:23:50 +08:00
Zhongbing-Chen
3bafc5c344
exchange the order of replacement ( #893 )
2024-07-01 20:49:08 +08:00
liuyhwangyh
6995662f38
shorten check model timeout and snapshot_download file patter folder/ to folder/* ( #896 )
...
Co-authored-by: mulin.lyh <mulin.lyh@taobao.com >
2024-07-01 15:52:16 +08:00
zhyncs
5b8e1e971c
fix ignore_file_pattern ( #886 )
...
* fix ignore_file_pattern
---------
Co-authored-by: root <root@kunkun>
2024-07-01 14:31:39 +08:00
liuyhwangyh
ff77845772
add command line usage, optimize command line log, optimize packaging… ( #888 )
...
* add command line usage, optimize command line log, optimize packaging version compatible
* Update command.md
* fix logging dup
* Refactor install dependencies (#889 )
* refactor install dependencies, default only depencies of hub and datasets
* move pandas import to function
* update hub deps
* test
* remove generate ast file
* remove gast dependency
* replace gast with ast
* add dependency version
* remove lap for compile error
* fix comments issue
* add install ollama
---------
Co-authored-by: mulin.lyh <mulin.lyh@taobao.com >
---------
Co-authored-by: mulin.lyh <mulin.lyh@taobao.com >
2024-06-30 20:15:03 +08:00
Firmament-cyou
8cdbeec3f6
Support swift model inference for llm pipeline ( #880 )
...
* Support swift model inference for llm pipeline
* fix bug
* Add 'swfit' llm_framework and fix stream bug
* For pass the unittest run action
2024-06-30 19:43:13 +08:00
yfchenmodelscope
f65f45959d
Add Eres2netv2ep4 ( #892 )
...
* add eres2netv2ep4
2024-06-27 21:09:51 +08:00
wenmeng zhou
5d00b7c304
Merge pull request #876 from modelscope/release/1.15
...
Release/1.15
2024-06-06 18:01:22 +08:00
mulin.lyh
f34817e95d
modify version merge master
2024-06-06 16:23:19 +08:00
wenmeng zhou
894bc90d96
remove assets and download from oss when running app ( #875 )
2024-06-05 21:18:21 +08:00
mulin.lyh
909e54fa33
modify build script
2024-06-05 19:08:46 +08:00
mulin.lyh
339ebab9dd
fix download retry bug, add get_cache_dir to hut/util
2024-06-04 16:18:52 +08:00
mulin.lyh
7b18f99d03
release to 06.05
2024-06-03 10:09:07 +08:00
mulin.lyh
4a22b05891
restore http_get_file interface
2024-05-28 15:09:05 +08:00
mulin.lyh
e2d8a6d45f
restore TDNN.py
2024-05-28 14:57:05 +08:00
mulin.lyh
8947d69fe2
Merge branch 'master' into release/1.15
2024-05-28 14:55:46 +08:00
liuyhwangyh
17da5e2264
fix error report ( #868 )
...
Co-authored-by: mulin.lyh <mulin.lyh@taobao.com >
2024-05-28 14:38:19 +08:00
mulin.lyh
b46c5bc3f7
add TDNN.py and version to 1.15.0
2024-05-25 22:35:29 +08:00
mulin.lyh
f69ddba0bb
fix TDNN.py and tdnn.py
2024-05-25 22:34:11 +08:00
liuyhwangyh
f93a184d88
add donwload command line and local_dir parameter ( #866 )
...
* add donwload command line and local_dir parameter
Co-authored-by: mulin.lyh <mulin.lyh@taobao.com >
2024-05-25 14:21:55 +08:00
liuyhwangyh
5c470f8941
Download optimize ( #862 )
...
* fix #845
Supports resumption of downloads from breakpoints, optimized download progress bar, finer display granularity, better experience under low bandwidth, and added function of downloading specified directories.
* restore push to hub
* fix merge issue
* fix ut issue
---------
Co-authored-by: mulin.lyh <mulin.lyh@taobao.com >
2024-05-24 15:37:43 +08:00
yfchenmodelscope
f9451bfe38
add sdpn and tdnn ( #865 )
...
* add sdpn and tdnn
2024-05-24 14:39:16 +08:00
Alpha Hinex
da985ad92d
Fix json.decoder.JSONDecodeError when load pipeline_schema.json ( #859 )
...
When we use `modelscope server` to serve the model, the comma after last json object in this file will cause the error bellow:
```log
ERROR: Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 732, in lifespan
async with self.lifespan_context(app) as maybe_state:
File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 608, in __aenter__
await self._router.startup()
File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 711, in startup
handler()
File "/opt/conda/lib/python3.10/site-packages/modelscope/server/core/event_handlers.py", line 37, in startup
_startup_model(app)
File "/opt/conda/lib/python3.10/site-packages/modelscope/server/core/event_handlers.py", line 22, in _startup_model
info['schema'] = get_task_schemas(app.state.pipeline.group_key)
File "/opt/conda/lib/python3.10/site-packages/modelscope/utils/input_output.py", line 837, in get_task_schemas
schema = json.load(f)
File "/opt/conda/lib/python3.10/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/opt/conda/lib/python3.10/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/opt/conda/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/opt/conda/lib/python3.10/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 3836 column 1 (char 100720)
```
2024-05-24 11:39:37 +08:00
liuyhwangyh
82ee20f447
fix issue #845 ( #861 )
...
* fix #845
Co-authored-by: mulin.lyh <mulin.lyh@taobao.com >
2024-05-23 20:34:52 +08:00
wenmeng zhou
88c60f114a
remove necessary dependency of transformer ( #857 )
...
* remove necessary dependency of transformer
* fix import error
* fix typo
2024-05-15 20:44:17 +08:00
