mirror of
https://github.com/modelscope/modelscope.git
synced 2026-09-01 19:49:03 +02:00
9ab427b29a
* fix(msdatasets): make HfFileSystem monkey-patch idempotent for repeated streaming loads
When streaming=True the patches on HfFileSystem._open and HfFileSystem.__init__
are intentionally kept alive after load_dataset_with_ctx exits. A second call to
load_dataset_with_ctx then snaps the already-patched wrappers as 'originals',
causing _hf_fs_init_with_cookie / _hf_fs_open to call themselves recursively
until RecursionError: maximum recursion depth exceeded.
Fix: detect whether HfFileSystem is already patched before overwriting
_hf_fs_{open,init}_original, and skip re-applying the patch when it is already
in place. The finally-block restores only patches that were applied in the
current invocation, leaving pre-existing patches intact.
Adds: test_hf_filesystem_patch_idempotent_for_repeated_streaming_loads
* fix(msdatasets): restore patches when streaming load fails; hermetic tests
Addresses both Gemini Code Assist review comments on PR #1754:
1. Bug fix – patch leak on failed streaming loads:
Replace guard with flag.
The flag is set True only after load_dataset() succeeds with streaming=True.
Any exception in load_dataset (network, auth, invalid dataset, etc.) leaves
_streaming_dataset_returned=False, so the finally block always restores all
monkey-patches, preventing permanent global-state corruption.
2. Test improvement – hermetic test environment:
Add _reset_hf_filesystem_patch() helper that strips pre-existing patches
before each unit test, ensuring tests are independent of execution order.
Add test_hf_filesystem_patch_restored_when_streaming_load_fails to cover
the bug scenario introduced in fix 1.
* update requirements for hub
* feat(hub): restore progress_callbacks on snapshot_download shim (issue #1757)
The progress_callbacks parameter was dropped from snapshot_download after v1.38, breaking legacy code and GUI progress reporting. Re-expose it on modelscope.hub.snapshot_download (positioned before token per the v1.34 signature) and forward it to modelscope_hub.compat. Requires modelscope-hub >= 0.1.8.
Add network-free forwarding tests that patch the compat delegate so the real shim executes.
* fix lint
* feat(hub): add legacy-cache capability guard to snapshot_download shim
Warn once (thread-safe) when the installed modelscope-hub lacks pre-1.38 legacy cache auto-detection (DownloadManager._find_legacy_repo_dir, added in modelscope-hub>=0.1.7), so programmatic snapshot_download/dataset_snapshot_download callers are not silently downloading into the new layout while an old cache exists. Capability is probed, not reimplemented (kept single-sourced in modelscope-hub). Adds network-free tests for present/absent capability and fire-once behavior.
55 lines
1.9 KiB
Python
55 lines
1.9 KiB
Python
import importlib
|
|
import sys
|
|
import unittest
|
|
from unittest import mock
|
|
|
|
|
|
class LegacyCacheGuardTest(unittest.TestCase):
|
|
"""Tests for the modelscope-hub capability guard in the shim.
|
|
|
|
Network-free: only probes whether the loaded modelscope-hub exposes the
|
|
legacy-cache auto-detection capability and warns once when it does not.
|
|
"""
|
|
|
|
def setUp(self):
|
|
# NOTE: ``modelscope.hub.snapshot_download`` the *attribute* is shadowed
|
|
# by the re-exported function in ``modelscope.hub.__init__``; use
|
|
# importlib to obtain the actual submodule object.
|
|
sd = importlib.import_module('modelscope.hub.snapshot_download')
|
|
self.sd = sd
|
|
# Reset the fire-once probe cache before each test.
|
|
sd._legacy_cache_capability = None
|
|
|
|
def tearDown(self):
|
|
self.sd._legacy_cache_capability = None
|
|
|
|
def test_capability_present_no_warning(self):
|
|
sd = self.sd
|
|
# The real modelscope-hub DownloadManager has _find_legacy_repo_dir.
|
|
with mock.patch.object(sd.logger, 'warning') as warn:
|
|
sd._warn_if_legacy_cache_detection_unavailable()
|
|
self.assertTrue(sd._legacy_cache_capability)
|
|
warn.assert_not_called()
|
|
|
|
def test_capability_absent_warns_once(self):
|
|
sd = self.sd
|
|
|
|
class _OldDownloadManager: # lacks _find_legacy_repo_dir
|
|
pass
|
|
|
|
fake_module = mock.MagicMock()
|
|
fake_module.DownloadManager = _OldDownloadManager
|
|
|
|
with mock.patch.dict(sys.modules,
|
|
{'modelscope_hub._download': fake_module}):
|
|
with mock.patch.object(sd.logger, 'warning') as warn:
|
|
sd._warn_if_legacy_cache_detection_unavailable()
|
|
sd._warn_if_legacy_cache_detection_unavailable()
|
|
|
|
self.assertFalse(sd._legacy_cache_capability)
|
|
self.assertEqual(warn.call_count, 1) # fire-once
|
|
|
|
|
|
if __name__ == '__main__':
|
|
unittest.main()
|