Add Hugging Face dataset streaming mode to Studio (#4946)
* Add HF dataset streaming mode to Studio * Added default value for datasetStreaming in training-config-store.ts * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Handle None max_steps for streaming validation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: fast-fail streaming validation and guard incompatible modes Reject dataset_streaming at the API boundary when hf_dataset is empty, the dataset is vision/audio, or max_steps is not set. Probe eval split with get_dataset_split_names before the streaming load so typos fail immediately instead of mid-training. Guard column_names=None after map on iterables. Hide the UI toggle for non-text configurations and clear the stale flag when config becomes incompatible. * studio: add streaming dataset tests, iterable helper, and streaming template/format support (WIP) Work-in-progress on top of feat/studio-dataset-streaming-mode (PR #4946): - new test_training_streaming.py and iterable.py dataset helper - streaming support in chat_templates.py and format_conversion.py - additional streaming guards in trainer.py / models / routes - frontend streaming wiring in params-section and training-config-store Committed to preserve uncommitted work before merging latest main. * studio: fix review-team findings for streaming + main merge BLOCKER: streaming + raw-text/CPT crashed on len(IterableDataset). Guard it in the start route (reject format_type=="raw" or training_type=="Continued Pretraining") and in isStreamingSupported (datasetFormat !== "raw"). Also: - models/training.py: validate hf_dataset/subset/split (charset+length, block ..//); cap dataset slice indices (le=1e9); note validator ordering - chat_templates.py: guard _apply_custom_mapping .map() for streaming - trainer.py: warn when packing+streaming - training-config-store.ts: persist-migration bump to v11 (standalone datasetStreaming backfill); add isVisionModel to NON_PERSISTED; toast on silent streamingCompatiblePatch mutations in the 4 indirect setters - tests: route rejections (max_steps, raw/cpt), slice cap, unsafe hf_dataset * studio: enable raw-text/CPT dataset streaming + streaming UX polish - raw_text: keep the lazy filter but skip len()-based row counting for IterableDatasets so raw-text / CPT can stream; guard the eval-size log - routes/trainer: drop the raw/CPT streaming block; add a defensive not-streaming guard on the eval auto-split (train_test_split) - dataset-section: streaming toggle is visible-but-disabled and lists the exact unmet requirement(s) in its tooltip; block embedding models - training-start-overlay: show "streaming (no full download)" instead of a stuck download bar for streaming runs - trim the streaming test suite to the high-value cases * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: address streaming review (MLX/embedding guards, sliced eval split, rehydrate timing) - routes: reject dataset_streaming for embedding training and on Apple Silicon (MLX); both loaders materialize the full dataset instead of streaming - trainer: validate the base eval split name so streaming eval accepts HF slice syntax such as "validation[:1000]" - training-config-store: defer the onRehydrateStorage setState to a microtask so it doesn't hit the store's TDZ during synchronous hydration - test: streaming start rejects embedding models * studio: harden HF dataset streaming (column_names, split slicing, empty/eval bounds, gating) Address a deeper streaming review: - raw_text: resolve_column_names() guards IterableDataset.column_names=None (from_generator / unresolved features) so raw-text and CPT streaming no longer raise TypeError before training - models/routes: reject HF slice syntax in train_split/eval_split when streaming (load_dataset(streaming=True) raises "Bad split"); reject mixed sources (local/S3) and embedding/MLX streaming at the API, not just in the UI - trainer: an empty post-slice/filter stream fails preflight with a clear message; streaming eval is capped (STREAMING_EVAL_MAX_SAMPLES) so each eval terminates; the manual-slice shortcut falls back to a regular load when train_split is sliced - format_conversion: streaming conversions preflight the first mapped row so format errors surface before training, not mid-iteration - frontend: block streaming on Apple Silicon; clear datasetStreaming when a dataset is detected as image/audio at start * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: fix CI for streaming PR (lint blocker + no-torch sandbox + preflight test) - trainer.py: drop unused `IterableDataset` import (hoist safety-net blocker). - test_training_streaming.py: only select real classes (isinstance type) when locating the trainer class, so a MagicMock-stubbed global is never passed to object.__new__ (fixes TypeError on the Python 3.10-3.13 jobs). - no-torch import sandboxes (test_e2e_no_torch_sandbox.py, test_studio_import_no_torch.py): teach the chat_templates/format_conversion exec stubs and the full-import-chain copy list about the new `.iterable` module so the AFTER/runtime cases import without torch again. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com> Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai> Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
This commit is contained in:
Sanat Bhargava
GitHub
parent
dbc13f02c9
commit
1fc8bf53c7
22 changed files with 1482 additions and 190 deletions
|
|
@ -27,6 +27,7 @@ CHAT_TEMPLATES = DATASETS_DIR / "chat_templates.py"
|
|||
FORMAT_DETECTION = DATASETS_DIR / "format_detection.py"
|
||||
MODEL_MAPPINGS = DATASETS_DIR / "model_mappings.py"
|
||||
VLM_PROCESSING = DATASETS_DIR / "vlm_processing.py"
|
||||
ITERABLE = DATASETS_DIR / "iterable.py"
|
||||
HARDWARE_PY = HARDWARE_DIR / "hardware.py"
|
||||
|
||||
# Studio venv for server tests
|
||||
|
|
@ -280,9 +281,13 @@ class TestBeforeAfterImportChain:
|
|||
mm = types.ModuleType('model_mappings')
|
||||
mm.MODEL_TO_TEMPLATE_MAPPER = {{}}
|
||||
sys.modules['model_mappings'] = mm
|
||||
it = types.ModuleType('iterable')
|
||||
it.is_streaming_dataset = lambda *a, **k: False
|
||||
sys.modules['iterable'] = it
|
||||
source = open({str(CHAT_TEMPLATES)!r}).read()
|
||||
source = source.replace('from .format_detection import', 'from format_detection import')
|
||||
source = source.replace('from .model_mappings import', 'from model_mappings import')
|
||||
source = source.replace('from .iterable import', 'from iterable import')
|
||||
exec(source)
|
||||
print("OK")
|
||||
""")
|
||||
|
|
@ -323,6 +328,7 @@ class TestBeforeAfterImportChain:
|
|||
VLM_PROCESSING,
|
||||
DATA_COLLATORS,
|
||||
CHAT_TEMPLATES,
|
||||
ITERABLE,
|
||||
]:
|
||||
if src.exists():
|
||||
shutil.copy2(src, pkg_dir / src.name)
|
||||
|
|
@ -431,10 +437,14 @@ class TestDataclassInstantiation:
|
|||
mm = types.ModuleType('model_mappings')
|
||||
mm.MODEL_TO_TEMPLATE_MAPPER = {{}}
|
||||
sys.modules['model_mappings'] = mm
|
||||
it = types.ModuleType('iterable')
|
||||
it.is_streaming_dataset = lambda *a, **k: False
|
||||
sys.modules['iterable'] = it
|
||||
ns = {{}}
|
||||
source = open({str(CHAT_TEMPLATES)!r}).read()
|
||||
source = source.replace('from .format_detection import', 'from format_detection import')
|
||||
source = source.replace('from .model_mappings import', 'from model_mappings import')
|
||||
source = source.replace('from .iterable import', 'from iterable import')
|
||||
exec(source, ns)
|
||||
assert 'Instruction' in ns['DEFAULT_ALPACA_TEMPLATE']
|
||||
print("OK")
|
||||
|
|
@ -544,11 +554,15 @@ class TestEdgeCasesBrokenTorch:
|
|||
mm = types.ModuleType('model_mappings')
|
||||
mm.MODEL_TO_TEMPLATE_MAPPER = {{}}
|
||||
sys.modules['model_mappings'] = mm
|
||||
it = types.ModuleType('iterable')
|
||||
it.is_streaming_dataset = lambda *a, **k: False
|
||||
sys.modules['iterable'] = it
|
||||
|
||||
ns = {{}}
|
||||
source = open({str(CHAT_TEMPLATES)!r}).read()
|
||||
source = source.replace('from .format_detection import', 'from format_detection import')
|
||||
source = source.replace('from .model_mappings import', 'from model_mappings import')
|
||||
source = source.replace('from .iterable import', 'from iterable import')
|
||||
exec(source, ns)
|
||||
|
||||
# Import succeeds -- this is the fix
|
||||
|
|
|
|||
|
|
@ -254,10 +254,15 @@ class TestChatTemplatesNoTorchVenv:
|
|||
model_mappings.MODEL_TO_TEMPLATE_MAPPER = {{}}
|
||||
sys.modules['model_mappings'] = model_mappings
|
||||
|
||||
iterable = types.ModuleType('iterable')
|
||||
iterable.is_streaming_dataset = lambda *a, **k: False
|
||||
sys.modules['iterable'] = iterable
|
||||
|
||||
# Read and transform the source: replace relative imports with absolute
|
||||
source = open({str(CHAT_TEMPLATES)!r}).read()
|
||||
source = source.replace('from .format_detection import', 'from format_detection import')
|
||||
source = source.replace('from .model_mappings import', 'from model_mappings import')
|
||||
source = source.replace('from .iterable import', 'from iterable import')
|
||||
|
||||
exec(source)
|
||||
|
||||
|
|
@ -295,10 +300,15 @@ class TestChatTemplatesNoTorchVenv:
|
|||
model_mappings.MODEL_TO_TEMPLATE_MAPPER = {{}}
|
||||
sys.modules['model_mappings'] = model_mappings
|
||||
|
||||
iterable = types.ModuleType('iterable')
|
||||
iterable.is_streaming_dataset = lambda *a, **k: False
|
||||
sys.modules['iterable'] = iterable
|
||||
|
||||
ns = {{}}
|
||||
source = open({str(CHAT_TEMPLATES)!r}).read()
|
||||
source = source.replace('from .format_detection import', 'from format_detection import')
|
||||
source = source.replace('from .model_mappings import', 'from model_mappings import')
|
||||
source = source.replace('from .iterable import', 'from iterable import')
|
||||
exec(source, ns)
|
||||
|
||||
assert 'DEFAULT_ALPACA_TEMPLATE' in ns, "DEFAULT_ALPACA_TEMPLATE not defined"
|
||||
|
|
@ -379,6 +389,10 @@ class TestFormatConversionNoTorchVenv:
|
|||
datasets_mod.IterableDataset = type('IterableDataset', (), {{}})
|
||||
sys.modules['datasets'] = datasets_mod
|
||||
|
||||
iterable_mod = types.ModuleType('iterable')
|
||||
iterable_mod.is_streaming_dataset = lambda *a, **k: False
|
||||
sys.modules['iterable'] = iterable_mod
|
||||
|
||||
# Stub utils.hardware
|
||||
utils_mod = types.ModuleType('utils')
|
||||
hardware_mod = types.ModuleType('utils.hardware')
|
||||
|
|
@ -390,6 +404,7 @@ class TestFormatConversionNoTorchVenv:
|
|||
# Read and exec format_conversion.py
|
||||
source = open({str(FORMAT_CONVERSION)!r}).read()
|
||||
source = source.replace('from .format_detection import', 'from format_detection import')
|
||||
source = source.replace('from .iterable import', 'from iterable import')
|
||||
ns = {{'__name__': '__test__'}}
|
||||
exec(source, ns)
|
||||
|
||||
|
|
@ -437,6 +452,10 @@ class TestFormatConversionNoTorchVenv:
|
|||
datasets_mod.IterableDataset = type('IterableDataset', (), {{}})
|
||||
sys.modules['datasets'] = datasets_mod
|
||||
|
||||
iterable_mod = types.ModuleType('iterable')
|
||||
iterable_mod.is_streaming_dataset = lambda *a, **k: False
|
||||
sys.modules['iterable'] = iterable_mod
|
||||
|
||||
utils_mod = types.ModuleType('utils')
|
||||
hardware_mod = types.ModuleType('utils.hardware')
|
||||
hardware_mod.dataset_map_num_proc = lambda n=None: 1
|
||||
|
|
@ -446,6 +465,7 @@ class TestFormatConversionNoTorchVenv:
|
|||
|
||||
source = open({str(FORMAT_CONVERSION)!r}).read()
|
||||
source = source.replace('from .format_detection import', 'from format_detection import')
|
||||
source = source.replace('from .iterable import', 'from iterable import')
|
||||
ns = {{'__name__': '__test__'}}
|
||||
exec(source, ns)
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue