ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
unsloth/tests
History
Andrew Chen e55d0e6c75
 fix(dataprep): skip .jsonl lines that are valid JSON but not objects (#7195) 
* fix(dataprep): skip .jsonl lines that are valid JSON but not objects

`_read_file_by_format` json.loads each line and hands the result to
`_extract_text_from_json`, which assumes a dict:

    for field in self._TEXT_FIELDS:
        if field in data and isinstance(data[field], str):

A JSON line does not have to be an object -- `"context"`, `["text"]` and
`42` are all valid JSON. For those, `field in data` stops being a key
lookup and becomes a substring/membership test, so `data[field]` raises:

    "context"        -> "text" in "context" is True (substring!)
                     -> TypeError: string indices must be integers
    ["text", "foo"]  -> TypeError: list indices must be integers
    42               -> TypeError: argument of type 'int' is not iterable

The TypeError escapes past `except json.JSONDecodeError: continue`, so the
whole load dies on one odd line.

That except clause is also the tell: a *malformed* line is already skipped
gracefully. A *well-formed* line that happens not to be an object should be
too -- it carries no text either way. This makes the two agree.

Reachable from `unsloth-cli.py:253` (`--dataset foo.jsonl` auto-detect) and
`RawTextDataLoader` is exported from `unsloth/__init__.py`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Slim the non-object jsonl regression test and shorten the guard comment

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
2026-07-18 05:54:50 -07:00
..
notebooks Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
python Propagate fp8 block_size before the early return in get_lora_parameters_bias (#7189) 2026-07-17 16:30:45 -07:00
qlora Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
saving fix(tokenizer): check for tokenizer.model after saving it, not before (#7194) 2026-07-18 05:53:33 -07:00
security Harden desktop release token permissions (#7172) 2026-07-16 05:01:48 -07:00
sh Studio: add UNSLOTH_SKIP_AUTOSTART installer flag (#7093) 2026-07-12 21:23:14 -07:00
studio Stabilize Studio regression tests (#7192) 2026-07-18 05:54:00 -07:00
studio_setup_ps1 Make Visual Studio + CMake optional on Windows (prebuilt llama.cpp needs no build tools) (#6499) 2026-06-22 01:11:09 -07:00
utils Probe xformers support on sm_120 instead of disabling it by version (#6828) 2026-07-14 00:01:25 -03:00
version_compat Fix FastSentenceTransformer Qwen embedding preprocessing (#6939) 2026-07-09 01:46:22 -07:00
vllm_compat Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
__init__.py Qwen 3, Bug Fixes (#2445) 2025-04-30 22:38:39 -07:00
_zoo_aggressive_cuda_spoof.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
_zoo_rocm_spoof.py Add RDNA 2/3/4 ROCm routing tests via a CPU-only torch spoof (#6935) 2026-07-07 04:41:37 -07:00
conftest.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
run_all.sh Installer: make UV_OVERRIDE space-safe on Apple Silicon (#6503) (#6639) 2026-06-24 17:34:18 -07:00
test_attention_implementation.py fix(gpt-oss): prefer flex attention over sdpa (#5701) 2026-05-22 08:38:38 -07:00
test_attn_impl_honor_explicit.py Honor an explicit sdpa or flex_attention request when flash is disabled (#6847) 2026-07-06 05:45:45 -07:00
test_bad_mappings_redirect.py Fix BAD_MAPPINGS not redirecting the -unsloth-bnb-4bit dynamic quants (#6949) 2026-07-08 17:37:44 -03:00
test_callback_signature_drift.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
test_cli_export_unpacking.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
test_enforce_kwargs_spacing.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
test_fast_gemv_dispatch.py Fix fast inference crash on compressed-tensors FP8 models (#7025) 2026-07-09 04:10:59 -07:00
test_fast_generate_slow_guard.py fast_generate: clear error for vLLM-style inputs when fast_inference=False (#6786) 2026-07-03 08:16:32 -07:00
test_finetune_last_n_layers.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
test_fp8_device_context.py Guard FP8 Triton launches with tensor device context (#6888) 2026-07-08 18:32:51 -03:00
test_fp8_restore_dropped_scale.py Restore dropped FP8 weight_scale_inv tensors on load (#6978) 2026-07-09 06:44:44 -07:00
test_fp8_tiny_e8m0.py Handle odd shapes and non-float scales in FP8BlockQuantLinear (#6848) 2026-07-06 05:44:55 -07:00
test_fused_ce_not_return_dict_logits.py fix: use EMPTY_LOGITS on the fused-CE not-return_dict path (#2068) (#6482) 2026-06-23 01:26:55 -07:00
test_gemma4_chat_template.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
test_gemma_2b_mapper_key.py Fix duplicate unsloth/gemma-2b-bnb-4bit mapper key routing the base 4bit repo to the instruct model (#6891) 2026-07-08 17:40:39 -03:00
test_generate_kwarg_gate.py Fix gpt-oss offload_embedding and generate() kwargs, and guard offload_embedding on tied/vLLM models (#6774) 2026-07-01 22:39:00 -07:00
test_get_model_name.py Fix DeepScaleR-1.5B mapper entry pointing its 16bit repo at DeepHermes-3-8B (#7088) 2026-07-14 17:56:12 -03:00
test_gradient_checkpointing_restore.py Fix TrainingArguments silently disabling unsloth gradient checkpointing (#6829) 2026-07-03 16:35:02 +01:00
test_ignored_tokenizer_casing.py Match IGNORED_TOKENIZER_NAMES case-insensitively (#6620) 2026-06-23 19:40:50 -03:00
test_import_fixes_drift.py fix: keep LoRA reloads working with PEFT 0.19 (#6748) 2026-06-30 20:26:57 +01:00
test_installer_skip_autostart.py fix(install): fail non-tauri installer errors (#7123) 2026-07-15 04:37:03 -07:00
test_loader_glob_skip.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
test_missing_torchvision_vlm.py Fix misleading 'only for image models' error for Qwen3-VL when torchvision is missing (#6525) 2026-06-23 01:28:09 -07:00
test_model_registry.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
test_moe_lora_targets.py Scope MoE expert LoRA detection to actual MLP projection targets (#6849) 2026-07-06 05:45:06 -07:00
test_multi_image_grpo_chunking.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
test_nvfp4_quant_load.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
test_offline_loading_helpers.py Fix offline checkpoint load/export: "tokenizer is weirdly not loaded" (#6554) 2026-06-25 23:16:53 -07:00
test_offload_embedding_hooks.py Fix gpt-oss offload_embedding and generate() kwargs, and guard offload_embedding on tied/vLLM models (#6774) 2026-07-01 22:39:00 -07:00
test_offload_tied_guard.py Fix gpt-oss offload_embedding and generate() kwargs, and guard offload_embedding on tied/vLLM models (#6774) 2026-07-01 22:39:00 -07:00
test_peft_tensor_parallel_compat.py fix: keep LoRA reloads working with PEFT 0.19 (#6748) 2026-06-30 20:26:57 +01:00
test_peft_weight_converter_compat.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
test_prefetch_snapshot_scope.py Auto Xet to HTTP download fallback in from_pretrained; share Studio's fallback via unsloth_zoo (#6638) 2026-07-06 05:13:25 -07:00
test_pretrain_compile_reset.py Add regression tests for the stray-forward compile-cache reset (#6569) 2026-06-22 07:22:47 -07:00
test_public_api_surface.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
test_raw_text.py fix(dataprep): skip .jsonl lines that are valid JSON but not objects (#7195) 2026-07-18 05:54:50 -07:00
test_resolve_model_class.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
test_studio_install_workspace_guard.py Update studio root-resilience tests for the inference-backend refactor (#6490) (#6553) 2026-06-22 01:10:49 -07:00
test_studio_root_resilience.py Update studio root-resilience tests for the inference-backend refactor (#6490) (#6553) 2026-06-22 01:10:49 -07:00
test_studio_shutdown_thread_wait.py Studio: fix Ctrl+C shutdown ordering (installer shell + uvicorn thread wait) (#6566) 2026-06-22 07:41:10 -07:00
test_synthetic_chunk_data.py Fix SyntheticDataKit.chunk_data emitting chunks over max_tokens (#7073) 2026-07-12 05:06:11 -07:00
test_tool_mask_zoo_compat.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
test_uninitialized_position_ids.py Load DeepSeek-OCR and other VLMs that register AutoModel in auto_map (#6421) 2026-06-18 07:03:43 -07:00
test_video_path_validation.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
test_vllm_broken_detection.py Fix fast_inference crash on ABI-broken vLLM: probe compiled extensions, not just import vllm (#6621) 2026-06-26 22:43:36 -07:00
test_windows_rocm_bnb_version.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00