ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Fix inference failing for transformers 5.x models (trust_remote_code) (#4652)

Browse source 
* Fix inference failing for transformers 5.x models (trust_remote_code)

The training worker in core/training/worker.py auto-enables
trust_remote_code for unsloth/* models that need transformers 5.x
(e.g. NVIDIA-Nemotron-3-Nano-4B). The inference worker did not have
the same logic, so loading these models for chat would fail with
"No config file found" while training worked fine.

Add the same auto-detection to the inference worker so
trust_remote_code is set automatically when needed.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Daniel Han 2026-03-27 04:51:30 -07:00 • committed by GitHub
commit a7c43bc46d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 17 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

18
studio/backend/core/inference/worker.py
View file

@ -156,12 +156,28 @@ def _handle_load(backend, config: dict, resp_queue: Any) -> None:
except Exception as e:
logger.warning("Could not read adapter_config.json: %s", e)
# Auto-enable trust_remote_code for unsloth/* transformers 5.x models
# (matches the training worker logic in core/training/worker.py)
trust_remote_code = config.get("trust_remote_code", False)
if not trust_remote_code:
from utils.transformers_version import needs_transformers_5
model_name = config["model_name"]
if needs_transformers_5(model_name) and model_name.lower().startswith(
"unsloth/"
):
trust_remote_code = True
logger.info(
"Auto-enabled trust_remote_code for unsloth/* transformers 5.x model: %s",
model_name,
)
success = backend.load_model(
config = mc,
max_seq_length = config.get("max_seq_length", 2048),
load_in_4bit = load_in_4bit,
hf_token = hf_token,
trust_remote_code = config.get("trust_remote_code", False),
trust_remote_code = trust_remote_code,
)
if success: