ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Studio: use (gguf) context length before max seq length (#5111)

Browse source 
* fix: use (gguf) context length before max seq length

For GGUF models context length is used instead of `maxSeqLength`

Fixes #4893

* Studio: split rollback max_seq_length from target load/validate

Restore params.maxSeqLength as the value passed to the target model's
validateModel and effectiveMaxSeqLength fallback, and introduce a
separate rollbackMaxSeqLength used only in the rollback loadModel call.
Without the split, switching from a GGUF model with a large native
context to a non-GGUF target polluted the new load via the
effectiveMaxSeqLength else-branch.

Also:
- Detect a previous GGUF via isGguf, activeGgufVariant, or a .gguf
  suffix on the checkpoint, so local/LM Studio paths not present in
  the models catalog still take the GGUF rollback path.
- Use 0 as the GGUF rollback fallback to match the sentinel used at
  the normal GGUF load site (ggufContextLength fallback to 0); 4096
  would reintroduce the original truncation bug when both
  customContextLength and ggufContextLength are null.
- Reuse the captured stateBeforeUnload for modelRequiresTrustRemoteCode
  and drop the now-unused DEFAULT_INFERENCE_PARAMS value import.

* Studio: drop pending customContextLength from GGUF rollback

rollbackMaxSeqLength previously preferred customContextLength over
ggufContextLength, but customContextLength is a pending, not-yet-applied
slider edit: chat-settings-sheet.tsx treats it as the "dirty" marker
(ctxDirty = customContextLength !== null) and use-chat-model-runtime.ts
clears it to null on every successful load. Rollback exists to restore
the previously-loaded model at its confirmed running context, so leaking
a pending slider value can load the rollback target at a context the
user never validated against VRAM, potentially causing the rollback
itself to fail.

---------

Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
This commit is contained in:
Konstantin Azizov 2026-04-24 12:53:10 +02:00 • committed by GitHub
commit c875dc17e4
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 13 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

19
studio/frontend/src/features/chat/hooks/use-chat-model-runtime.ts
View file

@ -419,12 +419,19 @@ export function useChatModelRuntime() {
let previousWasUnloaded = false;
const currentCheckpoint =
useChatRuntimeStore.getState().params.checkpoint;
const paramsBeforeLoad = useChatRuntimeStore.getState().params;
const trustRemoteCode = paramsBeforeLoad.trustRemoteCode ?? false;
const maxSeqLength = paramsBeforeLoad.maxSeqLength;
const hfToken = useChatRuntimeStore.getState().hfToken || null;
const stateBeforeUnload = useChatRuntimeStore.getState();
const trustRemoteCode = stateBeforeUnload.params.trustRemoteCode ?? false;
const maxSeqLength = stateBeforeUnload.params.maxSeqLength;
const previousIsGguf =
previousModel?.isGguf === true
|| previousVariant != null
|| (previousCheckpoint?.toLowerCase().endsWith(".gguf") ?? false);
const rollbackMaxSeqLength = previousIsGguf
? (stateBeforeUnload.ggufContextLength ?? 0)
: maxSeqLength;
const hfToken = stateBeforeUnload.hfToken || null;
const previousModelRequiresTrustRemoteCode =
useChatRuntimeStore.getState().modelRequiresTrustRemoteCode;
stateBeforeUnload.modelRequiresTrustRemoteCode;
try {
// Lightweight pre-flight validation: avoid unloading a working model
// if the new identifier is clearly invalid (e.g. bad HF id / path).
@ -542,7 +549,7 @@ export function useChatModelRuntime() {
await loadModel({
model_path: previousCheckpoint,
hf_token: hfToken,
max_seq_length: maxSeqLength,
max_seq_length: rollbackMaxSeqLength,
load_in_4bit: true,
is_lora: previousIsLora,
gguf_variant: previousVariant,