diff --git a/.github/ISSUE_TEMPLATE/bug---issue.md b/.github/ISSUE_TEMPLATE/bug---issue.md index 83e0fd73a9..ffa3d3c885 100644 --- a/.github/ISSUE_TEMPLATE/bug---issue.md +++ b/.github/ISSUE_TEMPLATE/bug---issue.md @@ -6,7 +6,7 @@ labels: bug assignees: '' --- - +Note: Please do not remove the questions. Answer beside them. 1. Did you update? `pip install --upgrade unsloth unsloth_zoo` 2. `Colab` or `Kaggle` or local / cloud 3. Number GPUs used, use `nvidia-smi` @@ -16,6 +16,7 @@ assignees: '' ```python Put Minimal code to reproduce error here ###Remove Hugging Face token### +###Please make sure to check formatting properly, edit if needed.### ``` 🦥 You can also ask via our Reddit page: https://reddit.com/r/unsloth/ diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 1879186a73..25eeaedd3c 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,6 +1,6 @@ repos: - repo: https://github.com/astral-sh/ruff-pre-commit - rev: v0.15.6 + rev: v0.15.7 hooks: - id: ruff args: diff --git a/README.md b/README.md index 7cc40c7644..8f783bf661 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ Run and train AI models with a unified local interface. Quickstart • Notebooks • Documentation • - Discord + Reddit

unsloth studio ui homepage @@ -32,12 +32,12 @@ Unsloth provides several key features for both inference and training: * We work directly with teams behind [gpt-oss](https://docs.unsloth.ai/new/gpt-oss-how-to-run-and-fine-tune#unsloth-fixes-for-gpt-oss), [Qwen3](https://www.reddit.com/r/LocalLLaMA/comments/1kaodxu/qwen3_unsloth_dynamic_ggufs_128k_context_bug_fixes/), [Llama 4](https://github.com/ggml-org/llama.cpp/pull/12889), [Mistral](models/tutorials/devstral-how-to-run-and-fine-tune.md), [Gemma 1-3](https://news.ycombinator.com/item?id=39671146), and [Phi-4](https://unsloth.ai/blog/phi4), where we’ve fixed bugs that improve model accuracy. * Upload images, audio, PDFs, code, DOCX and more file types to chat with. ### Training -* Train **500+ models** up to **2x faster** with up to **70% less VRAM**, with no accuracy loss. +* Train and RL **500+ models** up to **2x faster** with up to **70% less VRAM**, with no accuracy loss. * Custom Triton and mathematical **kernels**. See some collabs we did with [PyTorch](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/fp8-reinforcement-learning) and [Hugging Face](https://unsloth.ai/docs/new/faster-moe). * **Data Recipes**: [Auto-create datasets](https://unsloth.ai/docs/new/studio/data-recipe) from **PDF, CSV, DOCX** etc. Edit data in a visual-node workflow. -* Supports full fine-tuning, pretraining, 4-bit, 16-bit and, FP8 training. +* **[Reinforcement Learning](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide)** (RL): The most efficient [RL](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide) library, using **80% less VRAM** for GRPO, [FP8](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/fp8-reinforcement-learning) etc. +* Supports full fine-tuning, RL, pretraining, 4-bit, 16-bit and, FP8 training. * **Observability**: Monitor training live, track loss and GPU usage and customize graphs. -* **Reinforcement Learning**: The most efficient [RL](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide) library, using **80% less VRAM** for GRPO, [FP8](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/fp8-reinforcement-learning) etc. * [Multi-GPU](https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth) training is supported, with major improvements coming soon. ## ⚡ Quickstart @@ -49,7 +49,7 @@ Unsloth Studio (Beta) works on **Windows, Linux, WSL** and **macOS**. * **CPU:** Supported for Chat and Data Recipes currently * **NVIDIA:** Training works on RTX 30/40/50, Blackwell, DGX Spark, Station and more * **macOS:** Currently supports chat and Data Recipes. **MLX training** is coming very soon -* **AMD:** Chat works. Train with [Unsloth Core](#unsloth-core-code-based). Studio support is coming soon. +* **AMD:** Chat + Data works. Train with [Unsloth Core](#unsloth-core-code-based). Studio support is out soon. * **Coming soon:** Training support for Apple MLX, AMD, and Intel. * **Multi-GPU:** Available now, with a major upgrade on the way @@ -172,8 +172,9 @@ Train for free with our notebooks. Read our [guide](https://unsloth.ai/docs/get- |-----------|---------|--------|----------| | **Qwen3.5 (4B)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_(4B)_Vision.ipynb) | 1.5x faster | 60% less | | **gpt-oss (20B)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-Fine-tuning.ipynb) | 2x faster | 70% less | +| **Qwen3.5 GSPO** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_5_(4B)_Vision_GRPO.ipynb) | 2x faster | 70% less | | **gpt-oss (20B): GRPO** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-GRPO.ipynb) | 2x faster | 80% less | -| **Qwen3: Advanced GRPO** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb) | 2x faster | 50% less | +| **Qwen3: Advanced GRPO** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb) | 2x faster | 70% less | | **Gemma 3 (4B) Vision** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision.ipynb) | 1.7x faster | 60% less | | **embeddinggemma (300M)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma_(300M).ipynb) | 2x faster | 20% less | | **Mistral Ministral 3 (3B)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_VL_(3B)_Vision.ipynb) | 1.5x faster | 60% less | @@ -196,13 +197,13 @@ Train for free with our notebooks. Read our [guide](https://unsloth.ai/docs/get- - **FP8 & Vision RL**: You can now do FP8 & VLM GRPO on consumer GPUs. [FP8 Blog](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/fp8-reinforcement-learning) • [Vision RL](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/vision-reinforcement-learning-vlm-rl) - **gpt-oss** by OpenAI: Read our [RL blog](https://unsloth.ai/docs/models/gpt-oss-how-to-run-and-fine-tune/gpt-oss-reinforcement-learning), [Flex Attention](https://unsloth.ai/docs/models/gpt-oss-how-to-run-and-fine-tune/long-context-gpt-oss-training) blog and [Guide](https://unsloth.ai/docs/models/gpt-oss-how-to-run-and-fine-tune). -## 🔗 Links and Resources +## 💚 Community and Links | Type | Links | | ----------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | +|   **Discord** | [Join Discord server](https://discord.com/invite/unsloth) | |   **r/unsloth Reddit** | [Join Reddit community](https://reddit.com/r/unsloth) | | 📚 **Documentation & Wiki** | [Read Our Docs](https://unsloth.ai/docs) | |   **Twitter (aka X)** | [Follow us on X](https://twitter.com/unslothai) | -| 💾 **Installation** | [Pip & Docker Install](https://unsloth.ai/docs/get-started/install) | | 🔮 **Our Models** | [Unsloth Catalog](https://unsloth.ai/docs/get-started/unsloth-model-catalog) | | ✍️ **Blog** | [Read our Blogs](https://unsloth.ai/blog) | diff --git a/studio/backend/core/inference/inference.py b/studio/backend/core/inference/inference.py index 6cb077f4a9..1a265690ff 100644 --- a/studio/backend/core/inference/inference.py +++ b/studio/backend/core/inference/inference.py @@ -927,6 +927,12 @@ class InferenceBackend: logger.warning(f"Could not apply get_chat_template: {e}") # Step 2: Format with tokenizer.apply_chat_template() + if system_prompt: + template_messages = [ + {"role": "system", "content": system_prompt} + ] + messages + else: + template_messages = messages try: if not (hasattr(tokenizer, "chat_template") and tokenizer.chat_template): raise ValueError( @@ -937,7 +943,7 @@ class InferenceBackend: f"one via tokenizer.chat_template before inference." ) formatted_prompt = tokenizer.apply_chat_template( - messages, tokenize = False, add_generation_prompt = True + template_messages, tokenize = False, add_generation_prompt = True ) logger.debug(f"Formatted prompt: {formatted_prompt[:200]}...") except Exception as e: @@ -992,19 +998,40 @@ class InferenceBackend: # Prepare vision messages if image: - vision_messages = [ - { - "role": "user", - "content": [ - {"type": "image"}, - {"type": "text", "text": user_message}, - ], - } - ] + user_msg = { + "role": "user", + "content": [ + {"type": "image"}, + {"type": "text", "text": user_message}, + ], + } + if system_prompt: + vision_messages = [ + { + "role": "system", + "content": [{"type": "text", "text": system_prompt}], + }, + user_msg, + ] + else: + vision_messages = [user_msg] - input_text = processor.apply_chat_template( - vision_messages, add_generation_prompt = True, tokenize = False - ) + try: + input_text = processor.apply_chat_template( + vision_messages, add_generation_prompt = True, tokenize = False + ) + except Exception as e: + if system_prompt: + logger.warning( + f"Vision processor for '{self.active_model_name}' may not support " + f"system messages; retrying without. Original error: {e}" + ) + vision_messages = [user_msg] + input_text = processor.apply_chat_template( + vision_messages, add_generation_prompt = True, tokenize = False + ) + else: + raise inputs = processor( image, input_text, diff --git a/studio/backend/run.py b/studio/backend/run.py index 5c24c550c7..e32b912c37 100644 --- a/studio/backend/run.py +++ b/studio/backend/run.py @@ -73,18 +73,79 @@ def _resolve_external_ip() -> str: return "0.0.0.0" +def _get_pid_on_port(port: int) -> "tuple[int, str] | None": + """Return (pid, process_name) of the process listening on *port*, or None. + + Uses psutil when available. Falls back gracefully to None so callers + can still report the port conflict without process details. + + Works on Windows, macOS, and Linux wherever psutil is installed. + """ + try: + import psutil + except ImportError: + return None + try: + for conn in psutil.net_connections(kind = "tcp"): + if conn.status == "LISTEN" and conn.laddr.port == port: + if conn.pid is None: + return None + try: + proc = psutil.Process(conn.pid) + return (conn.pid, proc.name()) + except (psutil.NoSuchProcess, psutil.AccessDenied): + return (conn.pid, "") + except (psutil.AccessDenied, OSError) as e: + # psutil.net_connections() needs elevated privileges on some platforms + logger.debug("Failed to scan network connections for port %s: %s", port, e) + return None + + def _is_port_free(host: str, port: int) -> bool: - """Check if a port is available for binding.""" + """Check if a port is available for binding. + + When *host* is ``0.0.0.0`` (wildcard), we also check whether anything + is already listening on ``127.0.0.1`` (and ``::1`` when IPv6 is + available). An SSH tunnel or similar process may hold the loopback + address while our wildcard bind still succeeds, making Unsloth Studio + unreachable via ``localhost``. + + Works on Windows, macOS, and Linux. + """ import socket + # 1. Can we bind to the requested address? + # Use getaddrinfo so both IPv4 ("0.0.0.0") and IPv6 ("::") hosts + # resolve to the correct address family automatically. try: - with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: + addr_info = socket.getaddrinfo(host, port, socket.AF_UNSPEC, socket.SOCK_STREAM) + family, socktype, proto, _, sockaddr = addr_info[0] + with socket.socket(family, socktype, proto) as s: s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) - s.bind((host, port)) - return True + s.bind(sockaddr) except OSError: return False + # 2. When binding to all interfaces, verify that localhost is not + # already claimed by another process (e.g. an SSH -L tunnel). + # We attempt a TCP connect -- if it succeeds something is listening. + if host in ("0.0.0.0", "::"): + for loopback, family in [ + ("127.0.0.1", socket.AF_INET), + ("::1", socket.AF_INET6), + ]: + try: + with socket.socket(family, socket.SOCK_STREAM) as s: + s.settimeout(1) + if s.connect_ex((loopback, port)) == 0: + # Connection succeeded -- port is taken on loopback + return False + except OSError: + # IPv6 disabled or other OS-level restriction -- skip + continue + + return True + def _find_free_port(host: str, start: int, max_attempts: int = 20) -> int: """Find a free port starting from `start`, trying up to max_attempts ports.""" @@ -149,11 +210,11 @@ def _graceful_shutdown(server = None): logger.info("All subprocesses cleaned up") -# The uvicorn server instance — set by run_server(), used by callers +# The uvicorn server instance -- set by run_server(), used by callers # that need to tell the server to exit (e.g. signal handlers). _server = None -# Shutdown event — used to wake the main loop on signal +# Shutdown event -- used to wake the main loop on signal _shutdown_event = None @@ -205,9 +266,22 @@ def run_server( # Auto-find free port if requested port is in use if not _is_port_free(host, port): original_port = port - port = _find_free_port(host, port) + blocker = _get_pid_on_port(port) + port = _find_free_port(host, port + 1) if not silent: - print(f"Port {original_port} is in use, using port {port} instead") + print("") + print("=" * 50) + if blocker: + pid, name = blocker + print( + f"Port {original_port} is already in use by " f"{name} (PID {pid})." + ) + else: + print(f"Port {original_port} is already in use.") + print(f"Unsloth Studio will use port {port} instead.") + print(f"Open http://localhost:{port} in your browser.") + print("=" * 50) + print("") # Setup frontend if path provided if frontend_path: @@ -297,7 +371,7 @@ if __name__ == "__main__": sys.stderr.flush() sys.exit(1) - # ── Signal handler — ensures subprocess cleanup on Ctrl+C ──── + # Signal handler -- ensures subprocess cleanup on Ctrl+C def _signal_handler(signum, frame): _graceful_shutdown(_server) _shutdown_event.set() diff --git a/studio/frontend/src/components/assistant-ui/reasoning.tsx b/studio/frontend/src/components/assistant-ui/reasoning.tsx index 4f3f8075a4..0e37f6d433 100644 --- a/studio/frontend/src/components/assistant-ui/reasoning.tsx +++ b/studio/frontend/src/components/assistant-ui/reasoning.tsx @@ -151,7 +151,7 @@ function ReasoningTrigger({ & { streaming?: boolean }) { const scrollRef = useRef(null); + const shouldAutoScrollRef = useRef(true); useEffect(() => { if (!(streaming && scrollRef.current)) { return; } const el = scrollRef.current; + const updateAutoScroll = () => { + const distanceFromBottom = el.scrollHeight - el.scrollTop - el.clientHeight; + shouldAutoScrollRef.current = distanceFromBottom <= 24; + }; const observer = new MutationObserver(() => { - el.scrollTop = el.scrollHeight; + if (shouldAutoScrollRef.current) { + el.scrollTop = el.scrollHeight; + } }); + el.addEventListener("scroll", updateAutoScroll); observer.observe(el, { childList: true, subtree: true, characterData: true, }); + shouldAutoScrollRef.current = true; el.scrollTop = el.scrollHeight; - return () => observer.disconnect(); + return () => { + observer.disconnect(); + el.removeEventListener("scroll", updateAutoScroll); + }; }, [streaming]); return ( @@ -330,6 +342,7 @@ const ReasoningGroupImpl: ReasoningGroupComponent = ({ }); const [manualOpen, setManualOpen] = useState(false); + const [dismissedWhileStreaming, setDismissedWhileStreaming] = useState(false); const [duration, setDuration] = useState(0); const startTimeRef = useRef(null); @@ -345,17 +358,23 @@ const ReasoningGroupImpl: ReasoningGroupComponent = ({ } }, [isReasoningStreaming]); - const isOpen = isReasoningStreaming || manualOpen; + // Reset dismissed flag when a new stream starts + useEffect(() => { + if (isReasoningStreaming) { + setDismissedWhileStreaming(false); + } + }, [isReasoningStreaming]); - const variant = isReasoningStreaming - ? "outline" - : manualOpen - ? "outline" - : "ghost"; + // Derived: open during streaming (unless dismissed), or if user manually opened after + const isOpen = (isReasoningStreaming && !dismissedWhileStreaming) || manualOpen; + const variant = isOpen ? "outline" : "ghost"; + // Allow closing during streaming (matches ChatGPT) const handleOpenChange = useCallback( (open: boolean) => { - if (!isReasoningStreaming) { + if (isReasoningStreaming) { + setDismissedWhileStreaming(!open); + } else { setManualOpen(open); } }, @@ -368,14 +387,17 @@ const ReasoningGroupImpl: ReasoningGroupComponent = ({ onOpenChange={handleOpenChange} variant={variant} > -
+
- {isOpen && !isReasoningStreaming && ( - - )} +
+ {isOpen && !isReasoningStreaming && ( + + )} +
{ const prompt = useAuiState(({ suggestion }) => suggestion.prompt); const isDisabled = useAuiState(({ thread }) => thread.isDisabled); const isRunning = useAuiState(({ thread }) => thread.isRunning); - const supportsTools = useChatRuntimeStore((s) => s.supportsTools); - const supportsReasoning = useChatRuntimeStore((s) => s.supportsReasoning); - const allTools = SUGGESTION_TOOLS[prompt] ?? []; - const tools = allTools.filter((tool) => { - if (tool === "thinking") return supportsReasoning; - return supportsTools; - }); + const tools = SUGGESTION_TOOLS[prompt] ?? []; return (
+ { + setSavePresetOpen(nextOpen); + if (!nextOpen) { + setPresetNameDraft(""); + } + }} + > + + + Save Preset + + Enter a name for this inference preset. + + +
{ + event.preventDefault(); + savePresetWithName(presetNameDraft); + }} + className="space-y-4" + > + setPresetNameDraft(event.target.value)} + placeholder="Preset name" + maxLength={80} + /> + + + + +
+ + ); diff --git a/studio/frontend/src/features/chat/shared-composer.tsx b/studio/frontend/src/features/chat/shared-composer.tsx index 89b65d4d99..78cfcc66d2 100644 --- a/studio/frontend/src/features/chat/shared-composer.tsx +++ b/studio/frontend/src/features/chat/shared-composer.tsx @@ -237,6 +237,9 @@ export function SharedComposer({ const checkpoint = s.params.checkpoint; return s.models.find((m) => m.id === checkpoint); }); + const modelLoaded = useChatRuntimeStore( + (s) => !!s.params.checkpoint && !s.modelLoading, + ); const supportsReasoning = useChatRuntimeStore((s) => s.supportsReasoning); const reasoningEnabled = useChatRuntimeStore((s) => s.reasoningEnabled); const setReasoningEnabled = useChatRuntimeStore((s) => s.setReasoningEnabled); @@ -245,6 +248,8 @@ export function SharedComposer({ const setToolsEnabled = useChatRuntimeStore((s) => s.setToolsEnabled); const codeToolsEnabled = useChatRuntimeStore((s) => s.codeToolsEnabled); const setCodeToolsEnabled = useChatRuntimeStore((s) => s.setCodeToolsEnabled); + const reasoningDisabled = !modelLoaded || !supportsReasoning; + const toolsDisabled = !modelLoaded || !supportsTools; const setPendingAudioStore = useChatRuntimeStore((s) => s.setPendingAudio); const clearPendingAudioStore = useChatRuntimeStore((s) => s.clearPendingAudio); @@ -519,70 +524,73 @@ export function SharedComposer({ )} - {supportsReasoning && ( - - )} - {supportsTools && ( - + - )} - {supportsTools && ( - + - )} + )} + aria-label={codeToolsEnabled ? "Disable code execution" : "Enable code execution"} + > + + Code +
{dictationSupported && ( diff --git a/studio/frontend/src/features/chat/stores/chat-runtime-store.ts b/studio/frontend/src/features/chat/stores/chat-runtime-store.ts index fea5442187..920737a279 100644 --- a/studio/frontend/src/features/chat/stores/chat-runtime-store.ts +++ b/studio/frontend/src/features/chat/stores/chat-runtime-store.ts @@ -2,6 +2,7 @@ // Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0 import { create } from "zustand"; +import { toast } from "sonner"; import { DEFAULT_INFERENCE_PARAMS, type ChatLoraSummary, @@ -13,6 +14,8 @@ const AUTO_TITLE_KEY = "unsloth_chat_auto_title"; const AUTO_HEAL_TOOL_CALLS_KEY = "unsloth_auto_heal_tool_calls"; const MAX_TOOL_CALLS_KEY = "unsloth_max_tool_calls_per_message"; const TOOL_CALL_TIMEOUT_KEY = "unsloth_tool_call_timeout"; +const INFERENCE_PARAMS_KEY = "unsloth_chat_inference_params"; +let hasShownInferencePersistenceWarning = false; function canUseStorage(): boolean { return typeof window !== "undefined"; @@ -59,6 +62,65 @@ function saveInt(key: string, value: number): void { } } +function asFiniteNumber(value: unknown, fallback: number): number { + return typeof value === "number" && Number.isFinite(value) ? value : fallback; +} + +function asString(value: unknown, fallback: string): string { + return typeof value === "string" ? value : fallback; +} + +function asBoolean(value: unknown, fallback: boolean): boolean { + return typeof value === "boolean" ? value : fallback; +} + +function loadInferenceParams(): InferenceParams { + if (!canUseStorage()) return DEFAULT_INFERENCE_PARAMS; + try { + const raw = localStorage.getItem(INFERENCE_PARAMS_KEY); + if (!raw) return DEFAULT_INFERENCE_PARAMS; + const parsed = JSON.parse(raw) as Partial; + return { + temperature: asFiniteNumber(parsed.temperature, DEFAULT_INFERENCE_PARAMS.temperature), + topP: asFiniteNumber(parsed.topP, DEFAULT_INFERENCE_PARAMS.topP), + topK: asFiniteNumber(parsed.topK, DEFAULT_INFERENCE_PARAMS.topK), + minP: asFiniteNumber(parsed.minP, DEFAULT_INFERENCE_PARAMS.minP), + repetitionPenalty: asFiniteNumber( + parsed.repetitionPenalty, + DEFAULT_INFERENCE_PARAMS.repetitionPenalty, + ), + presencePenalty: asFiniteNumber( + parsed.presencePenalty, + DEFAULT_INFERENCE_PARAMS.presencePenalty, + ), + maxSeqLength: asFiniteNumber( + parsed.maxSeqLength, + DEFAULT_INFERENCE_PARAMS.maxSeqLength, + ), + maxTokens: asFiniteNumber(parsed.maxTokens, DEFAULT_INFERENCE_PARAMS.maxTokens), + systemPrompt: asString(parsed.systemPrompt, DEFAULT_INFERENCE_PARAMS.systemPrompt), + checkpoint: DEFAULT_INFERENCE_PARAMS.checkpoint, + trustRemoteCode: asBoolean( + parsed.trustRemoteCode, + DEFAULT_INFERENCE_PARAMS.trustRemoteCode ?? false, + ), + }; + } catch { + return DEFAULT_INFERENCE_PARAMS; + } +} + +function saveInferenceParams(params: InferenceParams): boolean { + if (!canUseStorage()) return false; + try { + const { checkpoint: _, ...rest } = params; + localStorage.setItem(INFERENCE_PARAMS_KEY, JSON.stringify(rest)); + return true; + } catch { + return false; + } +} + type ChatRuntimeStore = { params: InferenceParams; models: ChatModelSummary[]; @@ -117,7 +179,7 @@ type ChatRuntimeStore = { }; export const useChatRuntimeStore = create((set) => ({ - params: DEFAULT_INFERENCE_PARAMS, + params: loadInferenceParams(), models: [], loras: [], runningByThreadId: {}, @@ -144,7 +206,18 @@ export const useChatRuntimeStore = create((set) => ({ contextUsage: null, modelLoading: false, setModelLoading: (loading) => set({ modelLoading: loading }), - setParams: (params) => set({ params }), + setParams: (params) => + set(() => { + const persisted = saveInferenceParams(params); + if (!persisted && !hasShownInferencePersistenceWarning) { + hasShownInferencePersistenceWarning = true; + toast.warning("Chat settings could not be persisted", { + description: + "Your changes apply now, but may reset after refresh.", + }); + } + return { params }; + }), setModels: (models) => set({ models }), setLoras: (loras) => set({ loras }), setThreadRunning: (threadId, running) => diff --git a/unsloth/models/_utils.py b/unsloth/models/_utils.py index ce8da7910f..13acc98ea6 100644 --- a/unsloth/models/_utils.py +++ b/unsloth/models/_utils.py @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -__version__ = "2026.3.10" +__version__ = "2026.3.11" __all__ = [ "SUPPORTS_BFLOAT16", diff --git a/unsloth/models/rl_replacements.py b/unsloth/models/rl_replacements.py index 805086324e..9f555416d4 100755 --- a/unsloth/models/rl_replacements.py +++ b/unsloth/models/rl_replacements.py @@ -153,11 +153,18 @@ def sft_trainer_prepare_dataset(function_name, function): "if 'tokenizer' not in locals(): tokenizer = processing_class\n" "if 'formatting_func' not in locals(): raise RuntimeError('Unsloth: Please file a bug report - `formatting_func` does not exist!')\n" "if 'dataset_text_field' not in locals() and 'args' in locals(): dataset_text_field = args.dataset_text_field\n" - "if 'dataset_text_field' not in locals(): raise RuntimeError('Unsloth: Please file a bug report - `dataset_text_field` does not exist!')\n" - "test_text = dataset[0][dataset_text_field] if (formatting_func is None and dataset_text_field is not None) else formatting_func(dataset[0])[0]\n" + "if 'dataset_text_field' not in locals(): dataset_text_field = None\n" + "if formatting_func is None and dataset_text_field is None and 'prompt' in dataset[0] and 'completion' in dataset[0]:\n" + " test_text = (dataset[0]['prompt'] + dataset[0]['completion']) if (isinstance(dataset[0]['prompt'], str) and isinstance(dataset[0]['completion'], str)) else None\n" + "elif formatting_func is None and dataset_text_field is not None:\n" + " test_text = dataset[0][dataset_text_field]\n" + "elif formatting_func is not None:\n" + " test_text = formatting_func(dataset[0])[0]\n" + "else:\n" + " test_text = None\n" "chat_template = getattr(tokenizer, 'chat_template', None)\n" "chat_template = '' if chat_template is None else chat_template\n" - "has_bos_token_already = (test_text.startswith(tokenizer.bos_token) or tokenizer.bos_token in chat_template) " + "has_bos_token_already = ((test_text is not None and test_text.startswith(tokenizer.bos_token)) or tokenizer.bos_token in chat_template) " "if getattr(tokenizer, 'bos_token', None) is not None else False\n" "if 'add_special_tokens' not in locals() and has_bos_token_already:\n" " from functools import partial\n" diff --git a/unsloth/tokenizer_utils.py b/unsloth/tokenizer_utils.py index c445879df7..96c22f62ff 100644 --- a/unsloth/tokenizer_utils.py +++ b/unsloth/tokenizer_utils.py @@ -974,11 +974,18 @@ def patch_sft_trainer_tokenizer(): "if 'tokenizer' not in locals(): tokenizer = processing_class\n" "if 'formatting_func' not in locals(): raise RuntimeError('Unsloth: Please file a bug report - `formatting_func` does not exist!')\n" "if 'dataset_text_field' not in locals() and 'args' in locals(): dataset_text_field = args.dataset_text_field\n" - "if 'dataset_text_field' not in locals(): raise RuntimeError('Unsloth: Please file a bug report - `dataset_text_field` does not exist!')\n" - "test_text = dataset[0][dataset_text_field] if (formatting_func is None and dataset_text_field is not None) else formatting_func(dataset[0])[0]\n" + "if 'dataset_text_field' not in locals(): dataset_text_field = None\n" + "if formatting_func is None and dataset_text_field is None and 'prompt' in dataset[0] and 'completion' in dataset[0]:\n" + " test_text = (dataset[0]['prompt'] + dataset[0]['completion']) if (isinstance(dataset[0]['prompt'], str) and isinstance(dataset[0]['completion'], str)) else None\n" + "elif formatting_func is None and dataset_text_field is not None:\n" + " test_text = dataset[0][dataset_text_field]\n" + "elif formatting_func is not None:\n" + " test_text = formatting_func(dataset[0])[0]\n" + "else:\n" + " test_text = None\n" "chat_template = getattr(tokenizer, 'chat_template', None)\n" "chat_template = '' if chat_template is None else chat_template\n" - "has_bos_token_already = (test_text.startswith(tokenizer.bos_token) or tokenizer.bos_token in chat_template) " + "has_bos_token_already = ((test_text is not None and test_text.startswith(tokenizer.bos_token)) or tokenizer.bos_token in chat_template) " "if getattr(tokenizer, 'bos_token', None) is not None else False\n" "if 'add_special_tokens' not in locals() and has_bos_token_already:\n" " from functools import partial\n" diff --git a/unsloth/trainer.py b/unsloth/trainer.py index 65abe6801f..8bb4440021 100644 --- a/unsloth/trainer.py +++ b/unsloth/trainer.py @@ -132,8 +132,8 @@ except: class UnslothTrainingArguments(TrainingArguments): def __init__(self, embedding_learning_rate: float = None, *args, **kwargs): - embedding_learning_rate = embedding_learning_rate super().__init__(*args, **kwargs) + self.embedding_learning_rate = embedding_learning_rate def _create_unsloth_optimizer(