ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

studio: use largest single GPU for the diffusion catalog fit budget

Browse source 
The catalog fit budget used gpu.memoryTotalGb, which sums VRAM across
every GPU. That sum is right for the chat/llama.cpp path (tensor-split
shards across cards) but wrong for the diffusion/video catalog: those
backends place the whole pipeline on a single device (pipe.to or cpu
offload, never device_map), so on a multi-GPU host the fit toggle and
bare-group-click routing credited VRAM no single card has. On a 4x24 GB
plus 128 GB RAM host the 114 GB Wan A14B bf16 group passed the toggle
(0.7*96 + 0.7*128 budget) and a click would OOM, the exact load the
toggle exists to prevent. Expose maxDeviceMemoryGb (largest single
device) from use-gpu-info and use it for deviceBudget; the chat path
keeps the sum. Single-GPU hosts are unchanged.
This commit is contained in:
Daniel Han 2026-07-10 04:32:31 +00:00
commit 8218c9bf42
2 changed files with 16 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

7
studio/frontend/src/components/assistant-ui/model-selector/pickers.tsx
View file

@ -2074,8 +2074,13 @@ export function HubModelPicker({
[downloadedSet],
);
const deviceBudget = useMemo(
// Largest single device, NOT the multi-GPU sum: the diffusion/video
// backends place the whole pipeline on one device (pipe.to / cpu-offload,
// never device_map), so summed VRAM would pass groups no single card can
// hold (e.g. a 114 GB group "fits" a 4x24 GB host) and a bare group click
// would OOM -- the exact load the fit toggle exists to prevent.
() => ({
gpuGb: gpu.available ? gpu.memoryTotalGb : 0,
gpuGb: gpu.available ? gpu.maxDeviceMemoryGb : 0,
systemRamGb: gpu.systemRamAvailableGb || 0,
}),
[gpu],

10
studio/frontend/src/hooks/use-gpu-info.ts
View file

@ -8,7 +8,13 @@ import type { SystemInfoResponse } from "./use-system";
export interface GpuInfo {
available: boolean;
name: string;
/** Sum across every GPU. Right for the chat/llama.cpp path (tensor-split
* shards across cards); wrong as a single-placement budget. */
memoryTotalGb: number;
/** Largest single device. The diffusion/video backends place the whole
* pipeline on one device (pipe.to / cpu-offload, never device_map), so
* their fit budget must not credit VRAM from the other cards. */
maxDeviceMemoryGb: number;
cpuCore: number;
cpuThread: number;
systemRamAvailableGb: number;
@ -19,6 +25,7 @@ const DEFAULT_GPU: GpuInfo = {
available: false,
name: "Unknown",
memoryTotalGb: 0,
maxDeviceMemoryGb: 0,
cpuCore: 0,
cpuThread: 0,
systemRamAvailableGb: 0,
@ -59,6 +66,9 @@ async function fetchGpuOnce(): Promise<GpuInfo> {
available: true,
name: devices[0]?.name ?? "Unknown",
memoryTotalGb: devices.reduce((sum, d) => sum + (d.memory_total_gb ?? 0), 0),
maxDeviceMemoryGb: devices.reduce(
(max, d) => Math.max(max, d.memory_total_gb ?? 0), 0,
),
}
: { ...DEFAULT_GPU, ...base };
cachedGpu = info;