Trim and tighten code comments and docstrings across studio/ Python. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.
* Update VRAM estimator to cater to broader model configs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix attn backend check, better support for MoE etc
* Studio: tighten VRAM estimator structured-shape and attention paths
- Conservative attention fallback: when resolve_attention_implementation
fails, charge the quadratic non-flash activation path instead of
silently keeping the optimistic flash_attention_2 default.
- Resolve attention on a shallow config copy so _set_attn_impl does not
mutate the cached config returned by _load_config_for_gpu_estimate.
- Use getattr for AutoModelForCausalLM._model_mapping to avoid raising
on private-attribute renames in transformers.
- Treat sdpa as O(n) linear attention; PyTorch SDPA dispatches to flash
or memory-efficient backends, only eager needs the quadratic term.
- Per-layer activation accounting: structured archs (head_dim,
layer_types, attention_k_eq_v, num_kv_shared_layers, double-wide MLP)
now flow into compute_activation_bytes via _text_linear_dims, instead
of using the legacy hidden_size//num_attention_heads KV/MLP shape.
- Exclude MLA configs (q_lora_rank set) from the structured-shape path
so q_lora low-rank projection formulas keep applying when head_dim is
also present.
- _build_text_module_elements emits a single MLA self_attn aggregate
using _compute_attn_elements when q_lora_rank is set, avoiding the
~10% overcount that fed into _compute_skipped_quantizable_elements.
- Restrict _module_path_matches to known text-tower prefixes so VLM
skip names like vision_tower.model.layers.<i>.self_attn.q_proj no
longer falsely shadow the text alias model.layers.<i>.self_attn.q_proj.
- Pick up enable_moe_block from the config and add the per-layer dense
MLP alongside the MoE experts in compute_total_params and
compute_lora_params (Gemma4-style parallel dense + MoE block).
- Single-pass structured layer accounting in _compute_layer_elements,
removing the duplicate _text_linear_dims walks.
- Drop the now-zero (activations - activations_computed) shard term in
VramBreakdown.min_gpu_vram and the stale comment that referred to it.
- attention_implementation typed as Optional[str] to match call sites
that pass None.
- Inline rationale comments on DOUBLE_QUANT_4BIT_FACTOR and
NON_FLASH_ATTENTION_FACTOR pointing at VRAM_ESTIMATION.md.
* Studio: extend parallel-MoE accounting + non-prefix dense layer support
- Apply enable_moe_block / moe_has_dense_mlp symmetrically: activation
per-layer MLP size in _layer_qkv_mlp_sizes now adds the parallel dense
MLP for MoE layers, matching the weight and LoRA accounting added in
the prior commit. Skip-quantizable mapping in _build_text_module_elements
now registers both mlp.experts and per-projection mlp.{name} entries
for MoE layers when the parallel dense block is present, so an
llm_int8_skip_modules entry like "model.layers.N.mlp" covers both.
- Track dense layer indices as a tuple (dense_layer_indices) extracted
from first_k_dense_replace or decoder_sparse_step + mlp_only_layers,
and dispatch dense-vs-MoE accounting through _is_dense_mlp_layer. The
prior count-based path silently mis-bucketed layers when mlp_only_layers
was non-prefix (e.g. [3, 5] on an 8-layer model). num_dense_layers is
derived from len(dense_layer_indices) for backward compatibility.
- Drop the redundant ">0" check in _is_kv_shared_layer so configs with
num_kv_shared_layers == num_hidden_layers (every layer shared) are
correctly recognized as shared.
- Refresh VRAM_ESTIMATION.md section 5 to note that sdpa joins
flash_attention_2 in the linear activation path; refresh the
VramBreakdown.activations_computed comment now that the activation
floor is gone.
* Studio: Gemma4 PLE accounting, flex_attention, KV-share guard restore
- Add flex_attention to LINEAR_ATTENTION_IMPLS. Unsloth's
resolve_attention_implementation returns "flex_attention" when
HAS_FLASH_ATTENTION is False and the model class supports flex; PyTorch
FlexAttention is a memory-efficient kernel, not a quadratic eager
attention path. Without this, activation estimates over-charge ~36x.
- Restore the `> 0` guard in _is_kv_shared_layer. Transformers Gemma4
(modeling_gemma4.py:1031, modular_gemma4.py:863, :926) uses
`layer_idx >= first_kv_shared_layer_idx > 0`, so configs that mark
every layer as KV-shared raise on construction. Reverting the
unconditional acceptance avoids producing a detailed estimate for a
shape the actual model code rejects.
- Extend the parallel dense MLP path (`enable_moe_block`) in
_build_text_module_elements: when the arch is non-structured, use
arch.intermediate_size for the dense gate/up/down dims instead of
_text_linear_dims (which returns moe_intermediate_size via
_get_mlp_size). Prior code under-counted skipped quantizable elements
for the parallel dense block by up to 8x on GLM-style configs.
- Add Gemma4 per-layer-input (PLE) module accounting:
per_layer_model_projection (one global Linear) plus per-layer
per_layer_input_gate and per_layer_projection are added to the
quantizable text-linear total in _compute_layer_elements;
post_per_layer_input_norm and per_layer_projection_norm flow into
the non-quantizable bucket. compute_lora_params adds the same three
Linear modules to the all-linear total. References:
transformers_versions/5.7.0/.../gemma4/modular_gemma4.py:1077-1083,
:1247-1253.
- VRAM_ESTIMATION.md section 5 now lists flex_attention alongside sdpa
and flash_attention_2 as linear-memory backends.
* Studio: shared-expert variants, mlp_layer_types dispatch, PLE skip, all-linear str, deepcopy resolver
Five targeted estimator corrections:
- _compute_dense_layer_indices now reads `mlp_layer_types` ahead of
`first_k_dense_replace` / `decoder_sparse_step`. Transformers Exaone-MoE,
Laguna, Hy_v3, GLM-MoE-DSA, GLM4-MoE-Lite, Ernie4_5_VL_MoE etc. ship the
per-position list and may omit the prefix-style fields entirely.
- _build_text_module_elements registers per_layer_input_gate /
per_layer_projection (per layer) and per_layer_model_projection (global)
in the canonical element map and alias map. The PLE element count was
added to total_quantizable in a prior commit but skip-module matching
against names like model.layers.0.per_layer_input_gate produced 0-byte
delta. Layer aggregate text.layers.<i> now sums all layer modules so
prefix skip names cover the PLE pieces too.
- _targets_all_linear coerces a bare string `"all-linear"` to `["all-linear"]`
before set comparison; the previous set comprehension iterated chars.
PEFT LoraConfig.target_modules accepts the bare-string convention.
- ModelArchConfig gains `shared_expert_intermediate_size`. extract_arch_config
reads `n_shared_experts` / `num_shared_experts` aliases and infers
`n_shared_experts=1` when only `shared_expert_intermediate_size` is set.
_compute_moe_mlp_elements and the structured + non-structured LoRA paths
size the shared expert with its own intermediate (Qwen3.5-MoE: 512 vs
routed moe_intermediate_size).
- _determine_attention_impl_for_gpu_estimate uses copy.deepcopy so the
resolver does not mutate nested text_config on the cached source.
PreTrainedConfig._attn_implementation setter walks `sub_configs` and the
prior shallow copy still touched the inner objects.
* Studio: extend MoE/PLE/KV-share accounting to activation and skip-alias paths
Five activation-path corrections plus two LoRA / skip-alias corrections so
that shared-expert, per-layer-input, and KV-shared-layer support is symmetric
across weights, LoRA, skip-quantizable, and activation paths.
- _layer_qkv_mlp_sizes: include shared-expert FFN in mlp_size (live shared
expert per token alongside routed experts) and keep K/V activation memory
for KV-shared layers; only the WEIGHT path uses has_k/has_v from
_layer_attention_dims.
- _per_layer_activation_bytes / compute_activation_bytes: account for
per_layer_input_gate (hd-sized) and per_layer_projection (pli-sized) per
layer plus the global per_layer_model_projection [B,S,L,PLI] tensor when
hidden_size_per_layer_input is set.
- _build_text_module_elements: split mlp.experts into routed and
mlp.shared_expert canonical entries; register layers.<i>.experts alias for
Gemma4 enable_moe_block layouts and mlp.shared_experts (plural) alias for
Exaone-MoE / Laguna / GLM4-MoE-Lite shared-expert variants.
- _compute_moe_mlp_elements: split into _compute_routed_moe_elements and
_compute_shared_moe_elements; only count shared_expert_gate (hd->1 Linear
per shared expert) when shared_expert_intermediate_size is set, which is
the Qwen2-MoE / Qwen3.5-MoE discriminator. Other shared-expert families
(Exaone-MoE, HY-V3, GLM4-MoE-Lite, Laguna) lack the gate.
- compute_lora_params: when target_modules='all-linear' bare keyword, drop
routed and shared MoE expert LoRA contributions. PEFT's all-linear targets
nn.Linear only; Unsloth's get_moe_target_parameters expands MoE expert
nn.Parameter LoRA only when target_modules contains explicit
gate_proj/up_proj/down_proj/gate_up_proj names.
- _per_layer_input_lora_params: thread target_modules through and add the
per-PLE-module contribution when the corresponding name appears, not only
under all-linear.
* Studio: top-k MoE activations, ERNIE list configs, suffix skips, multimodal full bytes
Six estimator corrections aligning the detailed accounting paths with real
training behavior:
- _layer_qkv_mlp_sizes scales the MoE-layer mlp_size by num_experts_per_tok
so the active routed-expert intermediate tensors are charged for activations.
Adds num_experts_per_tok to ModelArchConfig and extracts it from
num_experts_per_tok / top_k_experts (Gemma4 alias) in extract_arch_config.
- compute_lora_params splits routed and shared MoE LoRA contributions so that
bare target_modules='all-linear' zeroes routed (nn.Parameter expert tensors,
which Unsloth's get_moe_target_parameters does NOT enable for the bare
keyword) but keeps shared-expert LoRA (regular nn.Linear MLPs that
Unsloth's get_peft_regex DOES match).
- extract_arch_config gains a _first_scalar helper for ERNIE-style
moe_intermediate_size = [routed, shared] lists, plus moe_num_experts and
moe_num_shared_experts attribute aliases. When moe_intermediate_size is a
pair and shared_expert_intermediate_size is unset, the second element is
treated as the shared-expert intermediate.
- estimate_required_model_memory_gb's detailed branch retains
max(0, model_size_bytes - compute_total_params(arch) * 2) on top of the
arch-derived breakdown.model_weights so multimodal models (vision/audio
towers) and partially-modeled families (Gemma3n AltUp/Laurel etc.) do not
silently drop bytes that the safetensors total includes.
- _module_path_matches accepts a tail-only match when the skip entry is
shorter than the alias path. Transformers' BNB quantizer suffix-matches
short skip entries like ['q_proj'] / ['lm_head'] against full module
paths; the previous len(skip) < len(alias) early-return missed those.
- _per_layer_input_lora_params drops the all_linear branch and only counts
PLE LoRA when the user explicitly names per_layer_input_gate /
per_layer_projection / per_layer_model_projection. Unsloth's
get_peft_regex requires module names to contain a component tag
(mlp/attn/...); PLE module names lack any tag, so all-linear training
does not attach LoRA to them.
* Studio: full-FT extra optimizer/gradient inflation, MoE top-k aliases, ERNIE position dispatch, sibling experts aggregate
When the safetensors total exceeds the text-arch fp16 estimate (multimodal
vision/audio towers, partially-modeled families), only inflate the model
weights line for adapter methods but extend optimizer + gradient bytes
under full fine-tuning, where the extra params are trainable.
DBRX exposes top-k routing as moe_top_k and Hunyuan-V1-MoE as moe_topk;
neither is aliased to num_experts_per_tok via attribute_map, so probe both
when extracting arch config.
ERNIE 4.5 MoE / VL MoE configs declare MoE layers via
moe_layer_start_index / moe_layer_end_index / moe_layer_interval (with -1
meaning the last layer); add the position-style dispatch alongside the
existing mlp_layer_types / first_k_dense_replace / decoder_sparse_step
paths.
When moe_has_dense_mlp is set (Gemma4 enable_moe_block) the routed experts
live as a sibling of self.mlp at layers.<i>.experts in the actual model
layout; keep the layer mlp aggregate to the dense path and add a separate
experts aggregate so a skip module model.layers.<i>.mlp does not collapse
the routed experts as well.
* Studio: extend MoE family extraction (Llama4 / DBRX / Hunyuan / ERNIE) and align dense vs routed MLP widths
- Llama4: pick up `config.moe_layers` (auto-populated from
interleave_moe_layer_step) so dense layer indices reflect the actual
is_moe_layer dispatch.
- Llama4: add a separate `dense_intermediate_size` derived from
`intermediate_size_mlp` (used for the dense feed_forward path) and keep
`intermediate_size` for the routed/shared expert width. Auto-attach one
shared expert per MoE layer when the dense-vs-MoE width split is present.
- DBRX: walk the `ffn_config` sub-config when extracting MoE attrs
(moe_num_experts / moe_top_k / ffn_hidden_size). Without this DBRX is
misclassified as a dense arch.
- Hunyuan: normalize layer-wise `moe_topk` (and the canonical
`num_experts_per_tok` lookup it shadows via attribute_map) through a
worst-case scalar so the int(...) cast cannot crash on list values.
- ERNIE 4.5 MoE: switch the start/end/interval dispatch to the model's
`(layer_idx + 1) % interval == 0` modulo gate so MoE layers match the
decoder when interval > 1.
- ERNIE 4.5 VL MoE: drop the heuristic that read
`moe_intermediate_size[1]` as the shared expert width; in VL configs [1]
is the vision-routed width and shared experts are sized from [0].
- estimate_fp16_model_size_bytes: prefer the larger of config-derived and
local-weight bytes so the multimodal extra_bytes correction can fire
for local VLM directories.
* Add tests for VRAM estimator extensions
* Studio: trim verbose comments in VRAM estimator
Collapse multi-paragraph rationale blocks to 1-3 lines stating the single
load-bearing fact. Fix one inverted "fall through ... last" comment whose
claim disagreed with the surrounding code.
* Consolidate added tests into existing test_vram_estimation.py and test_gpu_selection.py
Move Llama4 / DBRX / ERNIE arch-extraction tests into test_vram_estimation.py
as TestLlama4ArchExtraction / TestDbrxFfnConfigExtraction /
TestErniePhaseModuloDispatch / TestErnieVlSharedExpertWidth classes. Move
estimate_fp16_model_size_bytes prefer-larger-of-config-or-local tests into
test_gpu_selection.py as TestEstimateFp16ModelSizeBytesPrefersLocalWeights.
Drop one redundant Llama4 num_dense_layers assertion already covered by the
moe_layers dispatch test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
* Studio: fix 4 failing studio_unit_tests on main
Three of the failing tests had drifted from production:
1. test_health_response_reports_desktop_capability_fields stubbed
`routes` with a SimpleNamespace that omitted `inference_studio_router`,
so importing studio.backend.main raised ImportError. Add the missing
router stub.
2. test_local_recipe_token_preserves_desktop_marker and
test_local_recipe_token_keeps_web_marker_absent decoded the local
provider's api_key as a JWT, but _inject_local_providers now mints
a unified sk-unsloth-* internal API key (not a forwarded JWT), so
jwt.decode raised "Not enough segments". Renamed and rewrote both
tests to validate the API-key contract: starts with
storage.API_KEY_PREFIX and authenticates via get_current_subject as
the real admin user. The web vs desktop distinction is irrelevant
at this layer because the unified API-key path does not carry
session flags.
The fourth failure was a real production bug:
3. test_github_validate_skips_live_access_with_honest_note expected
github-seed validation to return valid=True per
_GITHUB_VALIDATE_NOTE ("GitHub access and rate limits are checked
when the run starts"). The validate route called
build_config_builder which lazy-imports the optional data_designer
module; when it is missing, the bare except blocked the recipe.
Catch ImportError specifically and treat it as a deferred check,
matching the documented intent.
Verified all 4 tests pass and the rest of studio/backend/tests still
pass (608 total, with the only remaining failures being environment
specific: 4 GPU-aware tests on a no-GPU host and 1 Anthropic-API
smoke test, both unrelated).
* Studio: fix 3 test_gpu_selection route tests after load_model signature change
`routes/inference.load_model` gained a `fastapi_request: Request`
positional argument (used to read `app.state.llama_parallel_slots`
inside the GGUF path), but the three TestRouteErrors cases that
exercise the early validation path were not updated and failed with
`TypeError: load_model() missing 1 required positional argument:
'fastapi_request'`.
Pass a SimpleNamespace mock that satisfies the attribute path the
production code reads. The validation under test fires before the
mock is consumed, but supplying the realistic shape protects against
regressions if the validation order changes.
Affected tests:
- test_inference_route_rejects_gpu_ids_for_gguf
- test_inference_route_returns_400_for_invalid_gpu_ids
- test_inference_route_returns_400_for_uuid_parent_visibility_gpu_ids
* Studio: address review feedback on validate.py ImportError handling
Two reviewers flagged the ImportError bypass added in b0d33cf:
- chatgpt-codex-connector[bot]: catching bare ImportError marks recipes
as valid even when build_config_builder fails for unrelated import
problems (broken internal imports, missing transitive deps after a
version bump), hiding real regressions until run start.
- gemini-code-assist[bot]: silent pass discourages troubleshooting;
the deferred-validation case should be logged at debug level.
Tighten the bypass to ModuleNotFoundError where the missing module name
starts with "data_designer". Other ImportErrors propagate to the outer
handler and surface as validation failures, restoring the visibility
the reviewers asked for. Add a debug-level log entry that names the
missing module so operators can trace why validation deferred.
* Revert to balanced for inference
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove unused for_inference parameter from get_device_map
Since inference and training both use "balanced" now, the for_inference
flag is dead code. Remove it from the function signature, the call site
in inference.py, and simplify the tests accordingly.
* Remove redundant TestDeviceMapForInference test class
TestGpuAutoSelection already covers the same multi-gpu and single-gpu
device_map assertions. The TestDeviceMapForInference class was left
over from when for_inference had distinct behavior.
* Remove redundant test_get_device_map_multi_gpu_uses_balanced
Its assertions ([0,1] -> balanced, [0] -> sequential) are already
covered by test_get_device_map_uses_explicit_gpu_selection.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* [WIP] balanced device map for studio
* gpus as a request parameter
* API for multi GPU stuff
* return multi gpu util in new API
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use balanced_low0 instead of balanced
* Use balanced_low0 instead of balanced
* Fix device_map typo, UUID parsing crash, set() filter bug, and broken tests
- balanced_low0 -> balanced_low_0 (transformers/accelerate rejects the old string)
- get_parent_visible_gpu_ids() now handles UUID/MIG CUDA_VISIBLE_DEVICES
gracefully instead of crashing on int() parse
- _get_backend_visible_gpu_info() set() or None bug: empty set is falsy so
CUDA_VISIBLE_DEVICES=-1 would disable filtering and report all GPUs
- test_gpu_selection.py: add missing get_visible_gpu_utilization import and
add required job_id arg to start_training() calls
* Smart GPU determinism using estimates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* disallow gpu selection for gguf for now
* cleanup
* Slightly larger baseline
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Treat empty list as auto
* Verbose logging/debug
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Cleanup and revert unnecessary deletions
* Cleanup excessive logs and guard against disk/cpu offload
* auth for visibility API. cleanup redundant imports. Adjust QLoRA estimate
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* support for non cuda gpus
* Fix multi-GPU auto-selection memory accounting
The multi_gpu_factor was applied uniformly to all GPUs including the
first one, which unfairly penalizes single-GPU capacity when
transitioning to multi-GPU. This created a discontinuity where a model
that barely fits 1 GPU would suddenly require 2 GPUs because the first
GPU's free memory was discounted by 20%.
Now the first GPU keeps its full free memory, and only additional GPUs
have an overhead factor (0.85) applied to account for inter-GPU
communication and sharding overhead. This gives more accurate
auto-selection and avoids unnecessary multi-GPU for models that
comfortably fit on one device.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add sandbox tests for multi-GPU selection logic
24 tests covering model size estimation, memory requirements, automatic
GPU selection, device map generation, GPU ID validation, and multi-GPU
overhead accounting. All tests use mocks so they run without GPUs on
Linux, macOS, and Windows.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix reviewer findings: 4bit inference estimate, fallback, GGUF gpu_ids, retry
1. 4-bit inference now uses reduced memory estimate (model_size/3 + buffer)
instead of the FP16 1.3x multiplier. This prevents over-sharding
quantized models across unnecessary GPUs.
2. When model size estimation fails, auto_select_gpu_ids now falls back to
all visible GPUs instead of returning None (which could default to
single-GPU loading for an unknown-size model).
3. GGUF inference route now treats gpu_ids=[] as auto-selection (same as
None) instead of rejecting it as an unsupported explicit request.
4. Training retry path for "could not get source code" now preserves the
gpu_ids parameter so the retry lands on the same GPUs.
5. Updated sandbox tests to cover the new 4-bit inference estimate branch.
* Remove accidentally added unsloth-zoo submodule
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix UUID/MIG visibility and update test expectations
1. nvidia.py: When CUDA_VISIBLE_DEVICES uses UUID/MIG tokens, the
visibility APIs now return "unresolved" with empty device lists instead
of exposing all physical GPUs. This prevents the UI from showing GPUs
that the backend process cannot actually use.
2. test_gpu_selection.py: Updated test expectations to match the new
multi-GPU overhead accounting (first GPU at full capacity, 0.85x for
additional GPUs) and 4-bit inference memory estimation formula.
All 60 tests now pass.
* Add CPU/disk offload guard to audio inference path
The audio model loading branch returned before the common
get_offloaded_device_map_entries() check, so audio models loaded with a
multi-GPU device_map that spilled layers to CPU/disk would be accepted
instead of rejected. Now audio loads also verify no modules are offloaded.
* Improve VRAM requirement estimates
* Replace balanced_low_0 with balanced
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* refine calculations for slightly easier nums
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* adjust estimates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use nums instead of obj to avoid seralisation error
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden nvidia-smi parsing and fix fallback GPU list
1. nvidia.py: Wrap int() casts for GPU index and memory in try/except
so MIG slices, N/A values, or unexpected nvidia-smi output skip the
unparseable row instead of aborting the entire GPU list.
2. nvidia.py: Handle GPU names containing commas by using the last
field as memory instead of a fixed positional index.
3. hardware.py: fallback_all now uses gpu_candidates (GPUs with verified
VRAM data) instead of raw devices list, which could include GPUs
with null VRAM that were excluded from the ranking.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* cleanup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* consolidate raise_if_offload
* Improve MoE support. Guard against nvidia-smi failures
* Improve MoE support. Guard against nvidia-smi failures
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix shared-expert LoRA undercount, torch VRAM fallback, and apply_gpu_ids edge case
1. vram_estimation.py: compute_lora_params now includes shared experts
(n_shared_experts) alongside routed experts when computing MoE LoRA
adapter parameters. Previously only n_experts were counted, causing
the estimator to undercount adapter, optimizer, and gradient memory
for DeepSeek/GLM-style models with shared experts.
2. hardware.py: _torch_get_per_device_info now uses mem_get_info (which
reports system-wide VRAM usage) instead of memory_allocated (which
only reports this process's PyTorch allocations). This prevents
auto-selection from treating a GPU as mostly free when another
process is consuming VRAM. Falls back to memory_allocated when
mem_get_info is unavailable.
3. hardware.py: apply_gpu_ids([]) now returns early instead of setting
CUDA_VISIBLE_DEVICES="" which would disable CUDA entirely. Empty
list inherits the parent visibility, same as None.
4. hardware.py: Upgraded fallback_all GPU selection log from debug to
warning so operators are notified when the model likely will not fit
in available VRAM.
* Guard nvidia-smi subprocess calls against OSError and TimeoutExpired
get_visible_gpu_utilization and get_backend_visible_gpu_info now catch
OSError (nvidia-smi not found) and TimeoutExpired internally instead
of relying on callers to wrap every invocation. Returns the standard
available=False sentinel on failure so the torch-based fallback in
hardware.py can take over.
* Guard get_primary_gpu_utilization and reset GPU caches between tests
1. nvidia.py: get_primary_gpu_utilization now catches OSError and
TimeoutExpired internally, matching the pattern already used in
get_visible_gpu_utilization and get_backend_visible_gpu_info. All
three nvidia-smi callers are now self-contained.
2. test_gpu_selection.py: Added _GpuCacheResetMixin that resets the
module-level _physical_gpu_count and _visible_gpu_count caches in
tearDown. Applied to all test classes that exercise GPU selection,
device map, or visibility functions. This prevents stale cache
values from leaking between tests and causing flaky results on
machines with real GPUs.
* Fix nvidia-smi fallback regression and physical GPU count validation
1. hardware.py: get_gpu_utilization, get_visible_gpu_utilization, and
get_backend_visible_gpu_info now check result.get("available") before
returning the nvidia-smi result. When nvidia-smi is unavailable or
returns no data (e.g., containers without nvidia-smi, UUID/MIG masks),
the functions fall through to the torch-based fallback instead of
returning an empty result. This fixes a regression where the internal
exception handling in nvidia.py prevented the caller's except block
from triggering the fallback.
2. hardware.py: resolve_requested_gpu_ids now separates negative-ID
validation from physical upper-bound validation. The physical count
check is only enforced when it is plausibly a true physical count
(i.e., higher than the largest parent-visible ID), since
torch.cuda.device_count() under CUDA_VISIBLE_DEVICES returns the
visible count, not the physical total. The parent-visible-set check
remains authoritative in all cases. This prevents valid physical IDs
like [2, 3] from being rejected as "out of range" when nvidia-smi is
unavailable and CUDA_VISIBLE_DEVICES="2,3" makes torch report only
2 devices.
* Fix UUID/MIG torch fallback to enumerate devices by ordinal
When CUDA_VISIBLE_DEVICES uses UUID or MIG identifiers,
get_parent_visible_gpu_ids() returns [] because the tokens are
non-numeric. The torch fallback in get_visible_gpu_utilization() and
get_backend_visible_gpu_info() previously passed that empty list to
_torch_get_per_device_info(), getting nothing back.
Now both functions detect the empty-list case and fall back to
enumerating torch-visible ordinals (0..device_count-1) with
index_kind="relative". This means the UI and auto-selection still
see real device data in Kubernetes, MIG, and Slurm-style UUID
environments where nvidia-smi output cannot be mapped to physical
indices.
Updated test_uuid_parent_visibility to verify the new torch fallback
path returns available=True with relative ordinals.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add type hint for gpu_ids parameter in InferenceOrchestrator.load_model
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Daniel Han