092fc91e08
The dense-quant re-plan passes transformer_resident_override_mib (the bf16 build peak) AND computes companions via _companion_cache_bytes(base), which sums every flat blob in the HF cache. Because the dense path prefetches the base transformer/ shards into that same cache before load_pipeline runs, the transformer is counted twice, inflating the footprint (~44 GB instead of ~20 GB in the reproduction) and wrongly forcing offload for models that fit resident -- the case this path exists to enable. Add companion_override_mib and pass the auto-policy's own text-encoder plus VAE estimate on the re-plan so the cache (with its prefetched transformer) is not read for this artifact. |
||
|---|---|---|
| .. | ||
| data_recipe | ||
| export | ||
| inference | ||
| rag | ||
| training | ||
| __init__.py | ||
| _torchao_stub.py | ||
| import_guards.py | ||
| tool_healing.py | ||