ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Updated Home (markdown)

Daniel Han 2024-11-01 12:12:17 -07:00
commit 663e67cf67
1 changed files with 14 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

27
Home.md

@ -1008,19 +1008,20 @@ We value engineers who are proactive, independent and who ship features and idea
**!! On our criteria for being considered for an internship / full time role !!**
1. Debug, solve urgent issues / bugs and make 3 merged PRs for interns / 6 merged PRs for a full time engineer
2. **OR** Create a high quality accepted PR on 1 item below (only 1 needed):
* Somewhat optimized QLoRA finetuning for FLUX / stable diffusion models. Diffusers just added 4bit QLoRA support - make this faster. Provide a Colab notebook tutorial on how to use it. Must fit in 1 x 16GB Tesla T4.
* Unoptimized Apple Silicon / Metal support LoRA - MLX, Core ML Tools etc support.
* Utilities to export Unsloth finetunes to vLLM, SGLang & Ollama - LoRA adapters only. Provide a standalone serving interface to vLLM, SGLang
* Add float8 + QLoRA finetuning support via Torch AO into Unsloth.
* Bitsandbytes 4bit QLoRA dequant Triton kernel - must be faster than CUDA version.
* Add TPU (maybe JAX?) & AMD support into Unsloth. TPU - Colab & Kaggle notebooks. AMD - Runpod equivalent.
* Use torch AO and add MXFP4 support in preparation for Blackwell. Show experiments on loss curves matching.
* Add fully optimized Deepseek finetuning support - investigate [Scatter MoE](https://github.com/shawntan/scattermoe) - confirm loss curves match.
* Add full finetuning / pretraining support in Unsloth - Triton kernels for all.
* FSDPv2 + QLoRA (maybe via Torch AO) + `torch.compile`. Investigate PyTorch native Pipeline, Sequence & Tensor parallelism.
* Using spot instances to train models with a checkpoint recovery mechanism (like SkyPilot)
* Use binary tensor cores for fast Hamming distances. Show this works in approximate nearest neighbors
2. **OR** Create a high quality accepted PR on 1 item below. 📚=Software Eng 🔢=Kernels 🛠️=Infra:
* 📚Somewhat optimized QLoRA finetuning for FLUX / stable diffusion models. Diffusers just added 4bit QLoRA support - make this faster. Provide a Colab notebook tutorial on how to use it. Must fit in 1 x 16GB Tesla T4.
* 📚Unoptimized Apple Silicon / Metal support LoRA - MLX, Core ML Tools etc support.
* 📚Utilities to export Unsloth finetunes to vLLM, SGLang & Ollama - LoRA adapters only. Provide a standalone serving interface to vLLM, SGLang
* 🔢Add float8 + QLoRA finetuning support via Torch AO into Unsloth.
* 🔢Bitsandbytes 4bit QLoRA dequant Triton kernel - must be faster than CUDA version.
* 📚Add TPU (maybe JAX?) & AMD support into Unsloth. TPU - Colab & Kaggle notebooks. AMD - Runpod equivalent.
* 🔢Use torch AO and add MXFP4 support in preparation for Blackwell. Show experiments on loss curves matching.
* 🔢Add fully optimized Deepseek finetuning support - investigate [Scatter MoE](https://github.com/shawntan/scattermoe) - confirm loss curves match.
* 🔢Add full finetuning / pretraining support in Unsloth - Triton kernels for all.
* 🔢FSDPv2 + QLoRA (maybe via Torch AO) + `torch.compile`. Investigate PyTorch native Pipeline, Sequence & Tensor parallelism.
* 🛠️Using spot instances to train models with a checkpoint recovery mechanism (like SkyPilot)
* 🛠️Modal but using spot instances
* 🔢Use binary tensor cores for fast Hamming distances. Show this works in approximate nearest neighbors
**For for details, email me, or ask me questions on Discord! For more information about us, see our:**
1. CUDA / GPU Mode lecture Talk. [Youtube link](https://www.youtube.com/watch?v=hfb_AIhDYnA)