From 663e67cf67efb00577a2904d5d16177150b5b387 Mon Sep 17 00:00:00 2001 From: Daniel Han Date: Fri, 1 Nov 2024 12:12:17 -0700 Subject: [PATCH] Updated Home (markdown) --- Home.md | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/Home.md b/Home.md index cd23cbf..2e65681 100644 --- a/Home.md +++ b/Home.md @@ -1008,19 +1008,20 @@ We value engineers who are proactive, independent and who ship features and idea **!! On our criteria for being considered for an internship / full time role !!** 1. Debug, solve urgent issues / bugs and make 3 merged PRs for interns / 6 merged PRs for a full time engineer -2. **OR** Create a high quality accepted PR on 1 item below (only 1 needed): - * Somewhat optimized QLoRA finetuning for FLUX / stable diffusion models. Diffusers just added 4bit QLoRA support - make this faster. Provide a Colab notebook tutorial on how to use it. Must fit in 1 x 16GB Tesla T4. - * Unoptimized Apple Silicon / Metal support LoRA - MLX, Core ML Tools etc support. - * Utilities to export Unsloth finetunes to vLLM, SGLang & Ollama - LoRA adapters only. Provide a standalone serving interface to vLLM, SGLang - * Add float8 + QLoRA finetuning support via Torch AO into Unsloth. - * Bitsandbytes 4bit QLoRA dequant Triton kernel - must be faster than CUDA version. - * Add TPU (maybe JAX?) & AMD support into Unsloth. TPU - Colab & Kaggle notebooks. AMD - Runpod equivalent. - * Use torch AO and add MXFP4 support in preparation for Blackwell. Show experiments on loss curves matching. - * Add fully optimized Deepseek finetuning support - investigate [Scatter MoE](https://github.com/shawntan/scattermoe) - confirm loss curves match. - * Add full finetuning / pretraining support in Unsloth - Triton kernels for all. - * FSDPv2 + QLoRA (maybe via Torch AO) + `torch.compile`. Investigate PyTorch native Pipeline, Sequence & Tensor parallelism. - * Using spot instances to train models with a checkpoint recovery mechanism (like SkyPilot) - * Use binary tensor cores for fast Hamming distances. Show this works in approximate nearest neighbors +2. **OR** Create a high quality accepted PR on 1 item below. 📚=Software Eng 🔢=Kernels 🛠️=Infra: + * 📚Somewhat optimized QLoRA finetuning for FLUX / stable diffusion models. Diffusers just added 4bit QLoRA support - make this faster. Provide a Colab notebook tutorial on how to use it. Must fit in 1 x 16GB Tesla T4. + * 📚Unoptimized Apple Silicon / Metal support LoRA - MLX, Core ML Tools etc support. + * 📚Utilities to export Unsloth finetunes to vLLM, SGLang & Ollama - LoRA adapters only. Provide a standalone serving interface to vLLM, SGLang + * 🔢Add float8 + QLoRA finetuning support via Torch AO into Unsloth. + * 🔢Bitsandbytes 4bit QLoRA dequant Triton kernel - must be faster than CUDA version. + * 📚Add TPU (maybe JAX?) & AMD support into Unsloth. TPU - Colab & Kaggle notebooks. AMD - Runpod equivalent. + * 🔢Use torch AO and add MXFP4 support in preparation for Blackwell. Show experiments on loss curves matching. + * 🔢Add fully optimized Deepseek finetuning support - investigate [Scatter MoE](https://github.com/shawntan/scattermoe) - confirm loss curves match. + * 🔢Add full finetuning / pretraining support in Unsloth - Triton kernels for all. + * 🔢FSDPv2 + QLoRA (maybe via Torch AO) + `torch.compile`. Investigate PyTorch native Pipeline, Sequence & Tensor parallelism. + * 🛠️Using spot instances to train models with a checkpoint recovery mechanism (like SkyPilot) + * 🛠️Modal but using spot instances + * 🔢Use binary tensor cores for fast Hamming distances. Show this works in approximate nearest neighbors **For for details, email me, or ask me questions on Discord! For more information about us, see our:** 1. CUDA / GPU Mode lecture Talk. [Youtube link](https://www.youtube.com/watch?v=hfb_AIhDYnA)