Updated Home (markdown)
parent
cb5e68a4e7
commit
4aae8a678c
1 changed files with 15 additions and 0 deletions
15
Home.md
15
Home.md
|
|
@ -167,6 +167,21 @@ tokenizer = get_chat_template(
|
|||
)
|
||||
```
|
||||
|
||||
### 2x Faster Inference
|
||||
Unsloth supports natively 2x faster inference. All QLoRA, LoRA and non LoRA inference paths are 2x faster. This requires no change of code or any new dependencies.
|
||||
```python
|
||||
from unsloth import FastLanguageModel
|
||||
model, tokenizer = FastLanguageModel.from_pretrained(
|
||||
model_name = "lora_model", # YOUR MODEL YOU USED FOR TRAINING
|
||||
max_seq_length = max_seq_length,
|
||||
dtype = dtype,
|
||||
load_in_4bit = load_in_4bit,
|
||||
)
|
||||
FastLanguageModel.for_inference(model) # Enable native 2x faster inference
|
||||
text_streamer = TextStreamer(tokenizer)
|
||||
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 64)
|
||||
```
|
||||
|
||||
### NotImplementedError: A UTF-8 locale is required. Got ANSI
|
||||
See https://github.com/googlecolab/colabtools/issues/3409
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue