Add MLX-aware public Unsloth trainer API (#6462)
* feat: add mlx public trainer api * test: cover mlx public trainer api * fix: preserve mlx epoch trainer configs * fix: pass mlx warmup ratio through config * fix: align mlx trainer dataset order * fix: keep mlx chat templates import-light * fix: infer mlx trainer context length * fix: mirror cuda mlx context defaults * fix: align mlx notebook trainer defaults * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: keep mlx public helpers import-light * refactor: reuse mlx optimizer normalization * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: address mlx review feedback * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: tighten mlx training argument parity * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: align mlx trainer eos default * Fix MLX trainer to accept DataCollatorForSeq2Seq and handle TokenizerWrapper in get_chat_template * Trim redundant docstrings on internal MLX helpers * MLX review fixes: Studio optimizer import-safe on non-MLX hosts, preserve explicit max_length, skip MLX tests before import * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * MLX review round 2: defer max_length to model context, optimizer alias fallback for older zoo, skip non-MLX test on missing GPU deps * MLX review round 3: keep chat_templates importable without torch on MLX * fix: preserve MLX trainer notebook shims * fix: ignore CUDA tokenizer moves on MLX * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: harden MLX trainer shims * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: unwrap MLX scheduler enum args * fix: coerce integral MLX epoch counts * fix: spoof CUDA compatibility APIs on MLX * fix: harden MLX notebook compatibility shims * MLX: add torch.cuda.mem_get_info to the compatibility shim Notebook memory cells call torch.cuda.mem_get_info()[0] directly (not gated by is_available), so on MLX it raises without a shim. Return (free, total) bytes from the MLX device stats, consistent with the other torch.cuda compat helpers, and add a matching assertion to the compat-API test. * MLX: use active memory for mem_get_info; fix BatchEncoding.to keyword device Address review on the MLX compatibility shim: - torch.cuda.mem_get_info() now derives free bytes from current active MLX memory instead of the peak high-water mark, so a capacity check stays accurate after a transient spike or a prior run. - BatchEncoding.to(device=...) passed by keyword no longer forwards a positional None alongside the keyword (which raised "multiple values for 'device'"), so non-CUDA keyword moves like .to(device="cpu") delegate correctly. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * MLX: accept preserve_dataset_order; stub RL trainers with a clear error Two fixes so unmigrated notebooks behave predictably on MLX (torch present): - preserve_dataset_order is a real MLXTrainingConfig field but was missing from the extra-argument allowlist, so passing it (as a config or trainer kwarg) could be rejected as unknown on a zoo without the field. Add it to _MLX_IMPLEMENTED_EXTRA_ARGUMENTS so the documented no-shuffle path is reachable. - GRPO/DPO/ORPO (and KTO/PPO/Reward) have no MLX trainer yet. Retarget the ones the installed trl exposes to a stub that raises a clear 'not supported on MLX' error instead of importing the real torch/CUDA trainer and crashing deep inside it. Only existing trainers are retargeted (no invented attributes), idempotent across re-imports. * MLX: make RL-trainer stubbing import-safe; back current-memory APIs with active memory Address review on the MLX shims: - The RL-trainer stub loop probed trl with getattr(_trl, name), which triggers trl's lazy trainer import and pulls torch -- that can crash import unsloth on a torch-free MLX install just to check existence. Decide what to stub from trl.__all__ + already-materialized attrs (vars) instead; never resolve the real trainer. All trl trainer names are in __all__, so they are still stubbed (even torch-free), and the probe no longer imports torch. - torch.cuda.memory_reserved / memory_allocated (the current, non-max APIs) were aliased to peak max_memory_reserved. Back them with current active MLX memory so cleanup / capacity checks see live usage; max_* keep the peak high-water mark. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * MLX: keep TRL's SFTConfig epoch default under the trl.SFTConfig alias Unmigrated notebooks import SFTConfig from trl, which the MLX build aliases to the public training-args class. TRL/HF SFTConfig defaults to num_train_epochs=3 (max_steps=-1); the native MLX config defaults to max_steps=60. So an SFTConfig built without an explicit length silently ran 60 MLX steps instead of TRL's 3 epochs under the alias. Alias trl.SFTConfig to a thin subclass that seeds the TRL epoch default only when neither max_steps nor num_train_epochs is given; explicit lengths pass through untouched, and the native public args class keeps its MLX default. Epoch mode is supported by the MLX trainer. * MLX CI: keep the GGUF reload smoke under the job timeout The RELOAD-GGUF-via-llama-cli step timed out at 300s. BF16 GGUF decode is CPU-bound on the macOS runner (~10s+/token), so generating 24 tokens landed right on the 300s cliff and killed the process. This step is a save/reload integrity smoke (it only needs a few chars of output), so the token count is incidental: generate 8 tokens with explicit threads and a small headroom on the subprocess timeout, all env-tunable (UNSLOTH_GGUF_RELOAD_N / _THREADS / _TIMEOUT). Cuts the reload well under the 25 minute job budget. * MLX: broaden trainer stubs, real peak-memory reset, fix shim tests Address review on the MLX public API: - The SFTConfig identity tests asserted trl.SFTConfig is UnslothTrainingArguments, but the alias now points at the _MLXSFTConfig subclass that preserves TRL's epoch default, so the MLX suite failed before testing the shim. Assert issubclass instead. - torch.cuda.reset_peak_memory_stats was a no-op, so max_memory_reserved kept earlier model-load peaks across a scoped run. Wire it to mx.reset_peak_memory with the same core/metal fallback used for the reads. - The unsupported-trainer stubs were a fixed list, so trainers outside it (a newer RLOOTrainer) still routed to the real torch trainer. Derive the set from trl.__all__ (every non-SFT *Trainer) so all non-SFT surfaces fail with a clear MLX message; names come from __all__ so trl is never resolved. - The non-MLX export smoke skipped only on missing bitsandbytes/triton; other absent GPU deps (numpy/torch/unsloth-zoo, or _gpu_init re-raising ImportError) made it fail on CPU hosts. Skip on any ImportError. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: keep MLX notebook compatibility minimal * MLX CI: force CPU + small context for the GGUF reload smoke The RELOAD-GGUF-via-llama-cli step timed out even at 8 tokens (>420s), so it is a fixed hang, not per-token cost: on the paravirtual macOS runner GPU llama.cpp's Metal backend stalls, and the gemma3 GGUF advertises a 32768 context that llama-cli would otherwise fully allocate. Run llama-cli CPU-only (-ngl 0) with a small context (-c 256); keep generation short. All env-tunable (UNSLOTH_GGUF_RELOAD_NGL / _CTX / _N / _THREADS / _TIMEOUT). Also print llama.cpp's partial stdout/stderr on timeout so a future hang is diagnosable instead of an opaque TimeoutExpired. * MLX CI: export the reload-smoke GGUF as q8_0, not bf16 The GGUF reload via llama-cli timed out on the runner even CPU-only with a tiny context and 8 tokens. Root cause is the format, not the flags: the smoke exported quantization_method='not_quantized', which maps to a bf16 GGUF, and llama.cpp's bf16 CPU decode is unusably slow on the paravirtual macOS runner. Export q8_0 (fast_quantized, the exporter default and what users deploy) instead -- llama.cpp has optimized q8_0 CPU kernels, so the fresh-process reload loads and generates in seconds. The reload stays CPU-only (-ngl 0) with a small context. * test: clear TRL shim before availability check --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: imagineer99 <samleejackson0@gmail.com>
This commit is contained in:
Long Yixing
GitHub
parent
73e8245ee8
commit
d918245834
6 changed files with 2595 additions and 62 deletions
|
|
@ -1253,32 +1253,48 @@ def _adapt_for_mlx_vlm(
|
|||
return adapted
|
||||
|
||||
|
||||
_MLX_STUDIO_OPTIM_MAP = {
|
||||
"adamw_8bit": "adamw",
|
||||
"paged_adamw_8bit": "adamw",
|
||||
"adamw_bnb_8bit": "adamw",
|
||||
"paged_adamw_32bit": "adamw",
|
||||
"adamw_torch": "adamw",
|
||||
"adamw_torch_fused": "adamw",
|
||||
"adamw": "adamw",
|
||||
"adafactor": "adafactor",
|
||||
"sgd": "sgd",
|
||||
"adam": "adam",
|
||||
"muon": "muon",
|
||||
"lion": "lion",
|
||||
}
|
||||
_MLX_STUDIO_LR_SCHEDULERS = {"linear", "cosine", "constant"}
|
||||
|
||||
|
||||
# Fallback alias map mirroring unsloth_zoo._normalize_mlx_optimizer_name, used
|
||||
# only when mlx (Apple Silicon) is not importable so Studio config validation
|
||||
# still works on non-MLX hosts. The zoo function stays the source of truth.
|
||||
_MLX_STUDIO_ADAMW_ALIASES = frozenset(
|
||||
(
|
||||
"adamw_8bit",
|
||||
"paged_adamw_8bit",
|
||||
"adamw_bnb_8bit",
|
||||
"paged_adamw_32bit",
|
||||
"adamw_torch",
|
||||
"adamw_torch_fused",
|
||||
"paged_adamw",
|
||||
"adamw_32bit",
|
||||
"adamw_hf",
|
||||
"adamw_anyprecision",
|
||||
"adamw_apex_fused",
|
||||
)
|
||||
)
|
||||
_MLX_STUDIO_NATIVE_OPTIMIZERS = ("adafactor", "adamw", "adam", "sgd", "muon", "lion")
|
||||
|
||||
|
||||
def _normalize_mlx_studio_optimizer(value):
|
||||
raw = str(value or "adamw_8bit").strip().lower()
|
||||
try:
|
||||
return _MLX_STUDIO_OPTIM_MAP[raw]
|
||||
except KeyError:
|
||||
supported = ", ".join(sorted(_MLX_STUDIO_OPTIM_MAP))
|
||||
raise ValueError(
|
||||
f"Unsupported optimizer for MLX training: {value!r}. " f"Supported values: {supported}."
|
||||
)
|
||||
from unsloth_zoo.mlx.trainer import _normalize_mlx_optimizer_name
|
||||
return _normalize_mlx_optimizer_name(value or "adamw_8bit")
|
||||
except (ImportError, ValueError):
|
||||
# Missing mlx, or an older unsloth-zoo whose normalizer lacks CUDA/TRL
|
||||
# aliases: map common adamw_* names locally so notebook defaults work.
|
||||
opt = str(getattr(value, "value", value) or "adamw_8bit").strip().lower()
|
||||
opt = opt.rsplit(".", 1)[-1].replace("-", "_")
|
||||
if opt in _MLX_STUDIO_ADAMW_ALIASES:
|
||||
opt = "adamw"
|
||||
if opt not in _MLX_STUDIO_NATIVE_OPTIMIZERS:
|
||||
supported = ", ".join(_MLX_STUDIO_NATIVE_OPTIMIZERS)
|
||||
raise ValueError(
|
||||
f"Unsupported optimizer for MLX training: {value!r}. "
|
||||
f"Supported optimizers: {supported}."
|
||||
)
|
||||
return opt
|
||||
|
||||
|
||||
def _normalize_mlx_studio_scheduler(value):
|
||||
|
|
|
|||
|
|
@ -76,7 +76,7 @@ def test_mlx_studio_optimizer_aliases_are_explicit():
|
|||
|
||||
|
||||
def test_mlx_studio_rejects_unknown_optimizer():
|
||||
with pytest.raises(ValueError, match = "Unsupported optimizer for MLX training"):
|
||||
with pytest.raises(ValueError, match = "Supported"):
|
||||
_normalize_mlx_studio_optimizer("adamw_typo")
|
||||
|
||||
|
||||
|
|
|
|||
1206
tests/python/test_mlx_public_trainer_api.py
Normal file
1206
tests/python/test_mlx_public_trainer_api.py
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -403,10 +403,14 @@ def cmd_train(args) -> int:
|
|||
metrics["gguf_dir"] = str(gguf_dir)
|
||||
with Phase("save_gguf", metrics):
|
||||
try:
|
||||
# q8_0 (the exporter default), not bf16: llama.cpp has optimized q8_0
|
||||
# CPU kernels, whereas bf16 CPU decode is unusably slow on the runner
|
||||
# and made the fresh-process llama-cli reload below time out. q8_0 is
|
||||
# also what users deploy by default.
|
||||
model.save_pretrained_gguf(
|
||||
str(gguf_dir),
|
||||
tokenizer = tokenizer,
|
||||
quantization_method = "not_quantized",
|
||||
quantization_method = "fast_quantized",
|
||||
)
|
||||
gguf_files = sorted(gguf_dir.glob("*.gguf"))
|
||||
if not gguf_files:
|
||||
|
|
@ -565,31 +569,36 @@ def _reload_gguf(save_dir: Path, metrics: dict) -> int:
|
|||
raise SystemExit(f"no .gguf files in {save_dir}")
|
||||
gguf_path = gguf_files[0]
|
||||
|
||||
# This is a save/reload-integrity smoke; a few generated tokens are enough.
|
||||
# Keep llama.cpp bounded on macOS runners where BF16 GGUF decode is CPU-bound.
|
||||
# Save/reload-integrity smoke (assert below only needs a few chars). The GGUF is
|
||||
# exported q8_0 (see save_gguf) because llama.cpp bf16 CPU decode is unusably slow
|
||||
# on the runner. Run CPU-only (-ngl 0), cap the context (-c 256, the model
|
||||
# advertises 32768), and keep generation short; all env-tunable.
|
||||
n_predict = os.environ.get("UNSLOTH_GGUF_RELOAD_N", "8")
|
||||
n_threads = os.environ.get("UNSLOTH_GGUF_RELOAD_THREADS", str(os.cpu_count() or 4))
|
||||
n_ctx = os.environ.get("UNSLOTH_GGUF_RELOAD_CTX", "256")
|
||||
n_gpu_layers = os.environ.get("UNSLOTH_GGUF_RELOAD_NGL", "0")
|
||||
reload_timeout = int(os.environ.get("UNSLOTH_GGUF_RELOAD_TIMEOUT", "420"))
|
||||
|
||||
argv = [
|
||||
str(llama_cli),
|
||||
"-m",
|
||||
str(gguf_path),
|
||||
"-p",
|
||||
PROMPT,
|
||||
"-n",
|
||||
n_predict,
|
||||
"-t",
|
||||
n_threads,
|
||||
"-c",
|
||||
n_ctx,
|
||||
"-ngl",
|
||||
n_gpu_layers,
|
||||
"--temp",
|
||||
"0",
|
||||
"--seed",
|
||||
str(SEED),
|
||||
"--no-warmup",
|
||||
]
|
||||
with Phase("reload_gguf", metrics):
|
||||
argv = [
|
||||
str(llama_cli),
|
||||
"-m",
|
||||
str(gguf_path),
|
||||
"-p",
|
||||
PROMPT,
|
||||
"-n",
|
||||
n_predict,
|
||||
"-t",
|
||||
n_threads,
|
||||
"--temp",
|
||||
"0",
|
||||
"--seed",
|
||||
str(SEED),
|
||||
"-c",
|
||||
"256",
|
||||
"--no-warmup",
|
||||
]
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
argv,
|
||||
|
|
@ -606,6 +615,7 @@ def _reload_gguf(save_dir: Path, metrics: dict) -> int:
|
|||
return stream.decode("utf-8", errors = "replace")
|
||||
return stream or ""
|
||||
|
||||
print(f" [reload:gguf] TIMEOUT running: {' '.join(argv)}", flush = True)
|
||||
print(f" [reload:gguf] TIMEOUT stdout:\n{_decode(exc.stdout)[:1000]}", flush = True)
|
||||
print(f" [reload:gguf] TIMEOUT stderr:\n{_decode(exc.stderr)[:1000]}", flush = True)
|
||||
raise
|
||||
|
|
|
|||
1270
unsloth/__init__.py
1270
unsloth/__init__.py
File diff suppressed because it is too large
Load diff
|
|
@ -27,18 +27,25 @@ __all__ = [
|
|||
"test_construct_chat_template",
|
||||
]
|
||||
|
||||
from transformers import StoppingCriteria, StoppingCriteriaList
|
||||
from torch import LongTensor, FloatTensor
|
||||
from transformers.models.llama.modeling_llama import logger
|
||||
from transformers.utils import logging
|
||||
try:
|
||||
from torch import LongTensor, FloatTensor
|
||||
except ImportError:
|
||||
LongTensor = FloatTensor = None
|
||||
logger = logging.get_logger(__name__)
|
||||
import os
|
||||
import shutil
|
||||
from .tokenizer_utils import *
|
||||
import re
|
||||
from .ollama_template_mappers import OLLAMA_TEMPLATES
|
||||
from unsloth_zoo.dataset_utils import (
|
||||
train_on_responses_only,
|
||||
standardize_data_formats,
|
||||
)
|
||||
try:
|
||||
from unsloth_zoo.dataset_utils import (
|
||||
train_on_responses_only,
|
||||
standardize_data_formats,
|
||||
)
|
||||
except ImportError:
|
||||
# dataset_utils pulls torch; keep chat_templates importable on torch-free
|
||||
# (MLX) hosts, which expose these via the backend-specific wrappers instead.
|
||||
train_on_responses_only = standardize_data_formats = None
|
||||
standardize_sharegpt = standardize_data_formats
|
||||
CHAT_TEMPLATES = {}
|
||||
DEFAULT_SYSTEM_MESSAGE = {}
|
||||
|
|
@ -1838,11 +1845,24 @@ def get_chat_template(
|
|||
map_eos_token = True,
|
||||
system_message = None,
|
||||
patch_saving = True,
|
||||
use_zoo_tokenizer_patch = False,
|
||||
use_zoo_tokenizer_patch = None,
|
||||
):
|
||||
assert(type(map_eos_token) is bool)
|
||||
import sys
|
||||
is_mlx_backend = getattr(sys.modules.get("unsloth"), "DEVICE_TYPE", None) == "mlx"
|
||||
if use_zoo_tokenizer_patch is None:
|
||||
use_zoo_tokenizer_patch = is_mlx_backend
|
||||
old_tokenizer = tokenizer
|
||||
|
||||
# mlx-lm's TokenizerWrapper._tokenizer is the HF tokenizer, not the Rust
|
||||
# backend the vocab-edit paths below need; unwrap here, re-wrap before return.
|
||||
_mlx_tokenizer_wrapper = None
|
||||
if is_mlx_backend and tokenizer.__class__.__name__ == "TokenizerWrapper":
|
||||
_inner_tokenizer = getattr(tokenizer, "_tokenizer", None)
|
||||
if _inner_tokenizer is not None and hasattr(_inner_tokenizer, "is_fast"):
|
||||
_mlx_tokenizer_wrapper = tokenizer
|
||||
tokenizer = _inner_tokenizer
|
||||
|
||||
IS_GEMMA = False
|
||||
if tokenizer.__class__.__name__.startswith("Gemma"):
|
||||
if chat_template == "chatml": chat_template = "gemma_chatml"
|
||||
|
|
@ -1952,6 +1972,7 @@ def get_chat_template(
|
|||
pass
|
||||
|
||||
# Must fix the sentence piece tokenizer since there's no tokenizer.model file!
|
||||
from .tokenizer_utils import fix_sentencepiece_tokenizer
|
||||
tokenizer = fix_sentencepiece_tokenizer(tokenizer, new_tokenizer, token_mapping,)
|
||||
else:
|
||||
pass
|
||||
|
|
@ -1997,6 +2018,7 @@ def get_chat_template(
|
|||
|
||||
# Must fix the sentence piece tokenizer since there's no tokenizer.model file!
|
||||
token_mapping = { old_eos_token : stop_word, }
|
||||
from .tokenizer_utils import fix_sentencepiece_tokenizer
|
||||
tokenizer = fix_sentencepiece_tokenizer(tokenizer, new_tokenizer, token_mapping,)
|
||||
pass
|
||||
|
||||
|
|
@ -2057,13 +2079,25 @@ def get_chat_template(
|
|||
# stopping_criteria = create_stopping_criteria(tokenizer, stop_word)
|
||||
|
||||
# Patch saving functions
|
||||
if patch_saving:
|
||||
if patch_saving and not is_mlx_backend:
|
||||
from .save import patch_saving_functions
|
||||
tokenizer = patch_saving_functions(tokenizer)
|
||||
|
||||
# Add Ollama
|
||||
tokenizer._ollama_modelfile = ollama_modelfile
|
||||
tokenizer._system_message = system_message
|
||||
|
||||
# Re-wrap so the trainer gets the same TokenizerWrapper type back.
|
||||
if _mlx_tokenizer_wrapper is not None:
|
||||
_mlx_tokenizer_wrapper._tokenizer = tokenizer
|
||||
eos_token_id = getattr(tokenizer, "eos_token_id", None)
|
||||
if eos_token_id is not None:
|
||||
_mlx_tokenizer_wrapper._eos_token_ids = {eos_token_id}
|
||||
_mlx_tokenizer_wrapper._chat_template = None
|
||||
_mlx_tokenizer_wrapper.has_chat_template = (
|
||||
getattr(tokenizer, "chat_template", None) is not None
|
||||
)
|
||||
tokenizer = _mlx_tokenizer_wrapper
|
||||
return tokenizer#, stopping_criteria
|
||||
|
||||
|
||||
|
|
@ -2749,6 +2783,15 @@ extra_eos_tokens = None,
|
|||
|
||||
|
||||
def create_stopping_criteria(tokenizer, stop_word = "eos_token"):
|
||||
try:
|
||||
import torch
|
||||
from transformers import StoppingCriteria, StoppingCriteriaList
|
||||
except ImportError as exc:
|
||||
raise ImportError(
|
||||
"Unsloth: create_stopping_criteria requires PyTorch and is only "
|
||||
"supported on Torch backends."
|
||||
) from exc
|
||||
|
||||
class StoppingCriteriaSub(StoppingCriteria):
|
||||
__slots__ = "stop_token", "single_match", "length",
|
||||
|
||||
|
|
@ -2828,10 +2871,10 @@ def test_chat_templates():
|
|||
for j in range(len(messages)-1):
|
||||
correct_prompt.append_message(correct_prompt.roles[j%2==1], messages[j+1]["content"])
|
||||
correct_prompt.append_message(correct_prompt.roles[1], "")
|
||||
correct_prompt = tokenizer.bos_token + correct_prompt.get_prompt()
|
||||
|
||||
template = vicuna_template
|
||||
correct_tokenizer = AutoTokenizer.from_pretrained("lmsys/vicuna-7b-v1.5")
|
||||
correct_prompt = correct_tokenizer.bos_token + correct_prompt.get_prompt()
|
||||
correct_tokenizer.chat_template = template
|
||||
our_prompt = correct_tokenizer.apply_chat_template(messages[1:], tokenize = False, add_generation_prompt = True)
|
||||
assert(correct_prompt == our_prompt)
|
||||
|
|
@ -2845,10 +2888,10 @@ def test_chat_templates():
|
|||
for j in range(len(messages)-1):
|
||||
correct_prompt.append_message(correct_prompt.roles[j%2==1], messages[j+1]["content"])
|
||||
correct_prompt.append_message(correct_prompt.roles[1], "")
|
||||
correct_prompt = tokenizer.bos_token + correct_prompt.get_prompt()
|
||||
|
||||
template = vicuna_old_template
|
||||
correct_tokenizer = AutoTokenizer.from_pretrained("lmsys/vicuna-7b-v1.5")
|
||||
correct_prompt = correct_tokenizer.bos_token + correct_prompt.get_prompt()
|
||||
correct_tokenizer.chat_template = template
|
||||
our_prompt = correct_tokenizer.apply_chat_template(messages[1:], tokenize = False, add_generation_prompt = True)
|
||||
# We add </s> ourselves
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue