From a353fad514e2229f0c7a61f2fad66c2a190cefc3 Mon Sep 17 00:00:00 2001
From: Fizza Mukhtar <fizzamukhtar01@gmail.com>
Date: Tue, 10 Feb 2026 01:53:46 -0800
Subject: [PATCH] Fix #3397: Prevent trainer tokenization hang with safe
 num_proc (#4013)

* Fix #3397: Prevent trainer tokenization hang with safe num_proc

* Fix #3397: Add missing import sys for Windows-safe tokenization

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Consolidate with existing num_proc guard in dataset_utils.py

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>