Fix #3397: Prevent trainer tokenization hang with safe num_proc (#4013)

Browse source

* Fix #3397: Prevent trainer tokenization hang with safe num_proc

* Fix #3397: Add missing import sys for Windows-safe tokenization

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Consolidate with existing num_proc guard in dataset_utils.py

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>

...

This commit is contained in:

Fizza Mukhtar 2026-02-10 01:53:46 -08:00 • committed by GitHub

parent acfe670357

commit a353fad514

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download patch file Download diff file Expand all files Collapse all files

Diff content is not available