Commit graph

6 commits

Author SHA1 Message Date
pre-commit-ci[bot]
3620564025 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2026-01-08 11:35:21 +00:00
Daniel Han
16a2d901fa Fix bugs and add improvements to RawTextDataLoader
- Fix test file: use return_tokenized instead of return_tensors
- Fix test file: use text_dataset instead of undefined dataset variable
- Move parameter validation to constructor (fail fast on invalid params)
- Add labels field in tokenized output for causal LM training
- Add empty file handling with clear error message
- Add tests for constructor validation and labels field
2026-01-08 11:35:00 +00:00
pre-commit-ci[bot]
3bf8ca7da2 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2025-11-20 13:09:08 +00:00
vangmay
f05169e56a Make the chunk function efficient 2025-11-20 21:08:33 +08:00
pre-commit-ci[bot]
d429363c23 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2025-11-20 12:51:18 +00:00
vangmay
ee37dd9f92 Write simple test 2025-11-18 22:36:38 +08:00