Hugging Face · Datasets
Transform every row
Apply a function to the dataset, in batches for speed.
ds = ds.map(lambda batch: tokenizer(batch["text"], truncation=True), batched=True)
Official Hugging Face documentation
Hugging Face · Datasets
Apply a function to the dataset, in batches for speed.
ds = ds.map(lambda batch: tokenizer(batch["text"], truncation=True), batched=True)
Official Hugging Face documentation
Was this page helpful?