Hugging Face cheatsheet
Hugging Face: Datasets
8 common Hugging Face tasks.
Load a dataset
Load the training split of a dataset from the Hub.
from datasets import load_dataset
ds = load_dataset("imdb", split="train")
Load a local file
Load a CSV, JSON or Parquet file as a dataset.
ds = load_dataset("csv", data_files="data.csv")
Transform every row
Apply a function to the dataset, in batches for speed.
ds = ds.map(lambda batch: tokenizer(batch["text"], truncation=True), batched=True)
Filter rows
Keep only rows that satisfy a condition.
ds = ds.filter(lambda row: len(row["text"]) > 50)
Make a train and test split
Hold out 10% of the rows for testing.
splits = ds.train_test_split(test_size=0.1, seed=42)
Stream a huge dataset
Iterate without downloading the whole dataset first.
ds = load_dataset("allenai/c4", "en", split="train", streaming=True)
next(iter(ds))
Share a dataset
Upload a dataset to your account. Log in first.
ds.push_to_hub("my-user/my-dataset")