Hugging Face · Transformers
Tokenize text
Convert text to token IDs as PyTorch tensors.
tokens = tokenizer("Hello world", return_tensors="pt")
print(tokens["input_ids"])
Official Hugging Face documentation
Hugging Face · Transformers
Convert text to token IDs as PyTorch tensors.
tokens = tokenizer("Hello world", return_tensors="pt")
print(tokens["input_ids"])
Official Hugging Face documentation
Was this page helpful?