What is a token?

What is a token?

• A. Any word in the sentence.
• B. A token that appears only once.
• C. A punctuation mark.
• D. The sentence's total word count.

Answer: (A)

Tokenization turns text into the basic units a model works with. In common word-level tokenization, each word is treated as one token. So any word in the sentence is a token—the unit of analysis for many NLP tasks. The other ideas don’t define what a token is: a token isn’t just a single occurrence’s frequency, nor is it simply a punctuation mark, and the total word count isn’t the unit itself. (Note: some tokenizers do treat punctuation as separate tokens, but the standard interpretation used here is that a token is a word.)

Tokenization turns text into the basic units a model works with. In common word-level tokenization, each word is treated as one token. So any word in the sentence is a token—the unit of analysis for many NLP tasks. The other ideas don’t define what a token is: a token isn’t just a single occurrence’s frequency, nor is it simply a punctuation mark, and the total word count isn’t the unit itself. (Note: some tokenizers do treat punctuation as separate tokens, but the standard interpretation used here is that a token is a word.)