AI Learning Express
Platform 01 · Station 4 of 10

Tokens

01 · Watch / read

A token is a small piece of text. It may be a whole word, part of a word, a space or a punctuation mark, such as !. These tickets show one possible example. Real AI systems may split the text differently.

A word can use several token tickets

Learning!Learning!word partword partpunctuation
Illustrative split, not a real tokenizer · Different systems may split text differently.

This example splits Learning! into three tickets: Learn, ing and !. Tokens can be word parts or punctuation marks.

02 · Try

Put the example tickets in order to spell “Learning!”

Choose tickets in order. Use Tab and Enter or Space. Undo lets you revise.

    Progress saves in this browser.