← All skills
Skill · Tokens & text encodingSoon

Tokenization

Tokens are the atoms of every LLM — split text, count tokens, and decode IDs to demystify context limits and cost.

Coming soon. Here’s the planned set of tasks. Meanwhile, Python is live and runnable right now.
  1. Task 1What is a token
  2. Task 2Split text
  3. Task 3Subword tokens
  4. Task 4Count tokens
  5. Task 5Token IDs
  6. Task 6Decode back