What is Tokens?

When you send a message to an AI model like GPT-4 or Claude, the raw text is not processed character by character. Instead, a component called a tokenizer splits the input into tokens — small chunks that can be single characters, subwords, whole words, or even short phrases. For example, the word "tokenization" might be split into two tokens: "token" and "ization". Common short words like "the" or "is" are usually single tokens, while rarer or longer words require multiple tokens.

Why Tokens Matter for Cost

AI API providers charge per token consumed. A typical prompt might use 200–500 tokens, while a long document analysis could use tens of thousands. Both input tokens (what you send) and output tokens (what the model returns) are counted. Output tokens tend to cost more because generating text is computationally heavier than reading it. Understanding token counts helps you optimize prompts to reduce costs without sacrificing quality.

Context Windows and Token Limits

Every AI model has a context window — the maximum number of tokens it can process in a single request, including both the conversation history and the new message. GPT-4 Turbo supports up to 128,000 tokens; Claude 3 Opus handles up to 200,000. Once you exceed this limit, older parts of the conversation are dropped. Long documents, multi-turn conversations, and code analysis tasks are the most common scenarios where hitting context limits becomes a practical concern.

Tokens vs. Words vs. Characters

A useful rule of thumb is that 100 tokens ≈ 75 words ≈ 400 characters for English text. However, code, non-Latin scripts, and highly technical jargon tokenize differently — Python code tends to be token-efficient because keywords are common, while Chinese or Japanese text requires far more tokens per character due to a larger character set. Tools like OpenAI's Tokenizer playground let you paste text and see the exact token count before spending credits.

Tokens and Browser Extension AI Features

Extensions that integrate AI — such as Prompt Anything Pro — pass your selected text plus a system prompt to the model. Every word you highlight and every instruction in the prompt template consumes tokens. Keeping prompt templates concise and using models with large context windows ensures you get complete, accurate responses even when working with lengthy web pages or articles.