Tokens explain why LLM prompts cost what they cost
This article explains that an LLM does not read text as normal letters. It first breaks text into tokens, and each model may split the same sentence differently. That matters for AI agents because more tokens can mean slower runs and higher API cost.
Key points
Quick term guide
- FIR
- A First Information Report — the official complaint filed with police in India that kicks off a criminal investigation.
- tokens
- Tokens are small pieces of text that AI systems count when reading or writing.
- token
- A small piece of text used to measure AI input, output, and cost.
- diff
- A view that shows exactly what changed in the code.
- AI agents
- AI agents are AI tools that can carry out steps toward a goal, not just answer once.
- AI agent
- An AI program that can inspect information and suggest what to do next.
- agents
- AI helpers that follow your instructions and make changes for you.
- tokenizer
- The tool that splits text into tokens before the model sees it.