The hidden cost of tokens: the precautions to take

An executive shows me a quote. A vendor is offering them an AI assistant for their customer service: "less than 2 cents per request." The figure is reassuring, the contract gets signed. Three months later, the API bill looks nothing like the quote. Not because they were lied to, but because the price per request means almost nothing. The real cost hides in a unit no one explained to them: the token.

This isn't a technician's detail. It's the economic unit of generative AI. As long as you don't understand it, you're signing for a service whose price you don't know. Here's what you need to know before putting a signature at the bottom of an "AI" contract.

What exactly is a token

A language model doesn't read words, it reads tokens: fragments of text, roughly chunks of words. In English, count very roughly that a token is worth around three quarters of a word - "antidisestablishmentarianism" is several, "the" is one. (Order of magnitude, varying by language and model.)

What you need to remember is that you pay for tokens in both directions:

Input tokens : everything you send to the model - the user's question, but also the instructions, the conversation history, and the documents you give it to read.

Output tokens : what the model generates in response.

And - this is crucial - output costs much more than input. As of June 2026, a high-end model like GPT-5.5 is billed at around $5 per million input tokens and $30 per million output; Claude Opus 4.8, about $5 input and $25 output. So output costs five to six times input. (Public prices observed in June 2026, subject to change - sources at the end of the article.)

First concrete consequence: an assistant that answers in long paragraphs costs structurally more than an assistant that gets to the point. Verbosity has a price.

The four places where the bill swells without you seeing it

The quote shows a cost per request. But that cost depends on design choices that appear nowhere in the contract. Here's where the money goes.

1. Context length. For an AI to answer from your internal documents, you "give it to read" the relevant passages with each request. This is what's called RAG (Retrieval-Augmented Generation: you retrieve the right documents and inject them into the question). If the design is lazy and sends 30,000 tokens of context with each question instead of 3,000 well-targeted ones, you pay ten times the input - for every request, all year long.

2. Conversation history. A model has no memory between two messages. For it to "remember" the start of an exchange, you resend the whole history at each turn. A long conversation therefore costs more and more as it goes on, because you re-pay the input with each message.

3. Repeated calls. This is the most underestimated trap. An "agent" (an AI system that chains several steps to accomplish a task) doesn't make one call to the model, it makes five, eight, sometimes more - one per reasoning step. And at each step, it often resends all the accumulated context. A single user-side "task" can thus trigger a dozen billable calls. Add the failed attempts the system automatically retries, and the bill quietly multiplies.

4. Model choice. This is the most brutal lever. As of June 2026, the price gap between the low and high end exceeds a factor of 50: a small model like GPT-4.1 nano runs at around $0.10 per million input tokens, while a high-end reasoning model exceeds $30 - and the ranges shift with every release. Running a simple task on a premium model is like taking a taxi to cross the street.

Why "the price per request" is a trap

Let's put numbers on a generic case. A support assistant that answers from your documentation, 10,000 questions a month.

Naive design: you send 30,000 tokens of context per question, the model answers in 500 tokens, all on a high-end model at $5 / $30 per million. Cost per question: about $0.15 input + $0.015 output ≈ $0.17. Over 10,000 questions: about $1,700 a month.

Careful design: you retrieve only 3,000 tokens of genuinely relevant context, you cache the fixed instructions, and you switch the task to a lighter model at $1 / $5 per million. Cost per question: about $0.003 + $0.0025 ≈ $0.006. Over 10,000 questions: about $55 a month.

Same task. Same result for the user. A factor of 30 on the bill. That's the whole problem: the "price per request" on a quote isn't a property of the product, it's a consequence of design choices the decision-maker doesn't see. Two vendors can deliver the same assistant for operating costs separated by an order of magnitude.

And that cost isn't fixed. It varies with usage: users asking longer questions, conversations getting longer, a traffic spike - and the bill follows, because it's indexed on tokens, not on a flat fee.

The right questions to ask before signing

You don't need to know how to code to evaluate an AI project. You need to ask the vendor the right questions.

Which model does the solution run on, and why that one? If the answer is "the most powerful on the market" for a simple task, dig deeper. A good designer's instinct is to use the smallest model that does the job.

How does the cost evolve with volume? Ask for an estimate at 1,000, 10,000 and 100,000 requests a month. If the vendor can't answer, it means they haven't modelled their costs - so they don't control their margin, or yours.

What happens when a conversation gets longer or a user sends a large document? The answer will tell you whether they've anticipated the cases that blow up the bill.

Do you use caching and batch processing? These are two standard levers. As of June 2026, caching the fixed parts of the context can cut their cost by around 90%, and batch processing (for non-urgent tasks) by about 50%, depending on the provider. A vendor who ignores them is leaving money on the table - yours.

Who pays for the API, you or them? If it's an all-inclusive flat fee, who absorbs the overruns? If you're paying the provider directly, demand an alert ceiling.

The takeaway

The token isn't a technical detail reserved for engineers. It's what you're really buying when you sign for "AI." The advertised price per request is a fragile average, dependent on design decisions invisible on a contract, and it can vary by a factor of ten at equal design.

The good news is that these costs can be managed. A well-designed system - the right model for the right task, controlled context, caching and batching enabled - costs a fraction of a naive system, for the same service delivered. But it doesn't show in a demo. It shows in the architecture, and in the vendor's ability to explain their choices to you.

Before signing, don't just ask "how much does it cost." Ask "what does that cost depend on, and how do you control it." The answer will tell you whether you're facing an AI salesperson or someone who actually builds it.

Note: the API prices mentioned here reflect the public rates observed in June 2026 (official OpenAI and Anthropic pricing pages) and change frequently.