In AI, inference is the step in which a trained model is run on new input to produce an output, such as a reply, a classification, or a tool call. Training builds the model's knowledge once, at great cost. Inference uses it every time someone sends a request, so it is where a deployed system's running cost and response time come from.
In everyday English and in logic, an inference is a conclusion reached from evidence. The AI sense borrows that idea: the model draws a conclusion from the input in front of it. The EU AI Act uses the word the same way and defines an AI system partly by the fact that it infers, from its input, how to generate outputs.8
For anyone running AI in production, the split matters because the two phases are paid for differently. Training is a project. Inference is a utility bill that grows with every conversation, and its speed decides whether a voice agent sounds natural or a chat agent feels slow.
What happens during one inference call
When a large language model answers, the work splits into two phases. First the model reads the whole prompt at once, which can run in parallel. Then it writes the reply one token at a time, and each token needs a full pass through the model. Decoding K tokens takes K serial runs of the model.2
- Time per output token (PaLM 540B, as reported by Google researchers)
- 29 ms
- Reply length
- 300 tokens
- Generation time, before time to first token
- ≈ 8.7 s
Worked example using the published figure for one large model on Google's hardware.1 Today's models and serving stacks are faster or slower depending on size and setup. The point is the shape: response time grows with reply length.
Training and inference
| Training | Inference | |
|---|---|---|
| What happens | The model's weights are learned from data | The fixed weights are applied to new input |
| How often | Once per model version, plus occasional fine-tuning | On every request, all day |
| Who pays | Mostly the model provider | The company running the application, per token or per hour of hardware |
| Main concern | Data quality and compute budget | Cost per conversation, latency, and reliability |
| Where data goes | Into the model, if used for training | Into the prompt and logs, for one request |
| Example | Fine-tuning a model on 50,000 resolved tickets | Answering one customer's question about a refund |
Illustrative comparison. The ticket count is an example, not a recommendation.
Why inference shapes how agents are built
An AI agent makes several model calls to resolve one request: understand, plan, call a tool, read the result, answer. Each call adds cost and waiting time. Anthropic's guidance puts it directly: agentic systems often trade latency and cost for better task performance, so the trade should be made deliberately.5 Most design choices in a production agent are ways to spend less inference for the same result.
Route by difficulty
Send common, simple requests to a small, fast model and hard ones to a larger model. Anthropic gives exactly this as an example of a routing workflow for customer service.5
Cache what repeats
System prompts, policies, and tool definitions are resent on every call. Prompt caching lets the API reuse them; cache reads are billed at a tenth or less of the normal input price on Anthropic's list as of November 2026.67
Serve efficiently
Serving software matters. The vLLM system raised throughput of popular LLMs by 2 to 4 times at the same latency by managing GPU memory better.3 Speculative decoding sped up generation 2 to 3 times with identical outputs.2
Use smaller models
A model trained for one narrow task, often through distillation from a larger one, answers faster and costs less per call than a general model.
Inference budgets by agent type
Different agents put different limits on inference. The right model and setup follow from those limits.
Latency comes first
Callers notice pauses. Short replies, fast models, and streaming audio matter more than the largest model.
speech → fast model → stream replyCost per resolution
Chat can tolerate a few seconds. The budget is cost per resolved conversation across several model calls.
route → 2 to 5 calls → resolveFast first answer
Shoppers browse while they wait. Show search results quickly, then add the model's comparison as it arrives.
search → stream → compareThroughput over speed
Invoices and claims can be processed in batches overnight, where throughput and price per document matter more than seconds.
queue → batch inference → postIllustrative. The number of calls per conversation depends on the agent's design.
Inference risks and controls
Runaway cost
ControlAn agent stuck in a loop keeps calling the model. Cap steps per task, set spend alerts, and log tokens per conversation.
Slow answers at peak
ControlShared APIs slow down or rate-limit under load. Load-test for peak season and keep a fallback model or provider.
Silent model changes
ControlA provider can update or retire a model, and the same prompt then behaves differently. Pin model versions and rerun evaluations before switching.
Data leaving your control
ControlEvery inference call sends the prompt to wherever the model runs. Decide which data may go to an external API and which needs a model on your own infrastructure.
Common questions
What is the difference between training and inference?
Training teaches a model by adjusting its weights on large amounts of data. Inference uses the finished model to answer new inputs. A model is trained once per version and runs inference millions of times.
Why is inference expensive?
Large models need specialized chips with a lot of memory, and generating each output token requires a full pass through the model. The cost grows with the number of requests and the length of prompts and replies.
What is inference latency?
The time from sending a request to receiving the answer. It is usually split into time to first token and the time to generate the rest of the reply.
Can inference run on our own servers?
Yes, with open-weight models. You then pay for hardware and operations instead of per token, and the data never leaves your environment.
References
- Pope, R. et al. Efficiently Scaling Transformer Inference. Google, 2022.
- Leviathan, Y., Kalman, M. and Matias, Y. Fast Inference from Transformers via Speculative Decoding. ICML 2023.
- Kwon, W. et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.
- Snell, C. et al. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. 2024.
- Anthropic. Building effective agents. December 2024.
- Anthropic. Prompt caching. Claude documentation.
- Anthropic. Pricing. Claude documentation.
- European Commission. AI Act, Article 3: Definitions. Regulation (EU) 2024/1689.