AI Glossary Models and how they work

Inference (in AI)

inference/ˈɪn.fər.əns/noun

  • Tidio Agents team
  • Updated November 19, 2026
  • 5 min read
  • 8 sources
Definition

In AI, inference is the step in which a trained model is run on new input to produce an output, such as a reply, a classification, or a tool call. Training builds the model's knowledge once, at great cost. Inference uses it every time someone sends a request, so it is where a deployed system's running cost and response time come from.

In everyday English and in logic, an inference is a conclusion reached from evidence. The AI sense borrows that idea: the model draws a conclusion from the input in front of it. The EU AI Act uses the word the same way and defines an AI system partly by the fact that it infers, from its input, how to generate outputs.8

For anyone running AI in production, the split matters because the two phases are paid for differently. Training is a project. Inference is a utility bill that grows with every conversation, and its speed decides whether a voice agent sounds natural or a chat agent feels slow.

How it works

What happens during one inference call

When a large language model answers, the work splits into two phases. First the model reads the whole prompt at once, which can run in parallel. Then it writes the reply one token at a time, and each token needs a full pass through the model. Decoding K tokens takes K serial runs of the model.2

PromptRead inputNext tokenAppendone pass per output tokenreply doneGPU or TPU · model weights in memory
Reading the input is fast and parallel. Writing the output is sequential, which is why long replies take longer.
Response time of one call
response time ≈ time to first token + output tokens × time per output token
Time per output token (PaLM 540B, as reported by Google researchers)
29 ms
Reply length
300 tokens
Generation time, before time to first token
≈ 8.7 s

Worked example using the published figure for one large model on Google's hardware.1 Today's models and serving stacks are faster or slower depending on size and setup. The point is the shape: response time grows with reply length.

Compared

Training and inference

TrainingInference
What happensThe model's weights are learned from dataThe fixed weights are applied to new input
How oftenOnce per model version, plus occasional fine-tuningOn every request, all day
Who paysMostly the model providerThe company running the application, per token or per hour of hardware
Main concernData quality and compute budgetCost per conversation, latency, and reliability
Where data goesInto the model, if used for trainingInto the prompt and logs, for one request
ExampleFine-tuning a model on 50,000 resolved ticketsAnswering one customer's question about a refund

Illustrative comparison. The ticket count is an example, not a recommendation.

Agent design

Why inference shapes how agents are built

An AI agent makes several model calls to resolve one request: understand, plan, call a tool, read the result, answer. Each call adds cost and waiting time. Anthropic's guidance puts it directly: agentic systems often trade latency and cost for better task performance, so the trade should be made deliberately.5 Most design choices in a production agent are ways to spend less inference for the same result.

Route by difficulty

Send common, simple requests to a small, fast model and hard ones to a larger model. Anthropic gives exactly this as an example of a routing workflow for customer service.5

Cache what repeats

System prompts, policies, and tool definitions are resent on every call. Prompt caching lets the API reuse them; cache reads are billed at a tenth or less of the normal input price on Anthropic's list as of November 2026.67

Serve efficiently

Serving software matters. The vLLM system raised throughput of popular LLMs by 2 to 4 times at the same latency by managing GPU memory better.3 Speculative decoding sped up generation 2 to 3 times with identical outputs.2

Use smaller models

A model trained for one narrow task, often through distillation from a larger one, answers faster and costs less per call than a general model.

In practice

Inference budgets by agent type

Different agents put different limits on inference. The right model and setup follow from those limits.

Voice agents

Latency comes first

Callers notice pauses. Short replies, fast models, and streaming audio matter more than the largest model.

speech → fast model → stream reply
Customer support

Cost per resolution

Chat can tolerate a few seconds. The budget is cost per resolved conversation across several model calls.

route → 2 to 5 calls → resolve
Shopping

Fast first answer

Shoppers browse while they wait. Show search results quickly, then add the model's comparison as it arrives.

search → stream → compare
Back office

Throughput over speed

Invoices and claims can be processed in batches overnight, where throughput and price per document matter more than seconds.

queue → batch inference → post

Illustrative. The number of calls per conversation depends on the agent's design.

Risks

Inference risks and controls

Runaway cost

ControlAn agent stuck in a loop keeps calling the model. Cap steps per task, set spend alerts, and log tokens per conversation.

Slow answers at peak

ControlShared APIs slow down or rate-limit under load. Load-test for peak season and keep a fallback model or provider.

Silent model changes

ControlA provider can update or retire a model, and the same prompt then behaves differently. Pin model versions and rerun evaluations before switching.

Data leaving your control

ControlEvery inference call sends the prompt to wherever the model runs. Decide which data may go to an external API and which needs a model on your own infrastructure.

FAQ

Common questions

What is the difference between training and inference?

Training teaches a model by adjusting its weights on large amounts of data. Inference uses the finished model to answer new inputs. A model is trained once per version and runs inference millions of times.

Why is inference expensive?

Large models need specialized chips with a lot of memory, and generating each output token requires a full pass through the model. The cost grows with the number of requests and the length of prompts and replies.

What is inference latency?

The time from sending a request to receiving the answer. It is usually split into time to first token and the time to generate the rest of the reply.

Can inference run on our own servers?

Yes, with open-weight models. You then pay for hardware and operations instead of per token, and the data never leaves your environment.

Sources

References

  1. Pope, R. et al. Efficiently Scaling Transformer Inference. Google, 2022.
  2. Leviathan, Y., Kalman, M. and Matias, Y. Fast Inference from Transformers via Speculative Decoding. ICML 2023.
  3. Kwon, W. et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.
  4. Snell, C. et al. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. 2024.
  5. Anthropic. Building effective agents. December 2024.
  6. Anthropic. Prompt caching. Claude documentation.
  7. Anthropic. Pricing. Claude documentation.
  8. European Commission. AI Act, Article 3: Definitions. Regulation (EU) 2024/1689.

Tidio Agents

Build an agent that resolves, not only answers

We design and build custom AI agents for customer service, shopping and back-office work, on your infrastructure, with the code handed over to you. Projects start with a 21-day private AI audit.

Request a call