A large language model (LLM) is a neural network trained on very large amounts of text to predict the next token in a sequence, which lets it read, write, summarize, translate, and reason over language. It is the engine inside modern chat assistants and AI agents, but on its own it only produces text: it does not look anything up or take actions.
The key distinction for a buyer is between the model and the system built around it. An LLM is a general text engine. A support chatbot or an AI agent is a product that wraps an LLM with instructions, company data, tools, permissions, and checks. Most of what decides whether a customer gets a correct answer sits in that wrapper, not in the choice of model.
How a large language model is built
Almost every current LLM is a transformer, an architecture that uses a mechanism called attention to weigh how each part of the input relates to every other part.1 The model works on tokens, small pieces of text, and it is built in three broad stages.
1. Pretraining
The model reads a large share of the public web, books, and code, and learns to predict the next token. Doing that well forces it to absorb grammar, facts, styles, and patterns of reasoning. GPT-3, trained this way, had 175 billion parameters and could do new tasks from a few examples in the prompt.2
2. Instruction tuning
A pretrained model continues text; it does not follow requests. Developers fine-tune it on examples of instructions paired with good answers, so it learns to act as an assistant.
3. Preference tuning
People rank alternative answers, and the model is tuned toward the preferred ones. In OpenAI's InstructGPT work, outputs from a 1.3 billion parameter model tuned this way were preferred to those of the 175 billion parameter GPT-3.3
4. Use (inference)
In production the model receives a prompt and generates a reply token by token. Each new token is fed back in as input until the answer is complete.
The same model, with and without a system around it
A customer asks an online fashion retailer about a return. Both lanes use the same LLM. Only the second one can see the order and the policy.
LLM alone
Predicts- GenerateWrites the most likely answer for a typical retailer: "Most stores accept returns within 30 days."
- ReplySounds confident. The model has no access to this retailer's policy, which is 14 days for outerwear.
Fluent, plausible, wrong. The customer is promised a return the company will refuse.
LLM inside an agent
Grounded- Toolorders.find(customer_id)→ jacket · delivered 22 days ago
- RetrieveFinds the returns policy: outerwear within 14 days, store credit up to 30 days.
- GenerateThe LLM writes the reply from these facts, not from memory.
- ReplyExplains that a refund is no longer possible, offers store credit, and starts the return if the customer agrees.
Correct and actionable. The answer cites the real policy and the real order.
Illustrative example. The policy terms and tool names are placeholders, not any retailer's real rules.
What an LLM can and cannot do in a business
LLMs are strong at language and weak at facts they were never given. That splits business tasks into two groups.
- Reliable: understanding messy customer messages in many languages, classifying intent, extracting fields from documents, summarizing long threads, drafting replies in a set tone, and turning a request into a structured tool call.
- Reliable with grounding: answering policy and product questions, when the system retrieves the right documents and tells the model to answer only from them. This is retrieval-augmented generation.4
- Unreliable on its own: order status, prices, stock, account balances, or anything that changed after training. The model has no live data unless a tool supplies it.
- Not its job: exact arithmetic over many records, enforcing business rules, and deciding what it is allowed to do. Code and permissions should do that, with the model calling them.
LLM, chatbot, and AI agent
The three are often used as synonyms. They sit at different layers of the same stack.
| Term | What it is | Knows your data? | Can act? | In a support team |
|---|---|---|---|---|
| Large language model | A trained model that generates text | Only what is in the prompt | No | The engine behind the other two |
| Chatbot | An application that holds a conversation | If connected to a knowledge base | Rarely, and only along fixed paths | Answers FAQs and hands off |
| AI agent | A system that uses an LLM to pursue a goal with tools | Yes, through retrieval and APIs | Yes, within permissions | Checks the order and issues the refund |
| Small language model | A compact model, often distilled from a larger one | Only what is in the prompt | No | Fast intent routing or extraction at volume |
Where LLMs do the work inside an agent
In each case below the LLM handles the language, and the surrounding system supplies data, rules, and actions.
Understand and resolve
Reads a complaint about a delayed parcel or a double charge, identifies the intent, and drafts the resolution from order and payment data.
understand → fetch data → resolve → replyTurn needs into a search
Converts "a quiet dishwasher that fits under my counter" into filters, compares the results, and explains the trade-offs.
interpret → search catalog → compare → adviseAnswer from your content
Answers questions on a telecom's plans or an insurer's policies from approved documents, with the source shown.
retrieve → answer with citationRead documents
Extracts amounts, dates, and parties from claims, invoices, and contracts in several languages into structured fields.
read → extract → validate → postSummarize for employees
Summarizes a long customer history before a person takes over, so the customer does not repeat themselves.
collect → summarize → hand overQualify in conversation
Asks qualifying questions in natural language, by chat or voice, and writes the answers into the CRM.
ask → structure → update CRMLLM risks and the usual controls
Hallucination
ControlGround answers in retrieved documents and live data, instruct the model to say when it does not know, and test on real transcripts before launch.5
Prompt injection
ControlText inside an email, review, or web page can carry instructions the model may follow.6 Treat retrieved content as untrusted and give each tool the narrowest permission.
Data exposure
ControlDecide what customer data reaches the model and where it runs. Mask personal data the task does not need, and keep logs under your own retention rules.
Inconsistent output
ControlThe same prompt can produce different answers. Use structured outputs for anything a system consumes, and score answers continuously, not once.
Transparency duties
ControlUnder the EU AI Act, people must be told when they are interacting with an AI system unless it is obvious.9
Common questions
Is ChatGPT a large language model?
ChatGPT is an application. It runs on OpenAI's GPT family of large language models and adds a chat interface, memory, and tools such as web search on top.
What is the difference between an LLM and generative AI?
Generative AI is the whole category of models that create content, including images, audio, and video. LLMs are the part of it that works with text and code.
Do LLMs learn from my company's conversations?
Not by default. A deployed model's weights stay fixed. It only uses your data if you put it in the prompt, connect it through retrieval, or deliberately fine-tune a model on it.
Which LLM is best for customer service?
There is no single answer. Test two or three models on your own transcripts and languages, and compare resolution rate, latency, and cost per conversation. The system around the model usually matters more than the model.
Can an LLM run on our own infrastructure?
Yes. Open-weight models can run in your own cloud or data center, which keeps data inside your environment. The trade-off is that you operate and scale the hardware yourself.
References
- Vaswani, A. et al. Attention Is All You Need. NeurIPS 2017.
- Brown, T. et al. Language Models are Few-Shot Learners. NeurIPS 2020.
- Ouyang, L. et al. Training language models to follow instructions with human feedback. NeurIPS 2022.
- Lewis, P. et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020.
- Kalai, A. T. et al. Why Language Models Hallucinate. 2025.
- OWASP GenAI Security Project. LLM01: Prompt Injection. OWASP Top 10 for LLM Applications, 2025.
- Bommasani, R. et al. On the Opportunities and Risks of Foundation Models. Stanford CRFM, 2021.
- Anthropic. Context windows. Claude documentation.
- European Commission. AI Act, Article 50: Transparency obligations. Regulation (EU) 2024/1689.