A chatbot reply, a photo label, and a fraud alert all share one hidden step: a trained AI model is acting on new information.
The model has already learned patterns from examples. Now it uses what it learned to respond to a user, sort an image, or flag a payment.
This is where AI becomes useful inside an application or business workflow. A good answer must arrive at the right time and fit the task.
Here, you will see what happens during inference and why speed and cost matter. You will also learn how different setups affect the experience.
The practical question is not only what a model can learn, but how well it can respond when someone needs it.
Key Takeaways
- Inference turns a trained model’s knowledge into a result.
- Some models return a fixed answer; language models build responses piece by piece.
- Response speed depends on more than raw computing power.
- Where a model runs can affect privacy, cost, and connectivity.
- Input and output volume both shape many AI service bills.
What Is AI Inference and Why Does It Matter?
AI inference is the execution step in which a trained model applies its learned parameters to new input and returns a result. Inference can produce a prediction, decision, or generated output without changing the model’s learned weights.
Training finds patterns in data; inference applies those patterns to a new case. A forward pass is a model calculation that applies fixed learned parameters to input without updating them.
For example, a photo classifier receives an image as input. The model runs a forward pass and returns a label, such as “dog.”
Machine learning training can take hours or weeks as a model learns from a dataset. Individual inference predictions are usually much faster because the trained model applies what it already learned.
Serving means deploying and managing a model so it can handle inference requests. An application may send each request through an API endpoint, which connects the application to the model.
Inference is where learned capability becomes a practical result; its speed, reliability, and resource use shape what the user experiences.
The same model can respond quickly in one setting and slowly in another. The application, available hardware, request size, and number of active users all affect performance.
If you want a beginner-friendly comparison of inference and training, the key difference is what happens to the model’s parameters.
Classic machine-learning inference often uses one forward pass for a fixed-size result. A large language model can instead generate a response of changing length, one token at a time.
A normal conversation with a generative model does not by itself rewrite the model’s learned weights.
What Happens Inside an AI Inference Request?
A photo classifier can return one fixed-size label, while a language model must keep producing tokens until its answer is complete. That difference changes how a request feels to you.
An LLM processes the prompt first, then generates its answer in a series of steps. That first stage is called prefill, and it affects how long you wait for the first token.
When you send a ChatGPT message, the system splits your text into tokens, small pieces of text the model can process. The model processes the prompt and stores intermediate attention results in a KV cache.
During decode, the model generates response tokens sequentially. ChatGPT can stream those tokens to you as they arrive, rather than waiting for the full answer.
- Prepare and tokenize the input.
- Process the prompt during prefill.
- Generate response tokens during decode.
- Deliver the response to the application or user.
Classic models return a fixed-size result
A classifier may assign a label or score after one forward pass. Its output has a known shape, such as a class name or a set of scores.
A language model has no fixed answer length in the same way. It keeps generating until it reaches a stopping point or a set output limit.
Large language models generate tokens in stages
Time to first token measures the wait before the answer begins. Time per output token and tokens per second describe the pace after generation starts.
Long prompts can strain memory capacity during prompt processing. Token generation can depend on memory bandwidth, while each active request adds KV-cache memory use.
Compute may be available while memory still limits the request. A model handling many chats at once must hold more active KV caches than one serving a single user.
For a clear explanation of AI tokens and tokenization, remember that token counts vary by model and tokenizer. A short-looking sentence can use more tokens than you expect.
The prefill stage mainly shapes the wait for the first token, while decode shapes the pace afterward. A longer answer also needs more generation steps.
For a chat response, the delay before the first token and the pace of the following tokens describe different parts of the experience.
Which Inference Setup Fits the Job?
A chatbot needs an answer as the request arrives, but a nightly report can wait for a scheduled batch. The timing should fit the work.
Online, batch, and edge inference suit different needs for response time, workload size, and data handling. The right mode depends on when results matter and where the data begins.
Online inference handles requests as they arrive. A chatbot is a common example because a user expects a reply during the conversation.
Online inference serves requests as they arrive
Online requests may run in the cloud or on a company’s own on-premise systems. The location describes where processing happens, not whether requests arrive continuously or on a schedule.
Small requests can also be grouped through micro-batching. It raises throughput while aiming to keep added delays to milliseconds or a few seconds.
Batch inference trades immediacy for efficiency
Batch inference processes a group of requests at a scheduled time. A company might classify a large set of documents overnight when each result does not need to arrive immediately.
Grouping work can make better use of processing resources. The trade-off is that users wait for the batch to run and finish.
Edge inference keeps processing near the data
Edge inference processes data on or near the device that creates it. A sensor device can analyze its own readings instead of sending every reading elsewhere.
This can reduce network delay, limit the need to upload sensitive data, and use less bandwidth. It can also support local operation when the device is offline.
| Mode | Typical timing | Useful fit |
|---|---|---|
| Online | As requests arrive | Interactive service |
| Batch | Scheduled groups | Non-urgent large dataset |
| Edge | Near the data source | Local or offline response |
A 2024 study on edge and cloud latency found that edge processing can offer lower delay and better privacy. Cloud processing can support larger workloads and more complex tasks.
Cloud, on-premise, edge, and on-device describe deployment locations. Online and batch describe when requests run, so a cloud system can support either mode.
Choose edge processing when avoiding a network round trip or keeping data on-device matters more than access to larger remote compute.
Where Does Inference Show Up in Everyday Applications?
A new credit-card transaction can be scored before a payment is approved, just as a new email can be sorted before it reaches an inbox. Each result comes from applying a model to fresh data.
Prediction and classification systems label, score, or rank new inputs. Their outputs can help a person decide what to do or guide another system’s next step.
Predictions and classifications act on incoming data
These applications compare new data with patterns learned during development. They may return a category, risk score, or ranked list rather than a full explanation.
- Email filtering: incoming message data goes in; a spam or legitimate label comes out.
- Transaction fraud flags: purchase details go in; a possible-fraud score or flag comes out.
- Image interpretation: an X-ray or MRI goes in; a finding or image label comes out.
- Product recommendations: browsing or purchase data goes in; a ranked product list comes out.
Medical image systems can help clinicians review X-rays and MRIs, but an output still needs care in high-stakes settings. An AI result can be useful without replacing human judgment.
Generative tools create new output
Generative AI applies learned patterns to a prompt and creates text, images, or other output. A user might ask for a draft, a summary, or a picture based on a description.
The result can also feed another decision. For example, a support system may draft a reply for a worker to review before sending it.
Some Internet of Things devices process sensor input near where the data begins. A camera can sort images locally, while a connected system may send a result to another application.
If you want more examples of AI in everyday applications, look at what goes into a system and what comes out. That input-output pair makes the model’s task easier to see.
The common thread is that each application turns a fresh input into an output that a person or another system can use.
What Hardware Makes Inference Fast?
A small classifier and a large language model can need very different hardware, even when both are performing inference. The workload determines what matters most.
Training needs sustained compute to update parameters; inference repeats the forward pass and may favor speed, memory, or throughput. A model’s size and request pattern help guide hardware choices.
CPUs can run lighter workloads and handle general system tasks. GPUs can speed up parallel calculations, while TPUs and NPUs target AI workloads.
FPGAs and ASICs offer other options for specific tasks. No hardware type wins for every model, since model size and workload shape the fit.
A useful test should match the requests your application receives. Short prompts and long generated answers can put different demands on a system.
| Test profile | Input tokens | Output tokens |
|---|---|---|
| Short | 50 | 20 |
| Medium | 500 | 100 |
| Long | 2,000 | 500 |
The MLPerf Inference benchmark lists datacenter test results with details about submitted systems, software, and hardware. Its results can help you compare systems tested under stated scenarios.
Use this checklist for a repeatable load test:
- Warm up with 100-500 requests before measuring warm performance.
- Test short, medium, and long input and output sequences.
- Ramp up to target concurrency, then hold steady state for at least 5-10 minutes.
- Measure cold starts separately after restarting the platform.
Track time to first token, tokens per second, P95 latency, and cold-start time as separate measures. P95 latency shows the response time that 95 percent of requests meet or beat.
Benchmark with the model, request lengths, and concurrency your application will actually use.
How Much Does AI Inference Cost?
A single inference request can be billed differently depending on the model, the amount of input and output, and the service mode. Both sides of the request can affect the bill.
In 2026, standard API rates range from $0.10 to $10.00 per million input tokens across these GPT-6 models. Output rates also vary, so one model’s rate does not describe every workload.
| Model | Input rate per million tokens | Output rate per million tokens |
|---|---|---|
| GPT-6 Astra | $10.00 | $50.00 |
| GPT-6.1 Sol | $2.00 | $10.00 |
| GPT-6.1 Terra | $1.00 | $5.00 |
| GPT-6 Luna | $0.10 | $0.50 |
API token rates vary by model and output
At standard rates, GPT-6 Luna’s 2026 rates are $0.10 per million input tokens and $0.50 per million output tokens. GPT-6 Astra costs more per token, especially for generated output.
GPT-5.6 Sol lists at $4.00 per million input tokens and $20.00 per million output tokens in 2026. GPT-5.6 Cyber lists at $12.50 for input and $75.00 for output per million tokens.
A rough API estimate uses the input and output token counts separately, then applies each rate. Token counts vary by model and tokenizer, so text length alone does not give an exact bill.
Batch, fast, voice, and search billing differ
GPT-6 Astra’s Batch API rates in 2026 are $5.00 per million input tokens and $25.00 per million output tokens. Its Fast mode rates are $20.00 for input and $100.00 for output per million tokens.
Eligible regional-processing endpoints for models released on or after March 5, 2026 carry a 10% uplift. That condition applies to eligible endpoints, not every API request.
GPT-Live 1 voice sessions cost $0.05 per minute in 2026. Web search costs $10.00 per 1,000 calls, and search-content tokens use the model’s token rates.
Example, standard text API: One million input tokens and one million output tokens with GPT-6 Luna cost $0.60 at the listed standard rates.
Example, separate tools: A voice session lasting 10 minutes costs $0.50 at the listed rate, before any other charges. Search calls and their content tokens have separate billing.
Estimate a real workload from both input and output volume, then account separately for tools or modes that have their own charges.
How Is Inference Different From Reasoning and Agentic AI?
Inference describes applying a trained model to input, while reasoning and agency describe different aspects of how an AI system may solve a task. The terms name different parts of the process.
Inference is model execution; reasoning is a way a system may work through a problem. A model can produce a useful answer during inference without learning from that exchange.
Inference applies a model to input
Generative AI creates content from an input, such as a prompt for text or an image. Producing a response is an inference task, not proof that the model learned from the conversation.
AI is also grouped by capability: narrow AI, general AI, and super AI. Narrow AI is built for specific tasks, while general AI and super AI remain concepts that have not been achieved.
A second scheme groups AI by functionality: reactive machines, limited-memory AI, theory-of-mind AI, and self-aware AI. These labels describe different proposed types of system behavior.
Agentic AI plans and acts toward a goal
Agentic AI pursues a goal by planning steps, using tools, checking results, and taking actions. It can combine several model outputs with software calls or other system actions.
A system that drafts a response is generative. An agentic system may draft it, check an order record, and then choose a next step.
Using tools also brings security concerns, especially when prompts can affect connected resources. The NIST report on adversarial machine learning covers risks such as prompt injection and attacks on connected systems.
A system can generate content during inference, while an agentic system may use outputs to choose and carry out further steps.
Who Pays for Inference, and Where Does the Money Go?
Inference has paying users on one side and compute providers and AI model companies on the other. Those roles shape how money moves through the business.
Individuals and businesses pay for API use or subscriptions, while providers pay for the compute that serves requests. The companies that supply computing power are not always the companies that build the model.
Cloud providers supply compute to AI companies
AWS, Azure, and Google Cloud sell infrastructure that AI companies can use to run models. Their role is to supply compute, storage, and related cloud services.
AI model companies may rent that infrastructure and sell access through APIs or subscriptions. A business can pay for API use, while an individual may pay for an AI service subscription.
The global AI inference market was valued at $113.5 billion in 2025. It was projected to reach $132.7 billion in 2026 and $421.8 billion in 2033, with an 18.0% projected annual growth rate from 2026 to 2033.
North America accounted for 37.0% of global AI inference market revenue in 2025. High-bandwidth memory held over 60.0% of the memory segment, while GPUs made up around 35.0% of the compute segment.
Machine-learning models accounted for around 51.0% of the market by application in 2025. IT and telecommunications represented around 18.0% by end use.
These are 2025 market estimates and segment shares, not profit shares. A large market segment does not show how much money each company keeps.
AI companies earn through APIs and subscriptions
Barclays estimated that AWS, Azure, and Google Cloud together received $35-$40 in inference-compute revenue for every $100 in AI model-company revenue in 2026. That estimate describes revenue flowing to cloud providers.
Barclays estimated AI labs’ paid-inference margins rose from the low teens in 2025 to 50%-65% or higher in 2026. The estimates differ by product: direct API inference margins exceeded 80%, while subscription-product inference margins were around 70% in 2026.
Cloud providers’ operating margins on inference-compute revenue from AI labs were estimated at roughly 35%-45% in 2026. Operating margins and product margins describe different parts of the business.
Market growth and margin estimates do not mean every AI service or provider is profitable. Compute, product, and serving costs still have to fit the revenue a service earns.
Subscriptions can bring in steady payments, while API bills rise with usage. Both business models must cover the costs tied to delivering useful results.
A useful business check is whether revenue per request or subscription can cover compute, serving, and product costs at the scale the service actually reaches.
Conclusion
AI becomes useful to people when a trained model can turn new input into a dependable response. Inference is that working step, and its value depends on performance, deployment, and cost fitting the task.
When you try an AI tool, notice how long it takes to reply and whether it works locally or through an online service. Those details can tell you how the tool handles your request.
The next time you use an AI tool, notice what it takes in, what it returns, and how quickly the response arrives.
FAQ
What are the different types of inference in artificial intelligence?
The three common types are online, batch, and edge inference. Online inference handles requests as they arrive, while batch inference processes scheduled groups of work. Edge inference runs near the data source, often on or close to the device. Micro-batching sits between online and batch work by grouping requests while keeping added delays short.
What are 7 types of AI?
Seven common labels come from two classification schemes. By capability, AI is called narrow, general, or super AI; narrow AI handles specific tasks, while general and super AI have not been achieved. By functionality, the labels are reactive machines, limited-memory AI, theory-of-mind AI, and self-aware AI. The two schemes classify different traits.
What does running inference mean?
Running inference means a trained model applies fixed learned parameters to new input and returns an output. The model calculation is called a forward pass, and it does not update the model’s weights. For example, an image model can take a picture as input and return a label. The result could be a prediction, decision, or generated response.
What is the difference between AI inference and training?
Training updates a model’s parameters as it learns patterns from data; inference uses those trained parameters on new input. Training can take hours to weeks, while an individual inference prediction is usually much faster. Inference itself does not change the model’s learned weights. A separate training or fine-tuning process is needed to update them.
Is ChatGPT an inference engine?
ChatGPT is a user-facing service that uses model inference to process prompts and generate responses. The service includes more than the model’s calculation, such as the interface and the systems that handle requests. A conversation can produce a new answer without changing the model’s learned weights. Learning from conversations requires a separate training process.
How much does AI inference cost?
In a 2026 API example, GPT-6 Luna lists at $0.10 per million input tokens and $0.50 per million output tokens. One million of each would cost $0.60 at those standard rates. Other models charge different rates, and service modes can change the bill. Voice sessions and web search may also have separate charges.
Is AI inference actually profitable?
Some 2026 estimates put paid-inference margins at 50%-65% or higher for AI labs. Direct API margins were estimated above 80%, subscription-product margins around 70%, and cloud operating margins at roughly 35%-45% on inference-compute revenue from AI labs. These estimates cover different business measures and do not prove that every AI company or service earns a profit.
Is anyone paying for AI?
Yes, individuals and businesses pay for AI subscriptions and API usage. A business might pay each time its application sends an API request, while a person might pay a recurring subscription fee. Cloud providers such as AWS, Azure, and Google Cloud earn revenue when AI companies rent compute. Those infrastructure payments differ from what end users pay for an AI product.
