What Is AI Inference and How Does It Work?

Every time an app recognizes your face, suggests a video, or finishes your sentence, a trained model is doing one specific job. It looks at something new and gives an answer. That job is called inference.

Most beginners spend months learning how models are trained and almost no time thinking about what happens afterward. But training happens once, or occasionally. Inference happens every single time someone uses the model.

I only noticed this while looking back at one of my own beginner projects, a flower classifier built with scikit-learn. I had been writing model.predict() without ever thinking about what that line actually meant. It turned out that one line was the whole idea this article is about.

AI inference is the process of using a trained model to make a prediction or generate an output from new data it has never seen before.

A simple way to picture it is an exam. Training is the months of studying. Inference is sitting in the exam hall. You use what you already learned, and you do not rewrite the textbook halfway through the paper.

A model behaves the same way. During inference, its internal weights stay frozen. Nothing new is learned. An input goes in, and a prediction comes out.

This applies to every kind of AI. A spam filter marking an email as junk is inference. A phone unlocking when it sees your face is inference. A generative AI tool writing a reply is inference too. The task changes, but the idea does not.

If you are new to how models learn in the first place, my guide on what machine learning is covers the training side. This article covers what happens once that learning is done.

People often assume the hard part of AI ends when training finishes. In production, it is closer to the opposite. Training is a project with an end date. Inference runs for as long as the model stays live, and it grows with every new user.

Whatever the model is, inference usually follows the same six steps.

1. Input arrives: This can be text, an image, a sound clip, or a row of numbers.

2. Preprocessing: The input is converted into the exact format the model expects. Images get resized, numbers get scaled, and text gets split into tokens. These steps must match what was done during training.

3. Forward pass: The model pushes the input through its layers or equations using the weights it learned. Data only moves forward. There is no learning step here.

4. Raw output: The model produces numbers, usually a score or probability for each possible answer.

5. Post-processing: Those numbers are turned into something useful. That could mean picking the highest probability and converting a class number into a readable label.

6. Delivery: The result reaches the app and is presented to the user.

Large language models repeat the forward pass many times, producing one token at a time. That is why chatbot answers appear word by word.  

The Iris dataset is one of the oldest teaching datasets in machine learning. Statistician Ronald Fisher published it in 1936, and today it is hosted by the UCI Machine Learning Repository.

It contains 150 flowers from three species. Each flower is described by four measurements: sepal length, sepal width, petal length, and petal width. The goal is to predict the species from those measurements.

I split the data 80/20, using stratification so every species appeared equally. That left 120 flowers for training and 30 for testing, ten from each species.

Before any model saw the data, I scaled it:

python

scaler = StandardScaler()

X_train_scaled = scaler.fit_transform(X_train)

X_test_scaled = scaler.transform(X_test)

Look closely at those last two lines. The training data uses fit_transform, which learns the scaling values. The test data uses only transform, which reuses those same values. This is an inference rule in disguise. 

Whatever you do to data during training, you must repeat exactly the same way on new data later. The StandardScaler documentation explains how this works.

Then I trained and tested two models:

python

knn = KNeighborsClassifier(n_neighbors=5)

knn.fit(X_train_scaled, y_train)          # training

y_pred_knn = knn.predict(X_test_scaled)   # inference

The fit() line is training. The predict() line is inference. Same for logistic regression.

The two models handle inference very differently. Logistic regression does its learning during training, so predicting is a quick calculation. KNN does almost nothing during training. It simply stores the flowers, then compares every new flower against those stored examples at prediction time. With 120 flowers, nobody notices. With millions of stored examples, KNN inference becomes slow and expensive.

Both models scored 1.0 on my test split, and the confusion matrices showed all 30 flowers on the diagonal, 10 correct for each species. I treat that number carefully. Thirty flowers from a famously easy dataset is a very small test. It proves my pipeline works. It does not prove the model is brilliant.

KNN confusion matrix from my Iris classifier. All 30 test flowers were predicted correctly.

My comparison code also taught me something. It printed “Logistic Regression performed better” even though both scores were exactly 1.0. My if statement only checked whether KNN was strictly higher, so a tie fell into the else branch. It was a tiny bug, but it showed me that code can state a confident conclusion the data does not support.

To see inference on a single new flower, I used this:

python

new_flower = pd.DataFrame(

    [[5.9, 3.0, 4.2, 1.5]],

    columns=X.columns

)

new_flower_scaled = scaler.transform(new_flower)

prediction = knn.predict(new_flower_scaled)

print(le.inverse_transform(prediction))

This is the six-step process in four lines. The measurements are the input. scaler.transform is preprocessing. knn.predict is the forward pass. inverse_transform turns the model’s number back into a species name, which is post-processing. The model never saw this exact flower. It only used what it learned.

Not every system needs an answer instantly. The type of inference depends on how fast the answer must arrive.

My Iris test was effectively batch inference, since all 30 test flowers were predicted in one call. The single flower example was online inference.

The same model can run in very different places.

• Cloud: Remote data centers with powerful hardware are used in cloud inference. This suits large models and heavy traffic.
• On-premises: A company’s own servers. Common when data rules or security policies forbid sending information outside.
• Edge: Hardware close to where data is created, such as a factory camera or a vehicle.
• On device: Directly on a phone or laptop, sometimes with no connection at all.

My Iris model ran on a regular laptop processor, so it counts as on-device inference.

There is a useful reality check here. Despite all the talk about edge devices, Deloitte expects most inference in 2026 to still run in data centers on costly, power-intensive chips rather than on cheap edge chips, according to its 2026 technology predictions. Edge is growing, but the cloud is not going anywhere.

The right chip depends on the model and the speed you need.

• CPU: The general-purpose processor in every computer. It handles small models easily, which is why my Iris project needed nothing more.
• GPU: Built with thousands of small cores that do math in parallel. This is the standard choice for deep learning and large language models.
• TPU: Google’s custom chips, designed specifically for the math inside neural networks.
• NPU: Low-power AI chips built into phones and newer laptops, designed for on-device inference.
• FPGA and ASIC: An FPGA is a chip that can be reconfigured for a specific job. An ASIC is built for one job only, which makes it very efficient but inflexible. Amazon’s Inferentia is an example of a chip purpose-built for inference.

This hardware is a big business now. Deloitte predicts the market for inference-optimized chips will pass $50 billion in 2026.

Inference is where users actually meet AI. Nobody experiences training. They experience the answer arriving quickly, or slowly, or wrongly.

The numbers show how much weight it carries. Deloitte predicts inference will account for roughly two-thirds of all AI compute in 2026, up from about a third in 2023 and half in 2025, in its TMT Predictions report.

Cost points the same way. AWS’s machine learning guidance says up to 90% of the infrastructure spend for developing and running ML applications is often on inference. That is a vendor figure and an upper limit, but it explains why companies care so much about running models efficiently.

The reason is simple. A model is trained once. It then answers requests for as long as it is live.

• Latency: Users feel inference as waiting time. A chatbot that takes ten seconds to start replying feels broken, even if its answers are excellent. Real-time uses such as fraud checks have much stricter limits.
• Cost: Every prediction uses compute, and the bill grows with usage. Cheaper hardware, smaller models, and batching all help.
• Scale: One user is easy. A million users sending requests at the same time is an engineering problem.
• Input mismatch: A model can look perfect in a notebook and fail in real use because live data looks different from training data. I hit this with my digit recognition project, which tested well but struggled on live webcam frames until I changed the training data to match real conditions. The Iris scaler rule is the small version of this same problem.
• Drift: The world changes over time. A model trained on last year’s behavior slowly gets worse, so deployed models need monitoring.

Four numbers matter most.

1. Latency is the time one prediction takes. 
2. Throughput is how many predictions the system handles per second. 
3. Cost per inference is what each prediction costs to run. 
4. Production accuracy is how well the model performs on real traffic, which often differs from test results.

For language models, teams also track time to first token, meaning how long before the answer starts appearing.

Before this project, I thought of machine learning as mostly training. Looking back at my Iris code changed that. The fit() line ran once. The predict() line is what a real user would trigger over and over.

If you are learning AI, I would suggest this exercise. Train any simple model, then write a few lines that predict a brand new input. Watch each step happen: preprocessing, prediction, and turning numbers back into an answer. It makes the idea far clearer than any definition.

Training builds the model. Inference is the part people actually use. As AI keeps spreading into everyday products, understanding that second half matters more every year, which is a theme I explored in my guide to the future of artificial intelligence.