Inference
The process of running a trained AI model on new inputs to produce predictions or generated outputs, as opposed to the training phase.
What Is Inference?
In machine learning, inference is the operational phase where a trained model is used to process new inputs and produce outputs — a prediction, a generated text, a classification. It is distinct from training, where model parameters are updated using gradient descent. Once a model is trained, inference is all that users and most production systems ever do.
For LLMs, inference is computationally intensive. Generating each token requires a forward pass through all model layers. Techniques like KV caching (reusing key-value attention computations from the context) and speculative decoding (using a small draft model to predict multiple tokens, then verifying with the large model) significantly reduce inference latency and cost.
Why It Matters
Inference cost dominates AI product economics. As usage scales, optimising inference through quantisation, batching, and hardware selection can cut costs by 10–100x. Understanding inference is critical for anyone building AI-powered products at scale.
Real-World Examples
ChatGPT Query
Every time you send a message to ChatGPT, you are triggering inference — the model processes your input and generates a response token by token.
Image Classification
A computer vision model runs inference on each uploaded photo to detect objects, faces, or medical anomalies.