Training Trains Internship
Student login
← MLOps learning path
MLOps · Module 5 of 10

Model deployment patterns

Serve predictions in batch or real time.

📖 Key ideas

  1. Batch inference scores many records on a schedule.
  2. Online (real-time) inference serves predictions via an API.
  3. Streaming inference scores events as they arrive.
  4. Edge deployment runs on devices.
  5. Serving tools: FastAPI, BentoML, KServe, TorchServe, SageMaker endpoints.
💡 Real-life example: Batch inference is printing tomorrow's newspaper overnight; online inference is answering a phone call instantly.
Batch inferenceScoring data in bulk
Online inferenceReal-time API predictions
KServeModel serving on Kubernetes
LatencyTime to return a prediction