AI.Cruz
Home/MLOps/Model Serving and Deployment

Model Serving and Deployment

MLOpsBy Updated

A model in a notebook is a prototype. A model that handles 10,000 requests per second — and rolls back instantly when something goes wrong — is a product.

Canary Rollout

Canary
10%
Stable
90%

Traffic Split

Serving Patterns

Online
<100ms REST/gRPC
Batch
Hours, large datasets
Streaming
Kafka/Kinesis events

Model Serving and Deployment

From trained artifact to production system handling real traffic safely.

Serving patterns

Once a model passes offline evaluation, it must serve predictions to real users. There are three fundamental serving patterns, each with different latency, throughput, and infrastructure requirements:

  • Online serving (real-time) — a REST or gRPC endpoint that returns a prediction for a single request in under 100ms. Used for fraud detection, content ranking, search ranking, personalisation. Requires low-latency feature retrieval (often from a feature store).
  • Batch inference — a scheduled job that runs predictions over a large dataset and writes results to storage. Used for next-day recommendation lists, risk scoring for all accounts overnight. High throughput, tolerant of hours of latency.
  • Streaming inference — predictions triggered by events on a message queue (Kafka, Kinesis). Used for real-time fraud detection on transactions, session-level personalisation. Latency between real-time and batch, better resource utilisation than pure real-time.

Containerisation

The gap between “works on my machine” and “works in production” is closed by packaging the model with its complete runtime in a Docker container. A containerised model image contains:

  • The model artifact (weights, parameters)
  • The inference code
  • The exact Python version and all library versions
  • Any preprocessing objects (scaler, encoder, tokeniser) fit on training data

The container is the deployment unit. Any environment that runs Docker — Kubernetes, ECS, Cloud Run, a VM — runs the model identically.

Canary deployments

A canary deployment routes a small percentage of production traffic (typically 1–10%) to a new model version while the stable version handles the rest. If the canary performs well on live traffic (latency, error rate, business metrics), traffic shifts incrementally: 10% → 25% → 50% → 100%. If it degrades, rollback is instant: redirect all traffic back to the stable version.

Canary deployments decouple model release from model exposure: a new version can be deployed to infrastructure before any real traffic touches it. The slider in the visualisation above illustrates this traffic split.

Shadow mode

In shadow mode, the new model runs in parallel with the production model on all live requests. Both receive the same inputs, but only the production model's outputs are returned to users. The new model's predictions are logged silently for comparison.

Shadow mode answers: “Would the new model have made different decisions on real traffic?” It catches edge cases invisible in offline evaluation without any user-facing risk. The cost: double the compute for the shadow period.

A/B testing vs canary

Canary and A/B testing both split traffic, but serve different purposes:

  • Canary — safety mechanism. Goal: detect regressions in the new version before full rollout. Traffic split is asymmetric (small % on new). Rollout progresses automatically if no regressions.
  • A/B test — measurement mechanism. Goal: measure the causal effect of the new model on a business metric. Traffic split is typically 50/50 or power-calculated. Requires a statistical significance test before concluding.

The feature store

Online serving requires sub-100ms feature retrieval. A feature store provides:

  • Offline store — historical feature values for training (BigQuery, S3 Parquet). Can reconstruct features as of any past point in time to prevent data leakage.
  • Online store — low-latency key-value store (Redis, DynamoDB) for serving. Features are pre-computed and cached.
  • Feature consistency — the same transformation code runs offline (training) and online (serving), eliminating training-serving skew.

Rollback strategy

Every production deployment needs a rollback path. The standard patterns:

  1. Blue-green deployment — two identical environments. Blue = current production. Green = new version. Switch router from blue to green. Rollback = switch back. Zero downtime, double infrastructure cost.
  2. Model Registry rollback — in MLflow or Vertex AI, demote the current production version and promote the previous one. Works when traffic routing is tied to registry state.
  3. Feature flag — toggle between model versions via configuration without redeployment. Near-instant, requires both models to be deployed simultaneously.

What to remember

  • Online serving targets <100ms; batch inference tolerates hours; streaming sits in between.
  • Containerise with exact runtime to eliminate environment divergence.
  • Canary deployments limit blast radius; shadow mode eliminates user-facing risk entirely during evaluation.
  • The feature store prevents training-serving skew by sharing transformation logic between training and serving.

Recall check

0 of …

Say the answer out loud before you reveal it.

  1. What are the three ML serving patterns and when do you use each?

  2. What is the difference between a canary deployment and A/B testing?

  3. What does shadow mode accomplish that canary deployment cannot?

  4. What problem does a feature store solve?

Cheat sheet

Model Serving and Deployment

Online serving <100ms, batch overnight, streaming event-triggered. Canary = safety (limit blast radius). Shadow mode = zero user risk. Feature store = same code offline and online. Blue-green = instant rollback.

MLOPS · AI.Cruz.in/mlops/model_serving_and_deployment.html