AWS ML Engineer Associate Prep Series — Part 4

Deployment Patterns: Real-Time, Serverless, Async, Batch, Multi-Model, Pipelines, and Rollout Strategies

Part 3 covered model training and tuning — how to split data properly, measure performance honestly, control overfitting, run hyperparameter optimization, and track experiments. If you've been following along, you now know how to build and tune a model.

Part 4 answers the question that comes next: how do you put that model in production?

This is the domain where the exam shifts from "can you train a model?" to "can you deploy it safely, cost-effectively, and at the right latency?" The scenarios always describe a concrete workload and ask which deployment option fits. Get the pattern matching right and these questions become straightforward. Get it wrong and you'll burn time second-guessing yourself.

This post covers eight tightly connected topics: real-time endpoints, serverless inference, asynchronous inference, batch transform, multi-model endpoints, rollout strategies, inference pipelines, and SageMaker Neo for edge deployment. In practice, you rarely use just one — a real system combines several.

1) Real-Time Endpoints: The Default, and When It's Wrong

This is where most people start, and for good reason. A real-time endpoint is a persistent, managed HTTPS endpoint that serves predictions on demand with low latency. You provision instances, SageMaker keeps them running, and your application sends requests whenever it needs an answer.

Model vs endpoint: they're not the same thing

A SageMaker model is a reference to your trained artifact — it points at the model file in S3 and the inference code image in ECR. An endpoint is the live infrastructure that loads that model and serves predictions. One model can power multiple endpoints (staging, production, per-region), and you can swap the model behind an endpoint without tearing down the infrastructure.

The lifecycle: train → artifacts land in S3 → create model resource → endpoint config (instance type, count, variants) → create endpoint (SageMaker provisions instances) → InvokeEndpoint.

Elastic Inference: GPU acceleration without GPU pricing

If your model needs GPU-level throughput but you don't want to pay for a full GPU instance, Elastic Inference attaches fractional GPU acceleration to a CPU instance. It's the right answer when a scenario describes deep learning inference with cost sensitivity and moderate throughput requirements. It's inference-only — not for training.

Auto Scaling: match capacity to traffic

SageMaker Auto Scaling dynamically adjusts the number of instances backing an endpoint based on metrics like InvocationsPerInstance or CPU utilization. Without it, you're paying for peak capacity around the clock. The exam likes scenarios where traffic is predictable during business hours but drops overnight — that's the Auto Scaling signal.

SageMaker Inference Recommender

Inference Recommender automates load testing to recommend the best instance type for your model. Two modes: Instance Recommendations (runs load tests on suggested types, ~45 min) and Endpoint Recommendations (custom load test with your traffic patterns and latency targets, ~2 hours). The output includes CostPerInference, MaximumInvocations, ModelLatency, and the recommended InstanceType. When a scenario asks "how do you determine the most cost-effective instance type for this model?", this is the tool.

Multi-AZ: don't deploy a single instance to production

SageMaker automatically attempts to distribute instances across Availability Zones, but only if you have more than one instance. Deploy at least two instances per production endpoint and configure your VPC with at least two subnets in different AZs.

The cost trap

Endpoints charge for instance hours regardless of traffic. An idle endpoint costs the same as a busy one. The exam will give you a scenario where someone deploys a model for occasional use and gets a surprising bill — the answer is either Auto Scaling, batch transform, or serverless inference.

Quick gut check: A team deploys a fraud detection model that gets ~10,000 requests per second during business hours and drops to ~50 overnight. They're using a single ml.c5.xlarge instance. What's missing? — Auto Scaling. Without it, the single instance becomes a bottleneck during peaks, and if they over-provision for peaks, they pay for idle capacity overnight. Auto Scaling with a target tracking policy on InvocationsPerInstance solves both.

2) Serverless Inference: Pay Per Invocation, Scale to Zero

Serverless inference is the answer when your traffic pattern looks like "sometimes a lot, sometimes nothing" and you can tolerate a cold start.

You specify your inference container, a memory requirement, and concurrency limits. AWS handles the rest, provisioning, patching, scaling. When there are no requests, it scales down to zero and you stop paying. When a request arrives, it spins up in seconds.

Cold starts are the headline limitation. If your application needs sub-100ms latency on every request, a warm real-time endpoint is the better choice. If a few seconds of startup on the first request after an idle period is acceptable, serverless gets you the same predictions without paying for an always-on instance.

CloudWatch metrics to know: ModelSetupTime (cold start duration), Invocations (request count), MemoryUtilization. Billing is based on invocation count and memory-time consumed, you pay for actual usage, not provisioned capacity.

When it shines:

• Dev and test environments where you deploy a model, test it, and move on
• Low-volume production APIs (a few hundred requests per day)
• Intermittent processing where you trigger inference on a schedule
• Any scenario where the words "idle periods" or "uneven traffic" appear

Quick gut check: A team has a text classification model used by an internal tool that gets maybe 200 requests per day, mostly during the morning. They deployed a real-time endpoint on ml.m5.large and got a $120/month bill for near-zero utilization. What should they switch to? — SageMaker Serverless Inference. It scales to zero during idle periods, and at 200 invocations per day, they'd pay pennies instead of $120.

3) Asynchronous Inference: Large Payloads, Queued Processing

Async inference solves a distinct problem: what if you need near-real-time latency but your payloads are huge?

It queues incoming requests, supports payload sizes up to 1 GB, far larger than real-time or serverless. Use it for large inputs (high-res images, long audio files) where processing takes long enough that a synchronous request would time out, but you still need results within minutes rather than hours.

The key distinction from batch transform: async still has a near-real-time expectation. You send a request, and you expect a response back within minutes, not "whenever the job finishes overnight." The key distinction from real-time: the payload is too big or the processing too slow for a synchronous call.

Quick gut check: You need to run inference on high-resolution satellite imagery, each image ~500 MB, and results are needed within 2–3 minutes for a downstream dashboard. Real-time, serverless, async, or batch? — Asynchronous Inference. The payload size rules out real-time and serverless, and the near-real-time requirement rules out batch transform.

4) Batch Transform: Score Everything at Once

Batch transform is the simplest deployment option and often the most overlooked. You provide a dataset in S3, it runs inference on the entire thing, and writes predictions back to S3. No endpoint. No latency concerns. No idle costs.

When to use it:

• Periodic scoring runs (daily risk scores, weekly churn predictions)
• Backfilling predictions on historical data
• Pre-computing embeddings or features for a downstream system
• Any scenario where predictions don't need to be returned in real time

The clue is always in the latency requirement. If the scenario says "the user clicks a button and expects a result," that's not batch transform. If it says "every Monday morning, score the entire customer base," that is.

One detail worth knowing: batch transform splits your input across multiple instances when you provide multiple input files. A single large CSV gets processed sequentially; a directory of many smaller files gets parallelized across instances. If a scenario asks how to speed up a batch transform job, the answer is "split your input into multiple files."

Quick gut check: A marketing team needs to generate product recommendations for 10 million customers every Sunday night. Results feed into an email campaign that goes out Monday morning. Real-time endpoint, serverless, or batch transform? — Batch transform. There's no real-time requirement (Sunday night is the window), the dataset is large enough to justify offline processing, and the output is used in batch anyway. A real-time endpoint would sit idle for 6.9 days a week.

5) Multi-Model Endpoints: Share Infrastructure Across Models

Multi-model endpoints (MME) solve a specific cost problem: what if you have hundreds or thousands of models, but only a few are actively used at any given moment?

A multi-model endpoint hosts multiple models on shared instance infrastructure. Models are stored in S3 and loaded into instance memory on demand. When a request arrives specifying a target model, SageMaker fetches that model from S3, loads it into memory, and runs inference. If memory is full, the least recently used (LRU) model gets evicted to make room.

Without MME, hosting 1,000 models means 1,000 separate endpoints — insanely expensive. With MME, those 1,000 models share a pool of instances, and you only pay for the aggregate capacity you provision. The trade-off is latency on cache misses: if a model isn't loaded when a request arrives, there's a delay while it's fetched from S3.

Best-fit scenarios:

• Per-tenant models in a SaaS platform (one model per customer)
• A/B testing many model variants simultaneously
• Model registries where any version might need to be served on demand

When MME is the wrong call: if all your models are used simultaneously at high throughput — for instance, if all 1,000 models receive requests every second, MME buys you nothing. The cache thrashing (constant load/evict cycles) would actually hurt latency. In that case, separate endpoints with Auto Scaling per model is the right answer.

Quick gut check: A SaaS platform builds a per-customer churn model — 500 customers, 500 models. Most customers log in once a week to check their dashboard. They're considering 500 separate ml.c5.large endpoints at ~$60/month each = $30,000/month. What's the alternative? — Multi-model endpoints. Host all 500 models on a handful of instances. The LRU cache handles the access pattern perfectly since each model is used briefly once a week.

6) Rollout Strategies: How Traffic Gets Shifted

Once your model is deployed, the next question is how you safely update it. SageMaker provides a framework for this built around Production Variants and Deployment Guardrails.

Production Variants

A SageMaker endpoint can host multiple production variants, different models or different versions of the same model, and split traffic between them. Each variant gets a weight, and SageMaker routes that percentage of inference requests to it.

This is the mechanism underneath all rollout strategies. You don't need separate endpoints to compare models; you configure variants on one endpoint.

Blue/Green with traffic shifting

Three traffic-shifting patterns, all under the blue/green deployment umbrella:

All at once: Deploy the new model (green), shift 100% of traffic to it immediately, monitor, then terminate the old fleet (blue). Simple, fast, and risky — there's no safety net if the green model is broken.

Canary: Shift a small percentage of traffic (e.g., 5%) to the green model, monitor for errors and metric degradation, then gradually ramp up. This is the right answer when a scenario mentions "minimizing blast radius" or "testing with a small subset of real users first."

Linear: Shift traffic in evenly spaced increments over a specified time period — e.g., 20% every 5 minutes until 100%. This gives you observation windows at each step, useful when you want a controlled, predictable rollout.

Auto-rollbacks

Deployment Guardrails can be configured to automatically roll back when CloudWatch metrics breach a threshold. If error rates spike or latency balloons after a canary shift, SageMaker reverts traffic to the previous version without manual intervention.

Shadow tests

Shadow testing evaluates a new model variant against the currently deployed model without any user impact. Production traffic is mirrored to the shadow variant, and predictions are logged but never returned to users. You monitor the shadow in the SageMaker console, compare performance, and decide when to promote it to a live variant.

This is distinct from A/B testing. A/B testing routes real traffic and compares real outcomes. Shadow testing routes mirrored traffic and compares predictions offline. The exam expects you to know the difference.

Quick gut check: A team wants to deploy a new fraud detection model. The cost of false negatives is extremely high (missed fraud), so they want to verify the new model catches at least as much fraud as the current one before any real traffic reaches it. A/B test, canary, or shadow? — Shadow test. Mirror production traffic to the new model, compare fraud detection rates offline, and only promote when you're confident. Neither A/B (real traffic) nor canary (real traffic, just small) eliminates user impact completely.

7) Inference Pipelines: Chain Processing on One Endpoint

Inference pipelines solve a problem that shows up in real ML systems constantly: preprocessing and postprocessing.

An inference pipeline is a linear sequence of 2–15 Docker containers deployed as a single endpoint. A common arrangement: a preprocessing container (feature engineering — encoding, scaling, imputation) feeds into a prediction container, which feeds into a postprocessing container (formatting, filtering, metadata joining).

Spark ML and scikit-learn are explicitly supported. Spark ML pipelines can be serialized into MLeap format for deployment without rewriting in another language. Pipelines work for both real-time inference and batch transforms, and they keep preprocessing logic modular, versioned, and independently deployable.

The exam signal: a scenario describes needing to apply the same Spark transformations used in training during inference, and the answer is an inference pipeline.

Quick gut check: A team trains a model using Spark ML for feature engineering (one-hot encoding, PCA) followed by XGBoost for prediction. At inference time, the same feature transformations must be applied before the XGBoost model makes predictions. Standalone endpoint with custom script, or inference pipeline? — Inference pipeline. It lets you chain the Spark ML preprocessing container and XGBoost prediction container on a single endpoint, guaranteeing the same transformations at training and inference time.

8) SageMaker Neo: When the Model Leaves the Cloud

Neo is worth knowing because it shows up in scenarios about edge deployment.

Neo compiles and optimizes trained models for specific target hardware — ARM, Intel, NVIDIA, AWS Inferentia (inf2), and Graviton3 processors. The compilation step reduces model size and improves inference speed without changing the model's behavior.

The deployment path:

1. Train the model in SageMaker
2. Compile with Neo for the target device architecture
3. Deploy via one of two paths:
   • HTTPS endpoint hosted on a SageMaker instance (must be the same instance type used for compilation)
   • AWS IoT Greengrass for deployment to actual edge devices — factory floors, autonomous vehicles, remote sensors

The sustainability angle is also worth knowing: efficient silicon (Inferentia, Graviton3) combined with Neo compilation can dramatically reduce the energy footprint of inference workloads.

Putting It All Together

Here's how these pieces fit together in a realistic deployment workflow:

You train and register a model, then run Inference Recommender to find the right instance type. For initial testing, deploy to a serverless endpoint — no infrastructure to manage, scales to zero when done. Once validated, create a production endpoint with at least two instances across multiple AZs and Auto Scaling on InvocationsPerInstance. Set up a shadow test to compare the new model against production without user impact, then configure production variants for a canary deployment with auto-rollback.

For weekly batch scoring, use batch transform. For per-customer models across hundreds of tenants, use multi-model endpoints to share infrastructure. For Spark ML preprocessing, chain an inference pipeline. For factory-floor edge devices with intermittent connectivity, compile with SageMaker Neo and deploy through IoT Greengrass.

What to remember for the exam

1. Match deployment pattern to latency, payload, and traffic profile: real-time (low latency, steady traffic, pay for instances), serverless (intermittent traffic, tolerate cold starts, pay per invocation), async (large payloads up to 1 GB, queued, near-real-time), batch (offline, whole dataset, S3 → S3). The exam scenario always gives you the signal — read the latency and traffic pattern carefully.
2. Rollout strategies are about risk tolerance: all-at-once (no safety net), canary (small % first, minimize blast radius), linear (evenly spaced steps), shadow (mirror traffic, zero user impact). Production Variants with variant weights is the mechanism underneath all of them, and auto-rollback with CloudWatch metric thresholds is the safety mechanism.
3. Multi-model endpoints solve the "many models, infrequent use" problem: models load on demand from S3, LRU eviction manages memory, and you pay for shared instances, not per-model endpoints. Know the cache-miss latency trade-off and recognize when all models being hot simultaneously makes MME the wrong call.
4. Inference pipelines chain 2–15 containers for pre/post processing: Spark ML (serialized to MLeap) and scikit-learn are explicitly supported. Use them when the same feature transformations from training must run during inference, and deploy the same pipeline for both real-time and batch.
5. Right-size before deploying: Inference Recommender runs automated load tests (Instance: ~45 min, Endpoint: ~2 hours) and recommends cost-optimal instance types. Elastic Inference adds fractional GPU to CPU instances for cost-sensitive deep learning inference. SageMaker Neo compiles models for edge hardware (Inferentia, Graviton3, ARM, NVIDIA) and deploys through IoT Greengrass for true edge scenarios.

Practice questions

Work through these before Part 5. The goal is to reason through the trade-off.

1. A team deploys a real-time endpoint on ml.c5.2xlarge for a customer-facing recommendation API. Traffic is 2,000 requests/second during business hours and drops to ~20 overnight. Their monthly bill is $350. They want to cut costs without sacrificing daytime latency. What two changes should they make? — Add Auto Scaling to reduce instance count during overnight low-traffic periods. Also run SageMaker Inference Recommender (Endpoint Recommendations mode) to verify ml.c5.2xlarge is the right size — a smaller instance type might handle 2,000 RPS just fine, or Elastic Inference on a cheaper CPU instance could handle the same throughput.

2. A medical imaging company runs inference on CT scans, each file ~800 MB. Processing takes about 90 seconds per scan, and results feed into a radiologist's queue. They need results within 5 minutes. Which inference option fits, and why are the others wrong? — Asynchronous Inference. Payload size (800 MB) rules out real-time (~6 MB limit) and serverless (~4 MB limit). Batch transform might work but the 5-minute SLO requires near-real-time latency that batch can't guarantee. Async queues the large payload, processes within 90 seconds, and returns results well within the 5-minute window.

3. A team is rolling out a new version of a fraud detection model. The current model processes 50,000 transactions per hour, and false negatives cost the company ~$50,000 per missed fraud case. They want to verify the new model performs at least as well as the current one before any real traffic hits it. Shadow test, canary, or A/B test — which one, and why? — Shadow test. It mirrors production traffic to the new model and logs predictions without returning them to users — zero blast radius. A canary (even 5% = 2,500 transactions/hour) still exposes real users to potential missed fraud. A/B testing has the same problem: real traffic reaches the new model. Given the $50K cost per false negative, zero user impact is the right call.

4. A SaaS platform builds a per-tenant product recommendation model. They have 200 tenants, each with their own model. Each tenant's model is only queried when that tenant's users are active — typically a few hours per week. They've been running 200 separate real-time endpoints at ~$24,000/month. What should they switch to, and what's the one scenario where that switch would backfire? — Multi-model endpoints. Host all 200 models on a shared pool of instances. The LRU caching works perfectly here because models are used intermittently — most are cold at any given moment, and the hot ones stay loaded. The backfire scenario: if all 200 tenants had simultaneous peak traffic (e.g., a Black Friday event), the constant model loading and eviction would thrash the cache and kill latency. In that case, separate endpoints or a larger MME instance pool would be needed.

5. An ML pipeline trains a model using Spark ML for feature engineering (scaling, encoding, PCA) followed by a PyTorch neural network. At inference time, the same Spark transformations must be applied before the PyTorch model runs. The team is considering wrapping both in a single custom Docker container. Is there a better approach? — An inference pipeline. Chain the Spark ML container (serialized to MLeap) with the PyTorch prediction container as a 2-container pipeline on a single endpoint. This keeps the Spark layer independently versioned, reusable across different models, and avoids marrying two frameworks in one Docker image. The same pipeline also works for batch transforms.

Closing

Deployment patterns are where the exam rewards architecture judgment over memorization. The scenarios test whether you understand the trade-offs: latency vs cost, safety vs speed, payload size vs pattern choice.

Once you can look at a set of requirements and immediately map them to the right inference option, the right rollout strategy, and the right endpoint architecture, this domain becomes one of the most straightforward on the exam.

Part 5 will cover Feature Engineering on AWS — SageMaker Data Wrangler, Feature Store, online vs offline features, and point-in-time correctness.