AWS ML Engineer Associate Prep Series — Part 3

Model Training and Tuning: Splits, Metrics, Overfitting, HPO, and Experiment Tracking

Part 2 covered SageMaker's core workflows, training jobs, built-in algorithms, Script Mode, and processing jobs. If you've been following along, you now know how to get data into SageMaker and run a training job.

Part 3 answers the questions that come right after: once your job runs, how do you know your model is any good? How do you tune it without burning through your budget? How do you keep experiments organized so you can actually compare results later?

This post covers five tightly connected topics: splitting data properly, measuring performance, preventing overfitting, hyperparameter optimization via SageMaker Automatic Model Tuning, and experiment tracking. They're tested as individual concepts on the exam, but in practice, they all feed into the same workflow.

1) Train/Validation/Test Split Strategy

Before any tuning begins, you need to decide how your data gets divided. This sounds basic, but the exam tests whether you know why three sets exist and what happens when you get it wrong.

Why three sets, not two

Training data teaches the model. Validation data guides tuning decisions: early stopping, hyperparameter selection, and architecture choices. Test data gives you an honest estimate of real-world performance.

If you tune hyperparameters against your test set, even indirectly, your test metrics become optimistic. The model has leaked information from the test into your decision-making. On the exam, this scenario shows up as "the team reports 99% accuracy on their held-out set, but production performance is significantly worse." The smoking gun is usually test data being used during tuning.

Common split ratios

• 70/15/15 or 80/10/10 for large datasets (100k+ records)
• 60/20/20 for moderate sizes
• Stratified splitting for classification: preserves class proportions across all three sets
• Random splitting for regression: no stratification needed unless the target distribution is heavily skewed

Special case: time series

Time series requires walk-forward validation. Standard random splits leak future information into the training window. SageMaker supports this through custom data channels in processing jobs, where you partition by time before training rather than relying on random sampling.

What the exam looks for

The scenario that tests this area is usually subtle: a question describes excellent training metrics but disappointing validation metrics. The root cause is often data leakage between splits, not a modeling problem. Recognizing the difference is the skill being tested.

Quick gut check: A team trains a classifier, achieves 99.5% accuracy on validation, but sees 83% accuracy on a held-out test set collected from the same distribution. The most likely cause isn't model architecture — it's leakage between training and validation during preprocessing (e.g., scaling before splitting). Always split first, transform second.

2) Measuring Model Performance

Accuracy alone is rarely enough. The exam tests whether you know which metric fits which business context.

Confusion matrix basics

Before any derived metric, understand the four quadrants:

• True Positives (TP): correctly predicted positive cases
• True Negatives (TN): correctly predicted negative cases
• False Positives (FP): predicted positive, actually negative (Type I error)
• False Negatives (FN): predicted negative, actually positive (Type II error)

Precision and recall

Precision = TP / (TP + FP). When you care about false positives — medical screening, drug testing, content moderation — precision is your primary signal. Every false positive costs time, money, or reputation.

Recall = TP / (TP + FN). When you care about false negatives — fraud detection, cancer diagnosis, safety systems — recall is what matters. Missing a positive case is worse than flagging a false alarm.

The F1 score is the harmonic mean of precision and recall: 2 × (P × R) / (P + R). Use it when you need a single number that balances both, but understand that it weighs precision and recall equally. If your business case doesn't treat them equally, F1 may not be the right summary.

ROC curve and AUC

ROC plots true positive rate (recall) against false positive rate across classification thresholds. AUC is the area under this curve. A score of 0.5 is random guessing; 1.0 is perfect.

AUC is useful for comparing classifiers holistically, but it can be misleading for highly imbalanced datasets where the negative class dominates. In those cases, P-R curves (precision vs recall) give a more honest picture. ROC can look optimistic when negatives vastly outnumber positives.

Regression metrics

Not all problems are classification. For regression:

• RMSE: penalizes large errors disproportionately. Use when big mistakes are much worse than small ones.
• MAE: treats all errors equally. More interpretable than RMSE.
• R²: proportion of variance explained by the model. Can be negative for models that perform worse than the mean baseline.

Quick gut check: You're building a fraud detection system where every missed fraudulent transaction costs $5,000 but false alarms only cost a $10 manual review. Which metric should you optimize? — Recall. False negatives are far more expensive than false positives. Maximizing recall captures more fraud cases, even if precision drops.

3) Overfitting Controls

Overfitting is when your model memorizes training data instead of learning generalizable patterns. The symptom is always the same: excellent training metrics, poor validation or test metrics.

L1 and L2 regularization

Both add a penalty term to the loss function, but they behave differently:

Practical note from the slides: L1's sparsity can make up for its computational inefficiency when you have 100 features and only 10 end up with non-zero coefficients. If you believe most of your features matter, L2 is the safer choice.

Dropout

Dropout randomly deactivates a fraction of neurons during each training pass. This prevents any single neuron from becoming overly relied upon. Common rates: 0.2–0.5 for hidden layers, lower for input layers. It's cheap to implement and consistently helps with deep networks.

Early stopping

Monitor validation loss during training and stop when it stops improving for N consecutive epochs (the "patience" parameter). This is the simplest overfitting control and usually the first one to try.

Vanishing and exploding gradients

In deep networks, gradients can shrink exponentially (vanishing) or grow exponentially (exploding) as they propagate backward through layers. Both prevent effective training. Common remedies include:

• ReLU activation instead of sigmoid/tanh — stays linear for positive values, reducing gradient decay
• Residual connections (ResNet) — skip connections let gradients flow directly through the network
• LSTM cells — gated memory paths preserve gradient flow in recurrent architectures
• Gradient clipping — caps gradient values to prevent explosions

Using SageMaker Debugger to detect overfitting at training time

SageMaker Debugger monitors training in real-time through built-in rules. Two that are directly relevant to overfitting:

• Overfit rule: compares validation and training loss. If the ratio of (val_loss − train_loss) / train_loss exceeds a threshold for a set number of steps, it fires. The default ratio threshold is 0.1.
• VanishingGradient rule: triggers when mean absolute gradient values drop below a threshold (default 1e-7). Early warning that learning has stalled.

You attach these rules when creating a training job, and they can trigger actions like stopping the job or sending an SNS notification. Debugger also generates a profiling report with system-level metrics (CPU bottlenecks, GPU memory pressure, I/O wait times) to help diagnose why training is slow, alongside why it's overfitting.

Note: AWS closed new customer access to SageMaker Debugger effective July 30, 2026. Existing customers can continue using it. The concept still matters for the exam since questions referencing Debugger were written before this date.

Bagging vs boosting and overfitting

From the slides: "Boosting generally yields better accuracy, but bagging avoids overfitting. Bagging is easier to parallelize." If you see a question asking to trade off between accuracy and overfitting risk, this distinction guides the answer. XGBoost (boosting) will generally outperform a random forest (bagging) on accuracy, but the random forest is harder to overfit.

4) Hyperparameter Optimization with SageMaker Automatic Model Tuning

Once you have clean splits, a reliable metric, and basic overfitting controls, the next question is: what hyperparameter values actually work best?

SageMaker Automatic Model Tuning (AMT) manages the entire search. You define the hyperparameters you care about, the ranges to explore, and the objective metric. SageMaker spins up training jobs, evaluates results, and returns the best configuration.

Search strategies compared

Key trade-off from the AWS docs: Bayesian optimization makes increasingly informed decisions but cannot massively scale due to its sequential nature. Random search can run the largest number of parallel jobs because subsequent jobs don't depend on prior results.

Hyperband should only be used for iterative algorithms that publish results at different resource levels, for example, a neural network that reports accuracy after every epoch.

Warm start

Warm start uses one or more previous tuning jobs as a starting point. There are two types:

• IDENTICAL_DATA_AND_ALGORITHM: same training data and algorithm image as parent jobs. You can adjust ranges and increase the number of training jobs. The new tuning job tracks OverallBestTrainingJob across all parent runs.
• TRANSFER_LEARNING: allows different input data, hyperparameter ranges, and algorithm versions. Best when your dataset or algorithm has evolved, but the tuning problem is structurally similar.

Restrictions: max 5 parent jobs, all must be in a terminal state (Completed, Stopped, or Failed). The objective metric must be the same across parent and child. Total tunable plus static hyperparameters must remain constant. Warm start is not recursive, if Job3 uses Job2 as a parent, and Job2 was itself a warm start of Job1, Job1's knowledge is not carried forward unless explicitly listed.

Early stopping in tuning jobs

When enabled (set TrainingJobEarlyStoppingType to Auto), SageMaker evaluates each training job after every epoch:

1. Compute the objective metric for the current job
2. Compute the running average of the objective metric for all previous jobs at the same epoch
3. Compute the median of those running averages
4. If the current job's metric is worse than the median, SageMaker stops it early

This saves compute and helps avoid overfitting by not wasting resources on unpromising configurations. Built-in algorithms that support early stopping include XGBoost, Linear Learner (using objective_loss), Image Classification, and Object Detection. Custom algorithms need to emit per-epoch metrics explicitly.

Code example: HyperparameterTuner with warm start

from sagemaker.tuner import HyperparameterTuner, IntegerParameter, ContinuousParameter
from sagemaker.pytorch import PyTorch
from sagemaker.core.shapes import HyperParameterTuningJobWarmStartConfig
from sagemaker.train.tuner import WarmStartTypes

estimator = PyTorch(
    entry_point='train.py',
    instance_type='ml.p3.2xlarge',
    instance_count=1,
    framework_version='2.0',
    py_version='py39',
    hyperparameters={
        'epochs': 20,
        'batch-size': 64
    }
)

hyperparameter_ranges = {
    'learning-rate': ContinuousParameter(0.0001, 0.1, scaling_type='Logarithmic'),
    'dropout': ContinuousParameter(0.1, 0.5),
    'layers': IntegerParameter(2, 8)
}

warm_start_config = HyperParameterTuningJobWarmStartConfig(
    warm_start_type=WarmStartTypes.IDENTICAL_DATA_AND_ALGORITHM,
    parents=['previous-tuning-job-name']
)

tuner = HyperparameterTuner(
    estimator,
    objective_metric_name='validation:accuracy',
    objective_type='Maximize',
    hyperparameter_ranges=hyperparameter_ranges,
    max_jobs=20,
    max_parallel_jobs=2,
    strategy='Bayesian',
    early_stopping_type='Auto',
    warm_start_config=warm_start_config
)

tuner.fit({'training': 's3://bucket/train', 'validation': 's3://bucket/val'})

Notice the logarithmic scale on learning-rate — This matches the AWS best practice for parameters that span multiple orders of magnitude. Linear scaling would spend 90% of the training budget on values between 0.01 and 0.1 while barely exploring 0.0001 to 0.01.

Best practices summary from AWS docs

• Limit to 30 tunable hyperparameters max; fewer converge faster
• Use log scales for parameters spanning multiple orders of magnitude (learning rate, regularization)
• Limit concurrent jobs when using Bayesian optimization — each run informs the next, so too much parallelism wastes information
• For small training jobs, random search or Bayesian optimization is fine; for large iterative jobs, Hyperband saves significant compute
• Specify a random seed for reproducibility, especially with random search or Hyperband (up to 100% reproducibility)

5) Experiment Tracking with SageMaker Experiments

As your tuning jobs multiply, keeping track of what was tried becomes impossible without a structured approach. SageMaker Experiments solves this by automatically capturing parameters, configurations, and results for every training run.

How it works

SageMaker Experiments organizes your ML jobs into runs, which can be grouped into experiments for higher-level organization. Each run automatically tracks:

• Input data channels and S3 locations
• Hyperparameters (both tunable and static)
• Algorithm and framework versions
• Instance type and count
• Objective metrics logged during training
• Output model artifacts

Practical workflow

Create an experiment at the start of a project, and every training job, tuning job, or processing job can be associated with it. The SageMaker Studio UI then gives you a side-by-side comparison view where you can select runs, plot metrics across steps, and identify which configuration produced the best result.

This is especially useful during HPO. Instead of maintaining a spreadsheet of "learning rate 0.01 / dropout 0.3 → accuracy 0.87," the experiment captures it automatically, including artifact lineage, so you can trace a deployed model back to the exact training run and data version that produced it.

Integration with the Model Registry

Experiments feed directly into the Model Registry. When you register a model version, you can link it to the experiment run that produced it. This creates an audit trail from raw data through training through deployment, a pattern that shows up directly in MLOps questions (covered in Part 6).

Current API note

The SageMaker Experiments Python SDK is currently available in Studio Classic. AWS recommends migrating to MLflow integration for the new Studio experience. For the exam, the concept of experiment tracking, organizing, comparing, and tracing runs matters more than the specific SDK calls.

6) Putting It All Together

Here's how these five pieces connect in a realistic workflow:

You start with raw data and split it into training, validation, and test sets using a processing job, with stratified sampling if needed. The test set gets locked away.

You establish a baseline metric using a built-in algorithm, say XGBoost, and compute precision, recall, and F1 from the validation results. This gives you a number to beat.

Before moving to a custom model, you add overfitting controls: early stopping on validation loss, L2 regularization on the XGBoost parameters, and, if you have existing Debugger access, the Overfit rule watching training in real time.

You launch an Automatic Model Tuning job with Bayesian optimization, tuning learning rate, max depth, and regularization. Every training job is automatically captured by SageMaker Experiments, so you can compare results side by side in Studio.

The best configuration from HPO gets promoted to a training job on the full training set (training + validation combined), evaluated once against the held-out test set, and registered in the Model Registry with a link back to the experiment run.

From there, Part 4 picks up: deployment patterns, real-time endpoints, batch transform, and rollout strategies.

What to remember for the exam

1. Train/validation/test strategy: split first, transform second. Never tune hyperparameters against test data. Use stratified splits for classification, walk-forward for time series.
2. Metrics: match the metric to the business cost. Precision (cost of false positives), recall (cost of false negatives), F1 (balanced), AUC (overall classifier comparison), RMSE/MAE (regression). The exam rewards knowing why, not just what.
3. Overfitting controls: L2 is the safer default (all features weighted), L1 for sparsity. Early stopping is the simplest first step. Debugger's Overfit and VanishingGradient rules catch problems during training, not after.
4. HPO strategies: Bayesian for efficient sequential exploration (limited parallelism). Random for maximum parallelism. Hyperband for iterative algorithms where early stopping saves significant compute. Grid for small categorical spaces. Warm start (IDENTICAL_DATA_AND_ALGORITHM or TRANSFER_LEARNING) to resume or extend previous tuning jobs.
5. Experiment tracking: SageMaker Experiments captures runs, parameters, and metrics automatically. It feeds into the Model Registry for full lineage from data to deployment. The architecture matters more than the SDK calls.

Practice questions

Work through these. The goal is to reason through the trade-off, not just recall an answer.

1. A team reports 99.2% training accuracy and 98.9% validation accuracy, but 82% accuracy on a newly collected test set from the same distribution. What's the most likely issue, and what should they check first? — Data leakage between splits. Check whether preprocessing (scaling, encoding, imputation) was applied before or after the split. If before, information from the test set leaked into the training statistics.

2. You're building a model to detect rare manufacturing defects. A false negative means a defective product ships to a customer. A false positive means a good product is flagged for manual review. Which metric should you optimize, and why? — Recall. False negatives (missed defects) have the higher business cost. Maximizing recall captures more actual defects, even if it increases false positives.

3. Your training job uses a deep neural network, and you notice validation loss stopped improving at epoch 20 while training loss continues to decrease. Name two overfitting controls that should have been applied and one Debugger rule that would have caught it. — Early stopping with patience of 5-10 epochs, dropout (0.3-0.5), and the Debugger Overfit rule which compares the ratio of validation-to-training loss.

4. A tuning job with Bayesian optimization is taking too long despite using 20 parallel training jobs. What's the likely cause, and what strategy would be more appropriate? — Bayesian optimization is sequential; each run informs the next. Running 20 parallel jobs wastes that information because many jobs launch before any results return. Reduce parallelism to 2-3 concurrent jobs, or switch to random search if maximum parallelism is required. (Per AWS docs: "Bayesian optimization cannot massively scale" and "random search is able to run the largest number of parallel jobs.")

5. You completed an initial HPO run with 30 training jobs and want to continue tuning with a narrower range around the best configuration. You're using the same dataset and algorithm. Which warm start type should you use, and what restriction applies to the number of parent jobs? — IDENTICAL_DATA_AND_ALGORITHM same data and image, refined ranges. Max 5 parent tuning jobs, all must be in a terminal state, and the objective metric must remain the same.

Closing

The thread that ties this part together: good model training isn't just about writing a script and calling fit(). It's about knowing how to measure success honestly, keeping overfitting in check, spending your compute budget on the right hyperparameter experiments, and tracking everything so you can compare and reproduce later.

Part 2 answered, "How do I run a training job on SageMaker?" Part 3 answered, "How do I know it's a good one?" Part 4 will answer "how do I put it in production?" — deployment patterns, real-time endpoints, serverless inference, batch transform, and rollout strategies.