Home/MLOps/MLflow Experiment Tracking

MLflow Experiment Tracking

MLOpsBy AI.CruzUpdated 1 October 2026

Without experiment tracking, ML research is folklore. MLflow turns every training run into a reproducible, comparable, promotable artifact.

Filter Runs

Best Run

run_2

Experiment Runs

MLflow Concepts

Experiment

Named group of runs

Run

One training execution

Registry

Staging → Production

MLflow Experiment Tracking

Reproducibility starts with tracking every run, every parameter, every result.

The problem without tracking

Without experiment tracking, ML research quickly becomes folklore: “I think I got 87% accuracy with learning rate 0.001 and dropout 0.3 — or was it 0.5?” Model development collapses into a long trail of notebooks with names like model_final_v3_ACTUAL_FINAL.ipynb, with no reliable way to recover the exact conditions that produced the best result.

Experiment tracking solves this by logging every training run with its complete context: hyperparameters, dataset version, code version, environment, metrics, and output artifacts. Any run can be replicated exactly. Any two runs can be compared on equal footing.

MLflow components

MLflow is an open-source platform built around four core components:

MLflow Tracking — logs parameters, metrics, tags, and artifacts (models, plots, data samples) for each run. Provides a UI and REST API for querying.
MLflow Projects — packages ML code in a reproducible format (conda or Docker environment spec). Any project can be reproduced with mlflow run.
MLflow Models — a standard format for packaging ML models for multiple deployment targets (REST API, batch, Spark UDF, etc.).
MLflow Model Registry — a centralised model store with staging, production, and archived states. Handles model lineage, versioning, and promotion workflows.

Core tracking concepts

Every logged item in MLflow belongs to a hierarchy:

Experiment — a named collection of runs for a specific ML task. “fraud-detection-v2” is an experiment.
Run — a single execution of training code. Contains parameters (lr=0.001), metrics (val_loss=0.234), tags (developer=alice), and artifacts (model.pkl).
Parameter — a key-value pair set before the run (hyperparameter, dataset path). Immutable after logging.
Metric — a numeric value that can be updated during a run (loss per epoch). MLflow stores the full history.
Artifact — any file output: model binaries, feature importance plots, confusion matrices, preprocessor objects.

Autologging

For popular frameworks (scikit-learn, XGBoost, PyTorch Lightning, TensorFlow), MLflow autolog captures common parameters and metrics automatically with a single call:

mlflow.sklearn.autolog()
with mlflow.start_run():
model.fit(X_train, y_train)

This logs estimator parameters, feature importances, the fitted model itself, train/test metrics, and a confusion matrix — all without any explicit log calls.

The Model Registry

The Model Registry separates the training concern (which run produced the best model?) from the deployment concern (which model version is in production?). The typical workflow:

Data scientists run experiments, logging models to MLflow.
The best model is registered: mlflow.register_model(run_uri, “FraudDetector”).
The model enters Staging for validation (shadow mode, A/B test, integration test).
After validation, it is promoted to Production via UI or API.
The previous Production version is archived, not deleted.

This creates a complete audit trail: every model in production is traceable to the exact training run, dataset, code commit, and environment that produced it.

Comparing runs

MLflow's comparison UI plots metrics across runs. For programmatic comparison, the MLflow client API provides a search interface:

runs = mlflow.search_runs(
filter_string=“metrics.val_accuracy > 0.85”,
order_by=[“metrics.val_loss ASC”]
)

This returns a pandas DataFrame of all matching runs with their parameters and metrics, enabling automated model selection pipelines.

Alternatives to MLflow

Tool	Strengths	Considerations
MLflow	Open source, self-hostable, broad framework support	UI less polished than cloud tools
Weights & Biases	Excellent visualisation, team collaboration	SaaS pricing, data leaves your infra
Neptune.ai	Strong metadata management	Smaller ecosystem
Comet ML	Strong NLP/CV integrations	SaaS-first

What to remember

MLflow tracks parameters, metrics, artifacts, and code version per run.
The Model Registry manages the path from training run to production deployment.
Autologging removes the need for explicit log calls for major frameworks.
Any run is reproducible: code + data + parameters + environment are all stored.

Experiment Tracker (stdlib)

import json
import math
import time
from datetime import datetime

# Minimal experiment tracker using only stdlib
# (MLflow itself is not available in the browser)

class SimpleTracker:
    def __init__(self, experiment_name):
        self.experiment = experiment_name
        self.runs = []
        self._active = None

    def start_run(self, run_name=None):
        self._active = {
            "run_id": "run_" + str(len(self.runs) + 1),
            "run_name": run_name or ("run_" + str(len(self.runs) + 1)),
            "start_time": datetime.now().isoformat(),
            "params": {},
            "metrics": {},
            "status": "RUNNING",
        }
        return self

    def log_param(self, key, value):
        self._active["params"][key] = value

    def log_metric(self, key, value):
        self._active["metrics"][key] = value

    def end_run(self):
        self._active["status"] = "FINISHED"
        self.runs.append(self._active)
        self._active = None

    def best_run(self, metric, maximize=True):
        fn = max if maximize else min
        return fn(self.runs, key=lambda r: r["metrics"].get(metric, float("-inf")))

tracker = SimpleTracker("iris-classification")

# Simulate 5 experiment runs
configs = [
    {"lr": 0.001, "depth": 3, "dropout": 0.2},
    {"lr": 0.01,  "depth": 5, "dropout": 0.3},
    {"lr": 0.01,  "depth": 7, "dropout": 0.3},
    {"lr": 0.1,   "depth": 5, "dropout": 0.4},
    {"lr": 0.001, "depth": 9, "dropout": 0.2},
]

for cfg in configs:
    tracker.start_run()
    tracker.log_param("learning_rate", cfg["lr"])
    tracker.log_param("max_depth", cfg["depth"])
    tracker.log_param("dropout", cfg["dropout"])

    # Simulated metrics (realistic spread)
    acc = 0.82 + cfg["depth"] * 0.01 - abs(cfg["lr"] - 0.01) * 0.5
    acc = min(0.96, max(0.70, acc + (hash(str(cfg)) % 5) * 0.005))
    val_loss = round(1 - acc + 0.05, 4)

    tracker.log_metric("val_accuracy", round(acc, 4))
    tracker.log_metric("val_loss", val_loss)
    tracker.end_run()

print("Experiment: " + tracker.experiment)
print()
print("{:<10} {:<8} {:<8} {:<10} {:<10}".format(
    "Run", "LR", "Depth", "Val Acc", "Val Loss"))
print("-" * 50)
for r in tracker.runs:
    print("{:<10} {:<8} {:<8} {:<10} {:<10}".format(
        r["run_name"],
        str(r["params"]["learning_rate"]),
        str(r["params"]["max_depth"]),
        str(r["metrics"]["val_accuracy"]),
        str(r["metrics"]["val_loss"]),
    ))

best = tracker.best_run("val_accuracy")
print()
print("Best run: " + best["run_name"])
print("  val_accuracy: " + str(best["metrics"]["val_accuracy"]))
print("  params: " + json.dumps(best["params"]))

Recall check

0 of …

Say the answer out loud before you reveal it.

What are the four core components of MLflow?
What is the difference between a parameter and a metric in MLflow?
What does the MLflow Model Registry do?
What does mlflow autolog capture automatically?

Cheat sheet

MLflow Experiment Tracking

MLflow tracks parameters (immutable config), metrics (time-series numbers), and artifacts (files) per run. Model Registry manages Staging → Production lifecycle. Every production model is traceable to exact training run + dataset + code commit.

MLOPS · AI.Cruz.in/mlops/mlflow_experiment_tracking.html