AI.Cruz - Learn AI Through Interactive Visualizations

RESEARCH

Reproducible ML Experiments

A result that can't be reproduced isn't a result. Fixing random seeds, tracking hyperparameters, and reporting variance are the minimum bar for credible ML research.

1The Reproducibility Crisis

A 2019 NeurIPS reproducibility study found that over 50% of published ML results could not be reproduced by independent teams. The causes are not fraud — they are mundane: missing hyperparameters, unlisted preprocessing steps, cherry-picked random seeds, and hardware differences. For a result to be science, it must be reproducible.

The most common reproducibility failure is not malicious. Authors report the best-of-N run, forget to fix the seed, or use a slightly different preprocessing pipeline. When a reviewer runs the released code, nothing matches. Reproducibility is a discipline, not a checkbox.

Unlisted hyperparameters
Learning rate, weight decay, dropout, batch size are often listed in a single table — but optimiser schedule, warmup, and clipping may be buried in appendix or code only.

Missing preprocessing
Normalisation constants, tokenisation settings, data splits, and augmentation pipelines can cause ±5% accuracy differences if even one step differs.

Cherry-picked seeds
Running 20 seeds and reporting the best one is p-hacking. Report all runs. The gap between best-of-20 and median-of-20 can exceed 2-3% on common benchmarks.

2Seeds, Environments, and Configuration

Three things determine whether your experiment is reproducible: the random seed chain (NumPy, PyTorch, CUDA), the software environment (Python version, library versions), and the configuration (every hyperparameter as a config file, not hardcoded). Miss any one of these and reproducibility is luck.

✓ Good practices

✓ Set seeds: random, numpy, torch, torch.cuda
✓ Pin all deps in requirements.txt with exact versions
✓ Store all hyperparameters in a YAML/JSON config
✓ Docker or conda env for full reproducibility

✗ Anti-patterns

✗ Hardcoded hyperparameters scattered across scripts
✗ No environment file — "it worked on my machine"
✗ Setting seed once but not for all libraries
✗ Using torch.backends.cudnn.benchmark = True in training (non-deterministic)

3Experiment Tracking

If you cannot reproduce a run from your tracking system alone, you have not logged enough. Every experiment should record: all hyperparameters, all metrics over time, the model checkpoint, the git commit hash, and the command used to launch it. Tools like Weights & Biases, MLflow, and Neptune automate most of this.

The golden rule: log everything at the start of the run, not the end. You never know which runs will fail mid-way. If you only log on success, you lose all partial data from crashes and will repeatedly run identical failing experiments.

Weights & Biases
Industry-standard ML experiment tracker. Automatic GPU/CPU metrics, hyperparameter sweeps, model comparison dashboards. Free for individual use.

MLflow
Open-source, self-hosted. Tracks runs, parameters, metrics, and artifacts. Integrates with most frameworks. Good if you need on-premise storage.

DVC
Data Version Control — versions datasets and model files with Git-like semantics. Complements W&B/MLflow by versioning the data alongside the code.

4Statistical Reporting: Error Bars Matter

A single number — "our method achieves 87.3% accuracy" — is meaningless without variance. Run every experiment with at least 5 different random seeds. Report mean ± standard deviation. Use significance tests (Wilcoxon signed-rank or paired t-test) when comparing methods. A difference is only a claim when it is statistically reliable.

Minimum: n = 5 seeds
5 runs is the informal floor for ML papers. For high-variance tasks (RL, few-shot) 10-20 runs are needed to get reliable estimates.

Report mean ± std
Mean shows the expected performance. Standard deviation shows consistency. If std > 1%, a difference of 0.5% between methods is not significant.

Ablation tables
Remove one component at a time and measure the effect. Ablations are often more credible than main results — they show which parts of the method actually matter.

Run experiments over multiple seeds and report statistics

import random
import math
import statistics

def set_seeds(seed: int):
    """Set all relevant seeds for reproducibility."""
    random.seed(seed)
    # In a real project you would also set:
    # numpy.random.seed(seed)
    # torch.manual_seed(seed)
    # torch.cuda.manual_seed_all(seed)

def train_model(config: dict, seed: int) -> float:
    """Simulate a training run. Returns final accuracy."""
    set_seeds(seed)
    # Simulate stochastic training with noise
    base_acc = config["base_accuracy"]
    noise = random.gauss(0, config["noise_std"])
    lr_effect = -abs(config["lr"] - 0.001) * 50  # peak at lr=0.001
    return min(1.0, max(0.0, base_acc + noise + lr_effect))

def run_experiment(name: str, config: dict, seeds: list) -> dict:
    """Run experiment over multiple seeds and collect stats."""
    results = []
    for seed in seeds:
        acc = train_model(config, seed)
        results.append(acc)
        print(f"  seed={seed:3d}  acc={acc:.4f}")
    mean = statistics.mean(results)
    std = statistics.stdev(results)
    return {"name": name, "mean": mean, "std": std, "n": len(seeds), "results": results}

seeds = [42, 123, 456, 789, 1337]

method_a = {"base_accuracy": 0.87, "noise_std": 0.015, "lr": 0.001}
method_b = {"base_accuracy": 0.88, "noise_std": 0.025, "lr": 0.001}

print("=== Method A ===")
exp_a = run_experiment("Method A", method_a, seeds)

print()
print("=== Method B ===")
exp_b = run_experiment("Method B", method_b, seeds)

print()
print("=== Results Summary ===")
for exp in [exp_a, exp_b]:
    ci_95 = 1.96 * exp["std"] / math.sqrt(exp["n"])
    print(f"{exp['name']:12s}  {exp['mean']:.4f} +/- {exp['std']:.4f}  "
          + f"95% CI: [{exp['mean']-ci_95:.4f}, {exp['mean']+ci_95:.4f}]  n={exp['n']}")

# Simple significance check: do CIs overlap?
ci_a = 1.96 * exp_a["std"] / math.sqrt(exp_a["n"])
ci_b = 1.96 * exp_b["std"] / math.sqrt(exp_b["n"])
overlap = (exp_a["mean"] + ci_a) >= (exp_b["mean"] - ci_b)
print()
print("Confidence intervals overlap:", overlap)
print("Conclusion:", "Difference is NOT statistically reliable at n=5." if overlap else "Methods are distinguishable at 95% confidence.")
print()
print("Lesson: Method B has a higher mean but also higher variance.")
print("Always report std — a small mean gain with high variance may not replicate.")