Experiment tracking and reproducibility

What a run must record

Parameters, metrics and artifacts are the obvious three. The ones people omit are the code commit, the dataset version, the random seeds and the resolved environment — and those are exactly what you need when a result cannot be reproduced.

Without them, comparing two experiments is guesswork, because you cannot tell whether the difference came from the change you made or from data that shifted between the runs.

import mlflow

mlflow.set_experiment("churn")
with mlflow.start_run(run_name="lgbm-depth8"):
    mlflow.log_params({"max_depth": 8, "lr": 0.05, "seed": 42})
    mlflow.set_tags({
        "git_sha": git_sha,          # exact code
        "data_version": data_version,  # exact data
    })

    model = train(X_train, y_train)
    mlflow.log_metrics({"auc": auc, "pr_auc": pr_auc})
    mlflow.log_artifact("confusion_matrix.png")
    mlflow.sklearn.log_model(model, "model", signature=signature)

Note: The signature records expected input and output schema, so a mismatched request fails at the boundary instead of producing nonsense.

Reproducibility in practice

Seeds are necessary but not sufficient. Full reproducibility needs pinned dependency versions with a lock file, a container image referenced by digest rather than tag, and awareness that GPU non-determinism means bit-identical results may be unattainable.

Aim for statistical reproducibility — the same conclusion within a stated tolerance — rather than bit-identical outputs, and state the tolerance explicitly.

Registry stages