Lesson 2 of 7 · 10 min
Data pipelines, versioning and feature stores
Make training data reproducible, keep serving consistent with training, and validate data before it corrupts a model.
Training-serving skew
Skew is any difference between how a feature is computed during training and during serving. A pandas transformation in a notebook and a hand-written reimplementation in the serving service will drift apart, and the model silently receives inputs unlike anything it was trained on.
The structural fixes are to share one transformation code path between both, or to use a feature store that computes a feature once and serves it to both. Logging serving inputs and comparing their distribution to the training set catches what slips through.
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score
# The scaler is fitted INSIDE each fold, so no test statistics leak in.
pipe = Pipeline([
("scale", StandardScaler()),
("model", model),
])
scores = cross_val_score(pipe, X, y, cv=5)
# For time series, never use a random split - it trains on the future.
# Use TimeSeriesSplit, which always tests on data after the training window.Versioning data
'Which data produced this model?' has to have an exact answer. The usual approaches are content-addressed storage with a pointer in Git (DVC, lakeFS), immutable snapshots keyed by a run ID, or a table format with time travel such as Delta Lake or Iceberg.
Whichever you pick, the training run should record the dataset version, the code commit, the hyperparameters and the environment. That tuple is what makes a result reproducible and an incident investigable.
Feature stores
A feature store holds one definition of each feature and serves it two ways: an offline store for training, over history, and an online store for low-latency lookup at prediction time.
The capability that justifies it is point-in-time correctness. When building a training set, the store returns each feature's value as of the moment of that label, not its current value. Doing this by hand with joins is where teams most often introduce leakage without noticing.
What to validate on every run
- Schema — expected columns, types, and no unexpected additions.
- Nulls and ranges — null rate within tolerance, values inside plausible bounds.
- Cardinality — categorical levels not exploding or collapsing.
- Distribution — key features compared against a reference window.
- Volume — row counts within expectation, so a partial load fails loudly.
- Freshness — the newest record is recent enough to be meaningful.
Worth remembering
- Training-serving skew is the most common cause of a good model performing badly live.
- Leakage inflates offline metrics and is invisible until production.
- Validate schema and distribution at the pipeline boundary, not after training.
Test it now
Data quality and leakage scenarios are heavily represented in the medium bank.