Data Drift and Concept Drift

Tunebox launches a cheap student plan. Within weeks, thousands of new subscribers arrive who skip songs far more than anyone in the training data. Months later, a rival service launches, and even loyal heavy listeners start cancelling. Both events hurt the churn model — but for different reasons, and you detect them in different ways.

The rule

Data drift: the inputs change. Concept drift: what they mean changes.

Data drift is a change in the distribution of the inputs. The student plan shifts skips_per_week upward. The model has seen few users like these, so its answers for them are less reliable. In symbols, the input distribution $P(X)$ has moved.

Concept drift is a change in the relationship between inputs and the outcome. Heavy listening used to mean loyalty; after the rival launch, the same listening pattern comes with cancelling. The inputs can look perfectly normal while the right answer for them has changed: $P(y \mid X)$ has moved.

They are detected differently:

• Data drift needs no labels. Compare each feature's live distribution with training. A common score is the population stability index (PSI): split the feature into bins, and for each bin compare the training share $e$ with the live share $a$, then add up $(a - e) \ln\frac{a}{e}$ over all bins. A widely used rule of thumb: below 0.1 is stable, 0.1 to 0.25 is a moderate shift worth watching, above 0.25 is a large shift.

• Concept drift usually needs labels. It shows up as falling recall or precision once true outcomes arrive — for churn, a month later.

The response is similar for both: retrain on recent data, and for concept drift also look for new features that capture the change, such as whether a user was sent a rival's promotion.

The shortcut

Ask "Do the inputs look different, or do the same inputs now lead to different outcomes?" Different inputs → data drift. Same inputs, different outcomes → concept drift.

Every bin's term is zero or positive, so PSI only grows as shares move apart.

Drills

Answer fast — trust the rule.

Remember

• Data drift: the inputs change. Concept drift: what they mean changes.
• Different inputs means data drift; same inputs with different outcomes means concept drift.
• Trap: waiting for labels to notice drift; input checks like PSI warn you weeks earlier.