Skip to content

Data Drift

Model performance can be poor if models trained on a specific dataset encounter different data in production. This is called data drift. In the Monitor tab for your model, Fiddler gives you a visual way to explore data drift and identify what data is drifting, when it’s drifting, and how it’s drifting. This is a first step in identifying possible model performance issues.

Monitor_Drift_Metrics You can change the time range using the button in the upper-right: Monitor_Drift_Bins

What is being tracked?

  • Prediction Drift – The Jensen–Shannon divergence of the data distribution for the time period of interest against a baseline distribution. The baseline should ideally be the training set (or a sample from it), as that is the data the model has seen, and we want to benchmark against that.
  • Prediction Average – The mean of the model’s predictions over time. This can be thought of as an intuitive drift score.
  • Drift Analytics – You can drill down into the features responsible for the prediction drift using the table at the bottom.
    • Feature Impact: The contribution of a feature to the model’s predictions, averaged over the baseline dataset. The contribution is calculated using random ablation feature impact.
    • Feature Drift: Drift of the feature, calculated using Jensen-Shannon divergence.
    • Prediction Drift Impact: A heuristic calculated using the product of the feature impact and the feature drift. The higher the score, the more this feature is likely to have contributed to the prediction drift.

In the Drift Analytics table, you can a feature to see the feature distribution for both the time period under consideration and the baseline dataset. If it’s a numerical feature, you will also see a time series of the average feature value over time.

Monitor_Drift_Feature

Why is it being tracked?

  • Data drift is a great proxy metric for performance decline, especially if there is delay in getting labels for production events. (e.g. In a credit lending use case, an actual default may happen after months or years.)
  • Monitoring data drift also helps you stay informed about distributional shifts in the data for features of interest, which could have business implications even if there is no decline in model performance.

What do I do next with this information?

  • High drift can occur as a result of data integrity issues (bugs in the data pipeline), or as a result of an actual change in the distribution of data due to external factors (e.g. a dip in income due to COVID). The former is more in our control to solve directly. The latter may not be solvable directly, but can serve as an indicator that further investigation (and possible retraining) may be needed.
  • You can drill down deeper into the data by examining it in the Analyze tab.

The image below shows how to open the Analyze view for a specific feature and time range identified in the Data Drift page.

Monitor_Drift_Analyze This will bring you to the Analyze tab, where you can then use SQL to slice and dice the data and apply custom visualization tools and operators to make sense of the model’s behavior. Monitor_Analyze

Reference


  1. Join our community Slack to ask any questions 

Back to top