Model performance can be poor if models trained on a specific dataset encounter different data in production. This is called data drift. In the Monitor tab for your model, Fiddler gives you a visual way to explore data drift and identify what data is drifting, when it’s drifting, and how it’s drifting. This is a first step in identifying possible model performance issues.
You can change the time range using the button in the upper-right:
What is being tracked?¶
- Prediction Drift – The Jensen–Shannon divergence of the data distribution against a baseline distribution, for the time period of interest. The baseline should ideally be the training set (or a sample from it), as that is the data the model has seen, and we want to benchmark against that
- Prediction Average – The mean of the model’s predictions, mapped over time. This can be thought of as an intuitive drift score.
- Drift Analytics – You can drill down into the features responsible for the prediction drift using the table at the bottom.
- Feature Impact: The contribution of a feature to the model’s predictions, averaged over the baseline dataset. The contribution is calculated using random ablation feature impact.
- Feature Drift: Drift of the feature, calculated using Jensen-Shannon divergence.
- Drift Impact: The product of the feature impact and the feature drift. This is a heuristic: the higher the score, the more this feature likely contributed to the prediction drift. In the Drift Analytics table, you can expand features to see the feature distribution for the time period under consideration and the baseline data are also plotted. If it’s a numerical feature, you will also see a time series of the average feature value over time.
Why is it being tracked?¶
- Data drift is a great proxy metric for performance decline, especially if there is delay in getting labels for production events. (For example, in a credit lending use case, an actual default may happen after months or years.)
- Monitoring data drift also helps you stay informed about distributional shifts in the data for features of interest, which could have business implications even if there is no decline in model performance.
What do I do next with this information?¶
- High drift could be because of data integrity issues (bug in data pipeline), or because of an actual change in the distribution of data due to external factors (dip in income due to COVID). The former is more in our control to solve directly. The latter may not be solvable directly, but something that could be possibly worked around.
- You can drill down deeper into the data by examining it in the Analyze tab.
The images below shows how to open the Analyze view for a specific feature distribution identified in the Data Drift page:
You will see the production query for this data in the Analyze view. You can then use SQL to slice and dice the data, and apply custom visualization tools and operators to make sense of the model’s behavior.
- See our article on The Rise of MLOps Monitoring