Customer Churn Prediction
Churn prediction is a common task for machine learning models. Companies use churn models to determine the likelihood of customers leaving." Identifying if and when customers may churn is extremely valuable. Having a robust and accurate churn prediction model helps businesses take action to prevent customers from leaving the company. However, like any ML model, if left unattended, these models can degrade over time, leading to customer loss.
The Fiddler AI Observability platform provides various tools for monitoring, explaining, analyzing, and improving the performance of your ML model.
This article will illustrate how Fiddler allows us to detect and diagnose problems using an example churn model.
👍 Refer to this quickstart guide to learn how to
Onboard this model to the Fiddler platform
Publish events on the Fiddler platform
Example - Model Performance Degradation due to Data Integrity Issues
This example will illustrate how Fiddler's model monitoring features will help us identify the root cause of model performance issues.
Step 1 - Monitor Drift
When we check out our model's default dashboard, we notice a drop in the predicted churn value and a rise in the predicted churn drift value. Our next step is to check if this has resulted in a decline in performance.
Step 2 - Monitor Performance Metrics
To further our analysis, we can add performance metrics to this chart—like precision, recall, and F1—to determine if the drift impacts model performance. We're choosing these metrics because they are suited for classification problems and help us identify the number of false positives and false negatives.
Although we notice that the precision has remained constant, there is a drop in the F1-score and recall, which means that a few customers are likely to churn, but the model cannot predict their outcome correctly.
📘 There could be a number of reasons for drop in performance:
Cases of extreme events (Outliers)
Data distribution changes / Drift
Model/Concept drift
Pipeline health issues
Step 3 - Check the cause of the drift
Our next step is to return to the Data Drift chart for root cause analysis. The drift is calculated using Jensen-Shannon Divergence, which compares the distributions of the two data sets being compared.
We can select the June 21st bin where we see an increase in output drift. Upon choosing that plot point, the Root Cause Analysis tabs below the chart populate using the selected time bin range data. In the Data Drift table below, we can see that the numofproducts feature is also drifting, and since it is an impactful feature, it contributes the most to the resulting output drift we already identified.
To see the feature distribution for the numofproducts feature, click the down arrow in the table to the left of its name. The baseline and production data distributions differ significantly, leading to data drift.
The next step is to determine if the change in distribution occurs for all inferences or a particular model segment/cohort. It could also be due to other factors like time (seasonality, etc.), fault in data reporting (sensor data), change in the unit in which the metric is reported, etc.
Seasonality could be observed by plotting the data across time (provided we have enough data); a fault in data reporting would mean missing values, and a change in the unit of data would mean a change in values for all subsections of data.
We will use the root cause analysis features to investigate whether the change affected only a data segment.
Step 4 - Root Cause Analysis
Within the Root Cause Analysis view, the Analyze tab offers several performance and feature analytics chart types that can be used to examine your data across various segments. Examples of performance analytics charts are displayed here. The first set of charts shows the June 20th time bin data before the prediction drift behavior. The second set of charts shows the data for the June 21st time bin, the first time we noticed the steep increase in prediction drift.
By accessing Root Cause Analysis, we can investigate our data based on the date selection from our drift chart. The platform provides various chart types and filtering options to help us understand the data patterns.
We check the distribution of the field numofproducts for our selection. We can do this by selecting Feature Distribution from the available Analytics chart types. To check if the change in the range violation has occurred for a data subsection, we can plot it against categorical variables. In our case, we can check the distribution of numofproducts against age and geography. We can create a feature correlation visualization by filtering our data and selecting the appropriate chart type.
On plotting the feature correlation of gender vs numofproducts, we observe the distribution to be similar.
For the sake of this example, let's say that the state of Hawaii (a value in the geography column) announced that it has eased restrictions on the number of loans allowed. Since loans are one of the products, we hypothesize that numofproducts would be higher for the state. To test this, we will check the feature correlation between geography and numofproducts.
We do see higher values for the state of Hawaii than for other states. We can further check the distribution of the column numofproducts just for the state of Hawaii by applying a filter for geography = 'Hawaii'. Let's create a new chart for the Dashboard that displays overall model prediction drift along with our top retail states, including Hawaii.
When checking for the subset of Hawaii, we see a huge prediction drift increase. Otherwise, we see normal drift for the other regions and only a moderate drift increase for all regions.
Step 5 - Measuring the impact of the 'numofproducts' feature
We notice the "Hawaii" cohort exhibits poor decisions of the model in the form of false negatives. Compare the confusion matrices and numofproducts feature distributions for the Hawaii and Non-Hawaii Customer segments.
We see that the feature numofproducts contributes significantly towards the data point being predicted not to churn.
We have seen that the churn model's performance drops due to data drift in a particular cohort for one particular feature. We can improve the performance by retraining the model with new data. Still, before that, we must perform mitigation actions, which would help us preemptively detect the model performance degradation and inform our retraining frequency.
Step 6 - Mitigation Actions
Define data segments: Create cohort-specific alerts and charts to identify higher-risk or more volatile subsets of data.
Add alerts: We can alert users to ensure we are notified the next time there is a performance degradation. In this example, there was a performance degradation due to feature drift within a segment. To mitigate this, we can define a segment and set up an alert that would notify us in the case of input drift on our problematic feature.
Create charts that monitor performance across key data segments.
Resources
Learn more about Fiddler's Dashboards and Charts
See how Segments can enhance alerts and improve analytics capabilities
Fiddler Alerts are the underpinning of ML and GenAI application monitoring
❓ Questions? Talk to a product expert or request a demo.
💡 Need help? Contact us at [email protected].
Last updated
Was this helpful?