Data Integrity

ML models are driven by feature pipelines and automated workflows that involve dynamic and increasingly complex data. Data is transformed from source to model input, which can result in data inconsistencies and errors.

Using Fiddler, you can automatically monitor the model inputs for three types of data integrity issues for your model: missing feature values, type mismatches (e.g. sending a float input for a categorical feature type) or range mismatches (e.g. sending an unknown US State for a State categorical feature).

You can track these violations in the Data Integrity tab.

What is being tracked?

The time series above tracks the violations of data integrity constraints set up for this model.

Missing value violations — You can specify either a percentage (relative to total traffic) or a count of missing value violations for an individual feature or for all features in a given period.
Type violations — You can specify either a percentage (relative to total traffic) or a count of data type mismatch violations for an individual feature or for all features in a given period.
Range violations — You can specify either a percentage (relative to total traffic) or a count of range mismatch violations for an individual feature or for all features in a given period.
All violating events — You can specify either a percentage (relative to total traffic) or a count of all (Missing values, Type and Range ) violations for an individual feature or for all features in a given period.

Why is it being tracked?

Issues related to data integrity can result in inaccurate data being fed into the model. This, in turn, can adversely affect model performance and have a detrimental impact on both business outcomes and the end-user experience.

How does it work?

It can be tedious to set up constraints for individual features when they number in the tens or hundreds. To avoid this, you can provide Fiddler with a baseline dataset that's representative of the data you expect your model to infer in production. This should be sampled from your model's training set, and can be uploaded to Fiddler using the Python API client.

Fiddler will automatically generate constraints based on the distribution of data in this dataset.

Missing values: If a feature has no missing values, then the data integrity violation will be set up to trigger when any missing values are seen. Similarly, if the feature has 50% of its values missing, then the data integrity violation will be set up to trigger when more than 50% of the values encountered are missing in a specified time range.
Type mismatch: A data integrity violation will be triggered when the type of a feature value differs from what was specified for that feature in the baseline dataset.
Range mismatch: For categorical features, a data integrity violation will be triggered when it sees any value other than the ones specified in the baseline. Similarly, for continuous variables, the violation will be triggered if the values are outside the range specified in the baseline.

What steps should I take with this information?

The visualization above informs us of the feature-wise breakdown of the violations. The raw counts of the violations are shown in parentheses.
If there is a spike in violations, or an unexpected violation occurs (such as missing values for a feature that doesn’t accept a missing value), then a deeper examination of the feature pipeline may be required.
You can also drill down deeper into the data by examining it in the Analyze tab. We can use SQL to slice and dice the data and try to find the root cause of the issues.

↪ Questions? Join our community Slack to talk to a product expert