Skip to content

Model Monitoring

Overview

Fiddler Monitoring helps you identify issues with the performance of your ML models after deployment. Fiddler Monitoring has six main features:

  1. Performance
  2. Data drift
  3. Data Integrity
  4. Outliers Detection
  5. Service Metrics
  6. Alerts

Integrate with Fiddler Monitoring

Integrating Fiddler monitoring is a four step process:

  1. Upload dataset

    Fiddler needs your training dataset. A dataset can be uploaded to Fiddler using our UI and Python package. For more information, see:

  2. Register model

    Fiddler needs your model binary in order to run explainability algorithms that help you troubleshoot production issues. Fiddler supports a wide variety of model formats. A model can also be uploaded to Fiddler using our Python package. For more information, see:

  3. Configure monitoring for this model

    You will need to configure bins and alerts for your model. These will be discussed in details below.

  4. Send traffic from your live deployed model to Fiddler

    Use the Fiddler SDK to send us traffic from your live deployed model.

Publish events to Fiddler

In order to send traffic to Fiddler, use the publish event API from the Fiddler SDK. Here is a sample of the API call:

sample code: publish_event sending prediction
    import fiddler as fdl
    fiddler_api = fdl.FiddlerApi(url=url, org_id=org_id, auth_token=token)
# Publish an event
    fiddler_api.publish_event(project_id='bank_churn',
                          model_id='bank_churn',
                          event = {
                           "CreditScore": 650,      # data type: int
                           "Geography": "France",   # data type: category
                           "Gender": "Female",
                           "Age": 45,
                           "Tenure": 2,
                           "Balance": 10000.0,      # data type: float
                           "NumOfProducts": 1,
                           "HasCrCard": "Yes",
                           "isActiveMember": "Yes",
                           "EstimatedSalary": 120000,
                           "probability_churned": 0.105
                           },
                          event_id=some_unique_id, #optional
                          update_event=False, #optional
                          event_timestamp=1511253040519 #optional
   )

The publish event API can either be called in real-time right after your model inference, or as part of a batch call after the fact. In the latter case, you will need to send Fiddler the original event timestamp so that we can accurately replay it in the time series charts.

Following is a description of all the parameters for publish_events:

  • project_id: Project ID for the project this event belongs to.
  • model_id: Model ID for the model this event belongs to.
  • event: The actual event as an array. The event can contain:

    • inputs
    • outputs
    • target
    • decisions
      • Fiddler currently allows only category values for decisions.
    • metadata
  • event_id: A customer generated unique event ID that Fiddler would use to join inputs/outputs to targets/decisions/metadata sent later as an update. update_event: Flag indicating if the event is a new event (insertion) or an update to an existing event. In case its an update_event, it is required that customer sends event_id.

  • event_timestamp: Timestamp at which the event (or update) occurred. The supported format is the UTC timestamp in milliseconds. In case customer is sending an update event, they should use whatever time the update happened, i.e., the time the target/decision were generated and not when the model predictions were made.

Updating events

Fiddler supports partial updates of events for decisions, metadata, and target. This can be useful when you don’t have access to all of this data at prediction time. Inputs and outputs can only be sent at insertion time (with update_event = false).

Set update_event = true to indicate that you are updating an existing event. You only need to provide the decisions, metadata, or target fields that you want to change; any fields you leave out will remain as they were before the update.

Example

Here’s an example of using the publish event API to update an existing event:

    import fiddler as fdl

    fiddler_api = fdl.FiddlerApi(url=url, org_id=org_id, auth_token=token)

    fiddler_api.publish_event(project_id='bank_churn',
                              model_id='bank_churn',
                              event = {
                                "LoanStatus": Approved,   # data type: category
                                "ModelDecision": "Approved",# data type: category
                                },
                              event_id=some_unique_id, #optional
                              update_event=True, #optional
                              event_timestamp=1511253040519 #optional
   )
The above publish_event call will tell Fiddler to add a target (LoanStatus=Approved) and a decision (ModelDecision=Approved) to an existing event (event_id=’some_unique_id’).

Once you’ve used the SDK to send Fiddler your live event data, that data should show up under the Monitor tab in the Fiddler UI:

Monitor_Dashboard

Reference


  1. Join our community Slack to ask any questions 

Back to top