Skip to content

Import ML model container into Fiddler

View In Github

Packaging ML models and their dependencies as a container where the /predict endpoint is exposed via RESTful APIs is not uncommon. Such container based models can be imported as-is into Fiddler and then can be used for monitoring or explanations.

RESTful API for an ML Model

The model is trained and saved, so what's next?

For the purpose of this exercise, we can use the Random Forest based sklearn model created in the bank_churn sample.

!wget -P bank-churn
!wget -P bank-churn

Expose Model Predictions as an API

One of the ways the model can be used by various entities is via APIs. Of the various different types of APIs, RESTful APIs are quite flexible and user friendly.

As an example, we can use the popular Flask web application framework to expose endpoints such as /predict which can be used to access the model.

Create RESTful APIs For the bank_churn Model Using Flask

%%writefile bank-churn/

# A python script to expose the bank_churn model.pkl via REST API.

import os
import pickle

import flask
from churn_random_forest import RFPredictor

PACKAGE_DIR = os.path.dirname(__file__)

# load the bank_churn model.pkl.
model = RFPredictor(PACKAGE_DIR, output_column=['probability_churned'])

def init_server():
    app = flask.Flask(__name__)

    @app.route('/health', methods=['GET'])
    def health():
      # return the health of the model.
        return 'OK'

    @app.route('/predict', methods=['POST'])
    def predict():
      # load the input data from the flask request which is
      # a python pickle.
        df = pickle.loads(
            # run a prediction on the model object using the
            # input dataframe.
            pred_obj = model.predict(df)
        except Exception as e:
            raise RuntimeError(
                f'Model and input loaded, but prediction failed. '
                f'Is your input correct? {e}'
        # return the pickled prediction as a response.
        return flask.Response(
            pickle.dumps(pred_obj), mimetype='application/octet-stream'

    # return the Flask app.
    return app

if __name__ == '__main__':
    # purposely running Flask in debug mode. Do not do this in production.
    init_server().run(host='', port=5101, threaded=False, debug=True)

The code above sets up RESTful API around the bank_churn model and the API is served by Flask app on port 5101. By the way, this example uses Flask in debug mode and is not meant for production workflows.

Package the Model Artifacts, the API Server and the Dependencies

There's a few things that are needed to make this model and API work reliably in different environments like laptops, Linux, Mac, kubernetes etc. To achieve that portability we need to package the model artifacts and along with their dependencies (python modules and packages etc). So, the next step is to package this model into a container thus making it a microservice.

Let's use a dockerfile to describe the recipe for the container.

%%writefile bank-churn/Dockerfile

# Dockerfile: package the model artifacts into a container

# Use a python-3 based base image.
FROM python:3.7

# Set /app as the working directory

# Install the dependencies
RUN pip install --upgrade pip \
    && pip install scikit-learn==0.21.2 flask==1.1.1 pandas==0.25.1 joblib==0.14.0

# Copy the relevant model artifacts into WORKDIR
COPY /app/
COPY model.pkl /app/model.pkl

# Copy the API server script to WORKDIR
COPY /app/

# Expose the port 5101 on the container

# Run
CMD [ "python", "./" ]

Given this dockerfile, we can now build a container image that packages up the model artifacts, dependencies and flask server script.

The container image is built using:

%cd bank-churn/
!docker build -t examples:bank-churn-1.0 .

Once the build is done, the image details can be seen by running:

!docker images

Thus built container can be run to make sure the container process starts up properly and the APIs are accessible.

!docker run -d --name bank-churn -p 5101:5101 examples:bank-churn-1.0

If all goes well, the bank_churn container should now be started up and ready to serve requests on port 5101 and you should be able to inspect the logs on the container using:

!docker logs bank-churn
This container image can now be pushed onto a container image registry. Once that's done, the image can be used by others either as a container or as a pod on kubernetes.

!docker tag examples:bank-churn-1.0
!docker login -u fiddler -p <password>
!docker push

Upload and Serve bank_churn model

The model is packaged, tagged and pushed onto a registry like dockerhub, ECR, quay etc and is ready to be used.

We will go through the steps needed to ingest that model into Fiddler and use it for predictions, explanations and monitoring.

To upload a model, you first need to upload a sample of the data of the model’s inputs, targets, and additional metadata that might be useful for model analysis. This data sample helps us (among other things) to infer the model schema and the data types and values range of each feature.

Initialize Fiddler Client

!pip3 install fiddler-client

We begin this section as usual by establishing a connection to our Fiddler instance. We can establish this connection either by specifying our credentials directly, or by utilizing our fiddler.ini file. More information can be found in the setup section.

import fiddler as fdl

# client = fdl.FiddlerApi(url=url, org_id=org_id, auth_token=auth_token)
client = fdl.FiddlerApi()
We will now create our model's directory. This will be where we save information relevant to the uploading portion.

import pathlib
import shutil

client = fdl.FiddlerApi()

PROJECT_ID = 'bank_churn'
DATASET_ID = 'bank_churn'
MODEL_ID = 'bank_churn'

# create a working dir to save the schema
MODEL_DIR = pathlib.Path(MODEL_ID)
shutil.rmtree(MODEL_DIR, ignore_errors=True)

Create Project

Here we will create a project, a convenient container for housing the models and datasets associated with a given ML use case. Uploading our dataset in the next step will depend on our created project's project_id.

# Creating our project using project_id
if PROJECT_ID not in client.list_projects():

Load Dataset

Next, we load our dataset and split into a train/test set. We also create some dataframe information that will be used by Fiddler in later steps.

import pandas as pd
from sklearn.model_selection import train_test_split

df = pd.read_csv('')
df = df.reset_index(drop=True)

train, test = train_test_split(df, test_size=0.3)
df_info = fdl.DatasetInfo.from_dataframe(df, max_inferred_cardinality=1000, display_name='bank-churn Dataset')

Upload Dataset

Next, we call upload_dataset to upload our training and test data, saving in a dataset held by the dataset_id parameter.

if DATASET_ID not in client.list_datasets(PROJECT_ID):
    upload_result = client.upload_dataset(
        dataset={'train': train, 'test': test},

Create and Save Model Schema

As you must have noted, in the dataset upload step we did not ask for the model’s features and targets, or any model specific information. That’s because we allow for linking multiple models to a given dataset schema. Hence we require an Infer model schema step which helps us know the features relevant to the model and the model task. Here you can specify the input features, the target column, decision columns and metadata columns, and also the type of model.

target = 'Churned'
feature_columns = list(df.drop(columns=['Churned']).columns)

model_info = fdl.ModelInfo.from_dataset_info(
    dataset_info=client.get_dataset_info(PROJECT_ID, DATASET_ID),
    display_name='bank_churn model',
    description='scikit-learn Random Forest model based on Kaggle Bank Customer Churn data'
Next, we save this schema as a model.yaml file. This schema will be uploaded in a later step.

import yaml

# save model schema
with open(MODEL_DIR / 'model.yaml', 'w') as yaml_file:
    yaml.dump({'model': model_info.to_dict()}, yaml_file) Wrapper

A wrapper is needed between Fiddler and the model. This wrapper can be used to translate the inputs and outputs to fit what the model expects and what Fiddler is able to consume. More information can be found here

python wrapper to transform the input and outputs of the model.
import logging
import os
import pickle
from typing import NamedTuple

import numpy as np
import pandas as pd
import requests

LOG = logging.getLogger(__name__)

class ModelId(NamedTuple):
    """Uniquely identifies a model."""

    org_name: str
    project_name: str
    model_name: str

class Predictor:
    Handles running predictions on an containerized model that exposes the
    `/predict` endpoint.

    # If an external endpoint is defined at `MODEL_ENDPOINT` use that.
    # Otherwise use the ingress.
    ENDPOINT = os.environ.get('MODEL_ENDPOINT', 'http://ingress-nginx.ingress-nginx')

    def __init__(self, model_id: ModelId, model_info):
        Initialize the mode predictor.

        model: the model_id, a NamedTuple which contains the org, project and
        model to uniquely identify the model.
        model_info: the model schema.
        self.model_id = model_id
        self.model_info = model_info

    def predict(self, input_df: pd.DataFrame) -> pd.DataFrame:
        Given the input_df, transform it to the input the model expects and
        call the `/predict` endpoint of the model.
            url = '/'.join([self.ENDPOINT, 'predict', *self.model_id])
            res =, data=pickle.dumps(input_df))
        except Exception as ex:
                error_info = res.json()
            except Exception:
                error_info = None
            raise RuntimeError(
                f'Model execution failed for {self.model_id}.\n'
                f'Exception:{ex}\nServer Exception:{error_info}'

            pred_obj = pickle.loads(res.content)
        except Exception as ex:
            raise RuntimeError(
                f'Model execution call was successful, but '
                f'failed to un-pickle the prediction. {ex}'

            if str(type(pred_obj)).endswith("pandas.core.frame.DataFrame'>"):
                pred_df = pred_obj
                pred_array = np.array(pred_obj)
                if pred_array.ndim == 1:
                    pred_array = pred_array[:, np.newaxis]
                pred_df = pd.DataFrame(pred_array)
            pred_df.columns = self.model_info.get_output_names()
        except Exception as ex:
            pred_obj_excerpt = repr(pred_obj)[:1_000]
            raise RuntimeError(
                f'Model execution call and result unpickling were successful, '
                f' but unable to massage results into a dataframe of model '
                f'predictions.\nException: {ex}\nReturned object type: '
                f'{type(pred_obj)}.\nObject: {pred_obj_excerpt}'

        return pred_df

def get_predictor(model_id, model_info):
    `get_predictor(model_id, model_info)` is the required hook that's called by
    the Fiddler's executor-service to get to the `Predictor`.
    return Predictor(model_id, model_info)

Ingest and Register the model container with Fiddler

Use the container image from the registry and specify the resources required as a part of the call into Fiddler.

upload_model_package when given the parameters like image, deployment_type etc, creates a set of kubernetes objects like a deployment, service, ingress etc in the background.

# `upload_model` expects model.yaml (the model schema created above)
# and (the interface between the model and Fiddler) in
# the MODEL_DIR.
# The image is used to create a pod on kubernetes using the port
# and the resources.
image = ""
    deployment_type='predictor',  # when the model just exposes a predict endpoint

Run Predictions on the Model

Once the model is ingested and deployed on Fiddler, we should be able to use it for predictions.

predictions = client.run_model(
explanations = client.run_explanation(

That's All Folks!

Back to top