Product Concepts
What is ML Observability?
ML observability is the modern practice of gaining comprehensive insights into your AI application's performance throughout its lifecycle. It goes beyond simple indicators of good and bad performance by empowering all model stakeholders to understand why a model behaves in a certain manner and how to enhance its performance. ML Observability starts with monitoring and alerting on performance issues, but goes much deeper into guiding model owners towards the underlying root cause of model performance issues.
What is LLM Observability?
LLM observability is the practice of evaluating, monitoring, analyzing, and improving Generative AI or LLM based application across their lifecycle. Fiddler provides real-time monitoring on safety metrics like toxicity, bias, and PII and correctness metrics like hallucinations, faithfulness and relevancy.
Projects
A project within Fiddler represents your organization's distinct AI applications or use cases. It houses all of the model schemas that have been onboarded to Fiddler for the purpose of AI observability. Projects are typically specific to a given ML application or use case. They serve as a jumping-off point for Fiddler's model monitoring and explainability features.
Additionally, Fiddler projects serve as the main organizational structure within the platform. They are also the entity to which authorization is controlled within the platform. Different users and teams are given access to view information within Fiddler at the project level.
Models
Model Schemas
In Fiddler, A model schema is the metadata about a model that is being observed. Model schemas are onboarded to Fiddler so that Fiddler understand the data in which it is observing. Fiddler does not require the model artifact itself to properly observe the performance of the model (however, model artifacts can be uploaded to Fiddler to unlock advanced explainability features). . Instead, we may just need adequate information about the model's schema, or the model's specification in order to monitor the model.
📘 Working with Model Artifacts
You can upload your model artifacts to Fiddler to unlock high-fidelity explainability for your model. However, it is not required. If you do not wish to upload your artifact but want to explore explainability with Fiddler, we can build a surrogate model on the backend to be used in place of your artifact.
Model Versions
Fiddler offers model versions to organize related models, streamlining processes such as model retraining or conducting champion vs. challenger analyses. When retraining a model, rather than uploading a new model instance, a new version of the existing model can be created, retaining its core structure while accommodating necessary updates. These updates can include modifications to the schema, such as adding or removing columns, modifying data types, adjusting value ranges, updating the model specifications, and even refining task parameters or Explainable AI (XAI) settings.
Users/Teams
Data
Environments
Within Fiddler, each model is associated with two environments; pre-production and production. These environments assign purpose to the data published to Fiddler, allowing it to distinguish between:
Non-time series data (pre-production datasets, eg. training data)
Time-series data (production data, eg. inference logs)
Pre-Production Environment
Pre-production environment contains non-time series chunks of data, called datasets. Datasets are used primarily for point-in-time analysis or as static baselines for comparison against production data.
Production Environment
Production environment contains time series data such as production or inference logs which are the "digital exhaust" coming off of each decision a model makes. This time series data provides the inputs and outputs of each model inference/decision and is what Fiddler analyses and compares against the pre-production data to determine if the model's performance is degrading.
Datasets
Datasets within Fiddler are static sets of data that have been uploaded to Fiddler for the purpose of establishing "baselines". A common dataset that is uploaded to Fiddler is the model's training data.
Baselines
Baselines are derived from datasets and used by Fiddler to understand what the expected data distributions that the model will encounter are. A baseline in Fiddler is a set of reference data used to compare the performance of a model for monitoring purposes. The default baseline for all monitoring metrics in Fiddler is typically the model's training data. Additional baselines can be added to existing models as well that are derived from other datasets, historical inferences, or rolling baselines that refer back to data distributions using historical inferences.
Segments
Segments are custom filters applied to your data, enabling you to analyze metrics for specific subsets of your data population (e.g., "People under 50"). Segments help you focus on relevant data for more precise insights. Additionally, you can set alerts on these Segments to stay informed about important changes or trends within your specified subsets.
Metrics
Metrics are computations performed on data received by the platform. Fiddler supports the ability to specify custom user-defined metrics (Custom Metrics) in addition to five out-of-the-box metric types:
Data Integrity
Data Drift
Performance
Statistics
Traffic
Alerts
Alerts are user-specified rules which trigger when some condition is met by data received on the Fiddler platform. For any alert rule, you can customize how you'd like to be notified (either via email, Slack, or using webhooks).
Enrichments
Enrichments augment existing columns with additional metrics to monitor different aspects of LLM applications. The new metrics are available for use within the analyze, charting, and alerting functionalities in Fiddler.
Users/Teams
Charts
Fiddler Monitoring provides three distinct types of visualizations: monitoring charts, embedding visualizations, and performance analytics, with the capability to generate reports using customizable dashboards. These comprehensive dashboards consolidate various visualizations, including charts for data drift, data integrity, and embedding visualizations, offering a detailed overview of model performance.
Monitoring Charts
Monitoring charts provide a comprehensive view of model performance or analyze model to model performance. With intuitive displays for data drift, performance metrics, data integrity, traffic patterns, and more, monitoring charts empower users to maintain model accuracy and reliability with ease.
Embedding Visualizations
Embedding visualization is a powerful charting tool used to understand and interpret complex relationships in high-dimensional data. Reducing the dimensionality of custom features into a 2D or 3D space makes it easier to identify patterns, clusters, and outliers.
Jobs
Upon pubishing data to Fiddler or creating a new model on Fiddler, a Fiddler job is queued to process the request. Upon successful job completion, the new data or model is available for use in the monitoring, alerting, and charting functionalities.
Bookmarks
Bookmarking enables quick access to projects, models, charts, and dashboards. The comprehensive bookmark page enhances your convenience and efficiency in navigating the Fiddler platform.
Check the UI Guide to Visualize Project Architecture on our User Interface
↪ Questions? Join our community Slack to talk to a product expert
Last updated