AI Observability: Turning Terabytes into Actionable Insights
Maintaining and developing modern e-commerce platforms, which process millions of transactions every minute, presents a significant challenge: managing the vast amounts of telemetry data generated. This data includes metrics, logs, and traces across numerous microservices. When critical incidents strike, on-call engineers often face the daunting task of sifting through an ocean of information, akin to finding a needle in a haystack, to uncover relevant signals and insights. This often turns observability—the ability to understand a system’s internal states from its external outputs—into a source of frustration rather than clarity.
To alleviate this major pain point, a solution has been explored utilizing the Model Context Protocol (MCP) to add context and draw inferences from logs and distributed traces. This approach underpins the development of an AI-powered observability platform, which aims to transform how organizations measure and understand system behavior, a foundational element for reliability, performance, and user trust. As the adage goes, “What you cannot measure, you cannot improve.”
Achieving true observability in today’s cloud-native, microservice-based architectures is more complex than ever. A single user request might traverse dozens of microservices, each continuously emitting logs, metrics, and traces. The sheer volume of this telemetry data is staggering: often tens of terabytes of logs, tens of millions of metric data points, millions of distributed traces, and thousands of correlation IDs generated every minute. Beyond the volume, the primary challenge lies in data fragmentation. According to New Relic’s 2023 Observability Forecast Report, half of all organizations report siloed telemetry data, with a mere 33% achieving a unified view across metrics, logs, and traces. Logs tell one part of the story, metrics another, and traces yet another. Without a consistent thread of context, engineers are forced into manual correlation, relying on intuition, tribal knowledge, and tedious detective work during incidents. This complexity begs the question: how can artificial intelligence help us move past fragmented data and offer comprehensive, actionable insights, particularly by making telemetry data intrinsically more meaningful and accessible for both humans and machines using a structured protocol like MCP?
This central question formed the foundation of the project. Anthropic defines MCP as an open standard designed to create a secure, two-way connection between diverse data sources and AI tools. This structured data pipeline encompasses three key elements: contextual ETL for AI, which standardizes context extraction from multiple sources; a structured query interface, enabling AI queries to access transparent and easily understandable data layers; and semantic data enrichment, which embeds meaningful context directly into telemetry signals. This integrated approach has the potential to shift platform observability from reactive problem-solving to proactive insights.
The system architecture for this MCP-based AI observability platform is layered. In the initial layer, contextual telemetry data is developed by embedding standardized metadata directly into telemetry signals, such as distributed traces, logs, and metrics. This enriched data then feeds into the second layer, the MCP server, which indexes, structures, and provides API-driven client access to this context-enriched information. Finally, the third layer, an AI-driven analysis engine, leverages this structured and enriched telemetry data for sophisticated anomaly detection, correlation, and root-cause analysis to troubleshoot application issues. This layered design ensures that both AI and engineering teams receive context-driven, actionable insights from the telemetry data.
The implementation of this three-layer system begins with context-enriched data generation. The core insight here is that data correlation needs to happen at the point of creation, not during analysis. By embedding a consistent set of contextual data—such as user ID, order ID, request ID, and service details—into every telemetry signal (logs, metrics, traces) as it’s generated, the system solves the correlation problem at its source. This ensures that every piece of data inherently carries the necessary context for later analysis.
The second layer involves building the MCP server, which transforms this raw, context-rich telemetry into a queryable API. Key operations at this stage include indexing for efficient lookups across contextual fields, filtering to select relevant subsets of data, and aggregation to compute statistical measures across time windows. This layer effectively transforms unstructured data into a structured, query-optimized interface that an AI system can efficiently navigate.
The final layer is the AI analysis engine. This component consumes data through the MCP interface and performs multi-dimensional analysis, correlating signals across logs, metrics, and traces. It also handles anomaly detection, identifying statistical deviations from normal patterns, and root cause determination, using contextual clues to isolate likely sources of issues. For instance, the engine can fetch relevant logs and metrics based on specific request or user IDs within a defined timeframe, analyze statistical properties of service metrics like latency and error rates, and then identify anomalies using statistical methods like z-scores, pinpointing high-severity deviations.
The integration of MCP with observability platforms promises significant improvements in managing and comprehending complex telemetry data. Potential benefits include faster anomaly detection, leading to reduced minimum time to detect (MTTD) and minimum time to resolve (MTTR) incidents. It also facilitates easier identification of root causes, reduces noise and unactionable alerts, thereby combating alert fatigue and enhancing developer productivity. Furthermore, it minimizes interruptions and context switches during incident resolution, improving the operational efficiency of engineering teams.
Key insights from this project highlight the importance of embedding contextual metadata early in the telemetry generation process to facilitate downstream correlation. Structured data interfaces are crucial for creating API-driven, structured query layers that make telemetry more accessible. Context-aware AI should focus its analysis on context-rich data to improve accuracy and relevance. Finally, both context enrichment and AI methods must be continuously refined based on practical operational feedback.
The amalgamation of structured data pipelines and AI holds enormous promise for the future of observability. By leveraging structured protocols like MCP and advanced AI-driven analyses, organizations can transform vast telemetry data into actionable insights, shifting from reactive problem-solving to proactive system management. Lumigo identifies logs, metrics, and traces as the three essential pillars of observability; without their seamless integration, engineers are forced into manual correlation of disparate data sources, significantly slowing incident response. This necessitates not only new analytical techniques to extract meaning but also structural changes in how telemetry is generated.