From five dashboards to one prompt: how we built an APM health monitor with Elastic Agent Builder — Elastic Observability Labs

From five dashboards to one prompt: how we built an APM health monitor with Elastic Agent Builder

Using Elastic Agent Builder, we built an agent that answers one question: is your service healthy?

This agent fans out to five ES|QL queries over your existing traces-* data and answers Red, Yellow or Green with the root cause attached, in one response, without switching dashboards or correlating anything by hand. It runs on Elastic Agent Builder, tested on Elasticsearch and Kibana 9.3, and one deployment covers every service in your fleet with no per-service setup.

Is my service healthy? One question, one agent, one answer

There is a question every engineer dreads during an incident: "Is the service healthy?" It sounds simple. It is not. Answering it properly means switching between dashboards, firing off multiple queries, correlating latency spikes with error logs, and checking whether downstream dependencies are contributing to the problem, all under pressure and often at an inconvenient hour.

What if that entire investigation collapsed into a single conversation?

That is what we set out to build: an AI agent that behaves like a knowledgeable SRE sitting beside you. It knows which questions to ask, knows how to query your APM data, and returns a clear Red, Yellow, or Green health verdict with context and recommendations. We call it the APM Service Health Monitor, and it runs entirely on Elastic.

Why APM health is a derived signal, not a single metric

Here is the thing about APM data: you already have everything you need. Latency percentiles, error rates, throughput, and span-level dependency traces are all sitting in your traces-* indices, indexed and ready.

The missing piece is not data. It is the reasoning layer that connects those signals into a coherent picture.

The APM Service Health Monitor encodes that reasoning (threshold logic, trend comparison, and dependency blast-radius analysis) into an Elastic AI agent that runs it on demand, for any service, anytime.

How does the APM Service Health Monitor architecture work?

The agent is registered in Elastic Agent Builder as a chat-type agent wired to five ES|QL-powered tools. Each tool does one thing precisely. The agent orchestrates, compares, and reasons across all five.

POST kbn:/api/agent_builder/agents
{
  "id": "apm_service_health_agent",
  "type": "chat",
  "name": "APM Service Health Monitor",
  ...
}

Tool 1: apm_metrics_overview_tool, the 24-hour baseline

Every health assessment starts with a baseline. This tool computes the aggregate picture for a service over the last 24 hours (average latency, p95 and p99 latency, error rate, and throughput) in a single ES|QL query:

FROM traces-*
| WHERE service.name == ?service
| WHERE transaction.type == "request"
| WHERE @timestamp >= NOW() - 24 hours
| EVAL is_error = CASE(event.outcome == "failure", 1, 0)
| STATS
    avg_latency_ms  = AVG(transaction.duration.us / 1000),
    p95_latency_ms  = PERCENTILE(transaction.duration.us / 1000, 95),
    p99_latency_ms  = PERCENTILE(transaction.duration.us / 1000, 99),
    error_rate      = 100.0 * SUM(is_error) / COUNT(*),
    throughput_rps  = COUNT(*) / (24*3600)

Tool 2: apm_latency_trend_tool, performance over time

Averages hide inflection points. The latency trend tool buckets p95, p75, and average latency into 5-minute intervals across the last 24 hours, giving the agent a time-series view of performance:

| STATS
    avg_latency_ms = AVG(transaction.duration.us / 1000),
    p95_latency_ms = PERCENTILE(transaction.duration.us / 1000, 95),
    p75_latency_ms = PERCENTILE(transaction.duration.us / 1000, 75)
  BY time_bucket = DATE_TRUNC(5 minutes, @timestamp)
| SORT time_bucket ASC

Tool 3: apm_error_trend_tool, failure shape detection

Error rate is directional. A gradual rise from 0.5% to 1.2% tells a different story than a sudden spike to 8%. The error trend tool captures this shape with 5-minute buckets:

| EVAL is_error = CASE(event.outcome == "failure", 1, 0)
| STATS
    total_requests = COUNT(*),
    error_count    = SUM(is_error),
    error_rate     = 100.0 * SUM(is_error) / COUNT(*)
  BY time_bucket = DATE_TRUNC(5 minutes, @timestamp)

Tool 4: apm_throughput_trend_tool, traffic as a health signal

Throughput is underrated as a health indicator. A 40% drop in requests per second is itself an incident.

| STATS requests_count = COUNT(*)
  BY time_bucket = DATE_TRUNC(5 minutes, @timestamp)
| SORT time_bucket ASC

Tool 5: apm_dependency_health_tool, the blast-radius view

Mapping every dependency via span.destination.service.resource and computing its error rate from span data:

FROM traces-*
| WHERE processor.event == "span"
  AND span.destination.service.resource IS NOT NULL
| EVAL is_error = CASE(event.outcome == "failure", 1, 0)
| STATS
    total_calls  = COUNT(*),
    failed_calls = SUM(is_error),
    error_rate   = 100.0 * SUM(is_error) / COUNT(*)
  BY dependency_name = span.destination.service.resource
| SORT error_rate DESC

How the health scoring works

The overall verdict follows a single rule: the worst individual metric wins.

Any metric is Red                → Overall = 🔴 Red
Any metric is Yellow (no Red)   → Overall = 🟡 Yellow
All metrics are Green            → Overall = 🟢 Green

Deploying the APM Service Health Monitor in Agent Builder

Registration follows Elastic's standard Agent Builder API pattern, tested against Elasticsearch 9.3 and Kibana 9.3:

# Step 1: Register the five ES|QL tools
POST kbn:/api/agent_builder/tools   # apm_metrics_overview_tool
POST kbn:/api/agent_builder/tools   # apm_latency_trend_tool
POST kbn:/api/agent_builder/tools   # apm_error_trend_tool
POST kbn:/api/agent_builder/tools   # apm_throughput_trend_tool
POST kbn:/api/agent_builder/tools   # apm_dependency_health_tool

# Step 2: Register the agent with tools wired in
POST kbn:/api/agent_builder/agents  # apm_service_health_agent

# Step 3: Verify the agent is live
GET kbn:/api/agent_builder/agents/apm_service_health_agent

Requirements and deployment

The next time someone asks "is the service healthy?", you will have a precise, data-backed answer before the question finishes echoing in the incident channel.