Elastic z/OS ingest: five architectures for mainframe data

This field guide walks through the ingest architectures I've seen work in production, the data quality checks that decide whether your dashboards actually work, and the ECS mapping that makes mainframe data usable to the platform.

Mainframe teams want what every other observability team already has

Anomaly detection, machine learning (ML) on the batch windows, and alerts that fire when something's actually wrong. Most of them have the data for it. What they don't have is data that the platform can recognize as unified, connected, and operationally meaningful.

A customer described it to me this way: A single transaction passes through three products on its way through the mainframe, and each one names the same field differently (system name, program name, user). Getting the data into Elastic isn't the hard part; getting it to correlate across products, so that Elastic's dashboards and ML jobs recognize it as the same data, is where most projects fall short.

Validate your mainframe source data before you write a pipeline

The most expensive failures I've seen in mainframe ingest projects are the ones that don't fail loudly. Pipelines run, data lands, dashboards render, and weeks later, someone notices that half the events from one logical partition (LPAR) never parsed or a quiet typing change has been silently corrupting a field.

Two mistakes come up frequently:

1. Format inconsistency across LPARs and time windows: Log formats vary across LPARs, between batch and online windows, and across shift changes. A format that parses cleanly in a dev LPAR may not match what production emits during peak batch. This is the single most common cause of partial parse failures I run into.

2. Sample configurations treated as production configurations: A common cause of "it broke overnight" incidents: The upstream collector configuration was based on a sample structure shipped by the vendor and then never replaced with a deliberate production configuration. When the vendor pushed an update, naming and typing changed (fields renamed, types shifted) and the downstream pipeline started rejecting records mid-flight.

Worth knowing first:

For mainframe environments, the Elastic integrations catalog is short. The IBM MQ integration is the most complete option, though the metrics data stream requires the containerized MQ distribution rather than native z/OS MQ. If your architecture includes Customer Information Control System (CICS) workloads or you need end-to-end distributed tracing, assess IBM Z Observability Connect before building custom pipelines.

ECS alignment: from data in Elastic to data Elastic can use

Before choosing an ingest strategy, it's worth understanding why ECS alignment comes first in practice, even if the pipeline gets built later. It's the decision that determines whether everything else pays off.

A mainframe team's core mission:

Trace a single transaction from a REST call into z/OS Connect, through to an Information Management System (IMS) application, and back. That flow touches three products, each emitting telemetry with its own field names for the same concepts (system name, program name, user, transaction ID). Without normalization, correlating that transaction means writing queries that explicitly union three different field names per concept. That’s expensive to write and fragile when any product changes its schema.

ECS solves this. It defines a consistent target schema (host.name, process.name, user.name, event.code) that every source maps into. Once z/OS Connect, IMS Connect, and IMS data all land in the same ECS fields for the same logical concepts, that cross-product transaction trace becomes a single query.

Reference: ECS process fields and ECS event fields.

Extend strategically with custom ECS fields

Mainframe-specific concepts have no ECS equivalent: job class, ASID, SMF record type and subtype, sysplex name, WTO routing codes, CICS transaction ID. Flattening these into labels.* as untyped strings destroys type information and makes them effectively unusable for queries and aggregations.

Define a zos.* custom namespace using ECS's documented extension mechanism. It keeps your core telemetry ECS-compliant while retaining the operational context your mainframe team needs for incident response.

Use ecs@mappings to stay current

Include ecs@mappings as a component template in your index template. It provides Elastic-maintained ECS field definitions automatically and keeps them current with each Elasticsearch release. For custom pipelines, this is what keeps your ECS alignment from drifting over time without manual upkeep.

One important caveat from the field: ecs@mappings provides the field definitions but doesn't enforce types at ingest. A return code arriving as a string is accepted and mapped as a string. Monitoring these discrepancies is critical.

Choose the right architecture for the source

Most environments I work with run more than one of these ingest architectures, and different data sources have different latency, throughput, and licensing characteristics. A single architecture rarely covers everything.

Flow diagram of the ingest paths from z/OS to Elastic.

IBM Z CDP: The workhorse for z/OS operational data

IBM Z CDP is the most widely deployed first-mile collector for z/OS operational data. It reads from SMF datasets in near-real time and forwards off-platform. CDP forwards to Logstash, which handles parsing, field extraction, and routing into Elasticsearch.

Build the pipeline, and don't start from scratch

For the CDP and IMS Connect Extension architectures, log data lands in Elastic reflecting the limited structure of its source. Writing a pipeline from scratch has historically been the largest time sink in any onboarding project. AI has changed that.

Verify mainframe data quality before you build on it

Don't build dashboards or alerting rules on data you haven't verified. The Data Quality dashboard tells you whether your ECS alignment is real or aspirational.

Getting started with mainframe data onboarding

The mainframe is a first-class observability target, and the path there is more concrete than it was a few years ago. Managed integrations cover IBM MQ. CDP and Kafka-based architectures have well-understood deployment patterns.

Recommended order of operations:

1. Validate the source data.
2. Use OOTB integrations where they exist.
3. Align to ECS early.
4. Choose architectures source by source.
5. Generate pipelines rather than write them from scratch.
6. Configure the failure store from day one.
7. Verify before building anything on top.