ECS Logging with Winston | ECS Logging Node.js

ECS Logging with Winston

Serverless

Stack

This Node.js package provides a formatter for the winston logger, compatible with Elastic Common Schema (ECS) logging. In combination with the Filebeat shipper, you can monitor all your logs in one place in the Elastic Stack. winston 3.x versions >=3.3.3 are supported.

Setup

Step 1: Install

$ npm install @elastic/ecs-winston-format

Step 2: Configure

const winston = require('winston');
const { ecsFormat } = require('@elastic/ecs-winston-format');

const logger = winston.createLogger({
  format: ecsFormat(/* options */),
  transports: [
    new winston.transports.Console()
  ]
});

logger.info('hi');
logger.error('oops there is a problem', { err: new Error('boom') });
  1. Pass the ECS formatter to winston here.

Step 3: Configure Filebeat

The best way to collect the logs once they are ECS-formatted is with Filebeat:

For Filebeat 7.16+

filebeat.inputs:
- type: filestream
  paths: /path/to/logs.json
  parsers:
    - ndjson:
      overwrite_keys: true
      add_error_key: true
      expand_keys: true

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

For Filebeat < 7.16

filebeat.inputs:
- type: log
  paths: /path/to/logs.json
  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
  json.expand_keys: true

processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~

Kubernetes

  1. Make sure your application logs to stdout/stderr.
  2. Follow the Run Filebeat on Kubernetes guide.
  3. Enable hints-based autodiscover.
  4. Add these annotations to your pods that log using ECS loggers.
annotations:
  co.elastic.logs/json.overwrite_keys: true
  co.elastic.logs/json.add_error_key: true
  co.elastic.logs/json.expand_keys: true

Docker

  1. Make sure your application logs to stdout/stderr.
  2. Follow the Run Filebeat on Docker guide.
  3. Enable hints-based autodiscover.
  4. Add these labels to your containers that log using ECS loggers.
labels:
  co.elastic.logs/json.overwrite_keys: true
  co.elastic.logs/json.add_error_key: true
  co.elastic.logs/json.expand_keys: true

Usage

const winston = require('winston');
const { ecsFormat } = require('@elastic/ecs-winston-format');

const logger = winston.createLogger({
  level: 'info',
  format: ecsFormat(/* options */),
  transports: [
    new winston.transports.Console()
  ]
});

logger.info('hi');
logger.error('oops there is a problem', { foo: 'bar' });

Error logging

By default, the formatter will convert an err meta field that is an Error instance to ECS Error fields. For example:

const winston = require('winston');
const { ecsFormat } = require('@elastic/ecs-winston-format');
const logger = winston.createLogger({
  format: ecsFormat(),
  transports: [
    new winston.transports.Console()
  ]
});

const myErr = new Error('boom');
logger.info('oops', { err: myErr });

will yield (pretty-printed for readability):

% node examples/error.js | jq .
{
  "@timestamp": "2021-01-26T17:25:07.983Z",
  "log.level": "info",
  "message": "oops",
  "error": {
    "type": "Error",
    "message": "boom",
    "stack_trace": "Error: boom\n    at Object.<anonymous> (..."
  },
  "ecs.version": "8.10.0"
}

HTTP Request and Response Logging

With the convertReqRes: true option, the formatter will automatically convert Node.js core request and response objects when passed as the req and res meta fields, respectively.

const http = require('http');
const winston = require('winston');
const { ecsFormat } = require('@elastic/ecs-winston-format');

const logger = winston.createLogger({
  level: 'info',
  format: ecsFormat({ convertReqRes: true }),
  transports: [
    new winston.transports.Console()
  ]
});

const server = http.createServer(handler);
server.listen(3000, () => {
  logger.info('listening at http://localhost:3000')
});

function handler (req, res) {
  res.setHeader('Foo', 'Bar');
  res.end('ok');
  logger.info('handled request', { req, res });
}

Log Correlation with APM

This ECS log formatter integrates with Elastic APM. If your Node app is using the Node.js Elastic APM Agent, then a number of fields are added to log records to correlate between APM services or traces and logging data:

Limitations and Considerations

The ecs-logging spec suggests that the first three fields in log records should be @timestamp, log.level, and message. As of version 1.5.0, this formatter does not follow this suggestion. It would be possible but would require creating a new Object in ecsFields for each log record. Given that ordering of ecs-logging fields is for human readability and does not affect interoperability, the decision was made to prefer performance.

Reference

ecsFormat([options])