Building Logstash Pipelines using @metadata

Its common for many companies to run multiple applications on a single physical host or virtual machine. Each of the applications usually has its own log file. A local logstash can be used to read all of these messages, process, and forward to Elasticsearch (or another Logstash or a message queue, anywhere really). You can even logically organize one logstash config file per application, complete with input, filters, and output. So what’s the problem?

How do I ensure that my filters/output only run on the right input?

A common practice is to add a “tags” field on the input and check for it in the filters and output. If you’re diligent about removing this tags field in the output, this can work… but ain’t nobody got time for that.Unfortunately, what often happens is that field is forgotten and ends up in your data downstream. Yuck. So what’s a better pattern?

Logstash 1.5 added the ability to add metadata to an event. This provides the building block for what I like to call the “Logstash Pipeline Pattern”. We can use this metadata to form an independent logstash pipeline (input/filters/output) for every application on the host without running multiple instances of logstash.

Here’s what this looks like in practice. In this example, we’re reading logs from a Kafka topic, dropping unparseable logs or other processing, and writing each type of log out into its own Kafka topic (based on the event’s “type” field).

# split-logs.conf
input {
  kafka {
    zk_connect => 'zookeeper.mydomain.com:2181'
    topic_id => 'logstash_logs'
    add_field => { "[@metadata][route]" => "split-logs" }
  }
}

filter {
  if [@metadata][route] == "split-logs" {
    if "_jsonparsefailure" in [tags] {
       drop {}
    }
  }
}

output {
  if [@metadata][route] == "split-logs" {
      kafka {
        topic_id => "%{[type]}"
        broker_list => 'kafka.mydomain.com:9092'
      }
  }
}

Clearly this pattern makes it much easier to isolate an individual Logstash Pipeline. Such clarity makes it much easier to reason about what’s going on, thus preventing bugs and making troubleshooting much easier.