Sometimes you need to know which SinceDB files map to which file inputs for Logstash. This could be for a bug with the file input plugin or to force logstash to reparse a specific file. The contents of a SinceDB file look like

479 0 64515 31175

Not very intuitive, is it?

A little googling will show you that the first field in this file is an inode number. A little more searching will show you how to map from an inode number back to a file path. The rest of this post shows how to put together a little two-liner that will just print the map of all SinceDB files to the monitored files.

Continued…

After having clients call for status updates during a production outage earlier this week, I started thinking more about the classic “status page.” This is for a side business, so I don’t really want to pay $30/mo to StatusPage.io for this functionality. How could I do something high quality but for low cost?

Luckily, I found an open source statuspage that gets me most of the way there. It allows for hosting on Github Pages, thus decoupling my production infrastructure from the status infrastructure. (Its not very useful if your status page goes down at the same time as your production stuff, so its best to decouple them as much as possible.)

Unfortunately, this project requires manually running a shell command every time we create, update, comment on, or close an issue. While this is OK, in the heat of the moment, the fewer things I need to remember to do the better. I saw an option to pay the creators $30/year to automate this for you, but clicking the link took me to a dead site. Plus I wanted to play with some of the newer AWS stuff anyway.

So without further adieu, let’s walk through using Amazon to automate updates to your status page… for free!

Continued…

Its common for many companies to run multiple applications on a single physical host or virtual machine. Each of the applications usually has its own log file. A local logstash can be used to read all of these messages, process, and forward to Elasticsearch (or another Logstash or a message queue, anywhere really). You can even logically organize one logstash config file per application, complete with input, filters, and output. So what’s the problem?

How do I ensure that my filters/output only run on the right input?

A common practice is to add a “tags” field on the input and check for it in the filters and output. If you’re diligent about removing this tags field in the output, this can work… but ain’t nobody got time for that.Unfortunately, what often happens is that field is forgotten and ends up in your data downstream. Yuck. So what’s a better pattern?

Logstash 1.5 added the ability to add metadata to an event. This provides the building block for what I like to call the “Logstash Pipeline Pattern”. We can use this metadata to form an independent logstash pipeline (input/filters/output) for every application on the host without running multiple instances of logstash.

Here’s what this looks like in practice. Continued…

Recently I started helping Cardbucks, a very early-stage startup team. They’re running pretty bare-bones during their early stage market-fit experiments and haven’t setup any application monitoring or business intelligence solution for their users yet. However, they’ve been logging all user actions to a Slack room from Day One, which is awesome. So for a hack day, I built a bot to scrape the historical messages as well as ingest all new incoming metrics from Slack into Elasticsearch.

Ingest Real-Time Metrics

The first thing was to find an easy bot framework that both lets me receive new messages in (near) real-time. The Slack Team have generously provided the skeleton with python-rtmbot. This is a callback-based bot engine, so we need only write a simple plugin and configure it with our Slack token for ingesting metrics from the real-time message stream.

Below is a simple example of how we did this for the Cardbucks team.  Continued…

Of late I’ve been interested in the intersection of containerization and 12factor apps.

This led me to piecing together a small proof-of-concept 12factor microservice on top of Docker. I’ve published this effort as a two-part tutorial on PacktPub.

1. Introduction and the first 4 Factors
2. The Remaining 8 Factors and PaaS Preview

My company has slowly started experimenting with containers… now to move our (stateless) apps toward 12factor! 🙂

Last night I was up hacking away on a side project for work. Some of my favorite “quick wins” involve automating and standardizing our operational infrastructure. This project involved generating HAProxy configs from Jinja templates instead of painstakingly managing each server’s (or server group’s) configuration by hand.

In the latest iteration, I was trying to automatically load a common macro in every template without requiring every template to specify it. This is equivalent to adding

{% from 'macros.txt' import macro1, macro2 with context %}

at the top of every template file. But of course I don’t want to repeat myself everywhere.

This article shows two techniques for achieving this goal, depending on Continued…