Skip to content


Restricting uploads to public PyPI

Many companies use an internal PyPI server for storing their proprietary python packages. This makes managing python libraries and application dependencies so much easier. But unfortunately this also makes it easy for people to accidentally upload their private code to the public PyPI unintentionally.

Lucky for us, there’s a cool extension to setuptools called restricted_pkg! Unlucky for us, it leaves something to be desired in terms of user experience. Let’s say we have an example library called xl which uses restricted_pkg to prevent accidental uploads. Continued…

Posted in Tutorials.


AWS User Policy for Single S3 Bucket

A common requirement is to have a backup service or script that uploads objects to S3 for storage. Since its good practice to scope user permissions as narrowly as possible, this leads to creating separate “api users” in Amazon for each service. Each user is only given permission for the buckets it needs to access. Unfortunately, the Resource URIs for AWS are non-intuitive and you have to remember to whitelist both the bucket and its contents. If you’re kind, you’ll also allow listing all buckets to make navigating through the UI or other tools possible.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::some-bucket-name",
                "arn:aws:s3:::some-bucket-name/*"
            ]
        }
    ]
}

#protip #selfreference

Posted in Tutorials.


How To See A Process’s Environment in Linux

One of our sysadmins recently taught me that we can see the environment with which a process is launched by looking in /proc. Whoa! That’s helpful.

Unfortunately, the environment file is null terminated so not pleasant to read or pipe together with other commands. So here’s a handy one-liner to print them “properly” for easier inspection or command chaining.

cat /proc/{pid}/environ | xargs --null --max-args=1 echo

#protip #selfreference

Posted in Tech Reference.


How Much Does Tax Deferral Save You?

Ever wondered exactly how tax deferral saves you money? Although deferring your income is often cited as a “good thing”, the most common explanation given is a tax arbitrage between your current tax rate and a presumably lower tax rate in retirement. Many people even go so far to say that if your tax rate is the same in retirement then its equivalent, excluding the time value of money. Let’s see if this is actually true.

I’ve put together a small scenario which compares a one-time contribution to a deductible Traditional IRA to a standard investment account (non-tax-advantaged) which is then allowed to grow for the next two decades.

Tax Deferral Advantage

Wow! We realized a 31% gain using a tax-deferred investment option given the same tax rate! (30% in this example)

So how does this deferral advantage change with different tax rates, investment returns, and holding periods? Continued…

Posted in Tutorials.


Life Insurance for the Financial Independence Crowd

As my wife and I move along on our journey to Financial Independence (FI) while thinking about starting a family, we’ve been wondering about getting life insurance. It feels like one of those things we “should” do, but does it really make sense for us? Does it make sense for anyone pursuing Financial Independence?

Life insurance is a bewildering, fear-driven world that, until recently, I didn’t know enough about to think systematically. This post is going to introduce a few systems that have helped me think about life insurance: how to determine how much you need, how seeking FI impacts your needs, and how to build an affordable life insurance plan.
Continued…

Posted in Tutorials.


Mapping SinceDB Files to Logstash File Input

Sometimes you need to know which SinceDB files map to which file inputs for Logstash. This could be for a bug with the file input plugin or to force logstash to reparse a specific file. The contents of a SinceDB file look like

479 0 64515 31175

Not very intuitive, is it?

A little googling will show you that the first field in this file is an inode number. A little more searching will show you how to map from an inode number back to a file path. The rest of this post shows how to put together a little two-liner that will just print the map of all SinceDB files to the monitored files.

Continued…

Posted in Tutorials.


Free StatusPage Hosted On Github & Amazon

After having clients call for status updates during a production outage earlier this week, I started thinking more about the classic “status page.” This is for a side business, so I don’t really want to pay $30/mo to StatusPage.io for this functionality. How could I do something high quality but for low cost?

Luckily, I found an open source statuspage that gets me most of the way there. It allows for hosting on Github Pages, thus decoupling my production infrastructure from the status infrastructure. (Its not very useful if your status page goes down at the same time as your production stuff, so its best to decouple them as much as possible.)

Unfortunately, this project requires manually running a shell command every time we create, update, comment on, or close an issue. While this is OK, in the heat of the moment, the fewer things I need to remember to do the better. I saw an option to pay the creators $30/year to automate this for you, but clicking the link took me to a dead site. Plus I wanted to play with some of the newer AWS stuff anyway.

So without further adieu, let’s walk through using Amazon to automate updates to your status page… for free!

Continued…

Posted in Tutorials.


Building Logstash Pipelines using @metadata

Its common for many companies to run multiple applications on a single physical host or virtual machine. Each of the applications usually has its own log file. A local logstash can be used to read all of these messages, process, and forward to Elasticsearch (or another Logstash or a message queue, anywhere really). You can even logically organize one logstash config file per application, complete with input, filters, and output. So what’s the problem?

How do I ensure that my filters/output only run on the right input?

A common practice is to add a “tags” field on the input and check for it in the filters and output. If you’re diligent about removing this tags field in the output, this can work… but ain’t nobody got time for that.Unfortunately, what often happens is that field is forgotten and ends up in your data downstream. Yuck. So what’s a better pattern?

Logstash 1.5 added the ability to add metadata to an event. This provides the building block for what I like to call the “Logstash Pipeline Pattern”. We can use this metadata to form an independent logstash pipeline (input/filters/output) for every application on the host without running multiple instances of logstash.

Here’s what this looks like in practice. Continued…

Posted in Tutorials.


How To Ingest App Metrics from Slack into Elasticsearch

Recently I started helping Cardbucks, a very early-stage startup team. They’re running pretty bare-bones during their early stage market-fit experiments and haven’t setup any application monitoring or business intelligence solution for their users yet. However, they’ve been logging all user actions to a Slack room from Day One, which is awesome. So for a hack day, I built a bot to scrape the historical messages as well as ingest all new incoming metrics from Slack into Elasticsearch.

Ingest Real-Time Metrics

The first thing was to find an easy bot framework that both lets me receive new messages in (near) real-time. The Slack Team have generously provided the skeleton with python-rtmbot. This is a callback-based bot engine, so we need only write a simple plugin and configure it with our Slack token for ingesting metrics from the real-time message stream.

Below is a simple example of how we did this for the Cardbucks team.  Continued…

Posted in Tutorials.


Guide to the Southwest Companion Pass

Many of us have large travel and lifestyle aspirations for our families, so I wanted to share a travel tip that I’ve put to use for my own family last year. If you haven’t noticed, Southwest has just added a bunch of international flights to beach, mountain, and island destinations like Puerto Rico, Costa Rica, and Belize. Here’s a few tips that will let you fly there for wayyyy cheaper.

While this tip itself won’t change your lifestyle, hopefully it’ll give you to enough juice to be able to explore a few of these locations without breaking the bank. See what you like, start living a couple of your dreams, and allow you more room to chart your course ahead.

Of course, some people aren’t comfortable with the tactics that I’m about to outline since they involve wise use of credit card bonuses and a bit of manufactured spending. So it may not be your thing either and I understand. But I thought I’d share anyway, just in case. :)

What You’ll Need

  1. Reasonable Credit (and Willingness to Use It)
  2. A Good Plan (and Expenses You Can Pay With Credit Cards)

Goal #1: Southwest Points

Southwest Points are awesome because Southwest doesn’t charge drastically more points even for far-flung flights. My wife and I booked honeymoon tickets last year to Costa Rica for 16,800 points each! (Plus $53.03 each in taxes and government fees, still not too shabby.) Continued…

Posted in Tutorials.




Log in here!