Our more general operations philosophy we can go into in other posts, but in this I wanted to introduce a tool I wrote recently, and I think it's a good microcosm of how we're thinking at the moment. We value simplicity and the composing of existing, known-good technologies over bringing in grand new systems.

Moltar is a command-line interface to the AWS API and SSH client. The first ingredient is a consistent concept of how to organise your AWS resources. We have an AWS account per client (or sometimes client-project), then descending: projects, environments, clusters, and instances within that. The next is a metadata convention. For the uninitiated, AWS allows you to tag nearly everything you create within it with short, key-value pairs. Now everything can be tagged with app, environment, and cluster.

Now it's coming together. The AWS API allows you to list resources (most commonly for us, EC2 instances) filtering by their tag. This is kind of the bedrock: it's difficult to build useful tools around AWS without this kind of categorisation.

I'm in a project repository in my shell:

It's a modest start. Moltar has picked up the context: the project name, and consequently the name of the profile in my ~/.aws/config file which holds my credentials (the same as the excellent AWS CLI tool uses). Other subcommands can match the package the current directory says it produces with a Packages tag on each instance, but for now we're showing everything. Then we can filter down to the cluster:

Now we have a set of similar instances we can SSH to. Let's SSH to one. -1 just selects the first encountered. This may change between re-runs.

We don't create instances with their own SSH key pair: instead they're configured with the team's public keys. In this case, there's only one instance in the jobs cluster. For the web cluster, though:

and then:

So the last argument can disambiguate over multiple possible machines, using the shortest unique identifier. And so the question naturally arises: could we execute commands in parallel, across a cluster? Sure:

We're taking inspiration from the Unix philosophy here: building small-ish, composable tools. Let's SCP a file to all instances in a cluster:

Yes, Moltar is seemingly becoming a bit of a toolbox, violating another Unix principle of a tool doing one thing well – but I think the benefit from having a consistent interface for talking about infrastructure outweighs that. If we felt like it, this could be split into a tool that outputs hostnames, and the multi-SSH/SCP tools so that you could pipe one to the other.

Moltar is a pretty stable v1.0 right now. Originally the deploy command was going to do much more clever things (clues still exist in the git history), so that deployment wasn't orchestrated by the tool, but those plans are on hold for the moment in favour of a more straight-forward Capistrano-style deployment. In this state, it's used daily by both regular developers and operations. Why not just use Capistrano? This is a factoring-out of the way we categorise EC2 instances, and as we have strong conventions it requires much less configuration. Also, it exists within a single binary with no other dependencies. Using Capistrano doesn't cause much friction in a Rails project, but isn't as hassle-free elsewhere.

The other main reason is that we're not doing anything particularly complex when we deploy. No bundle install, no asset precompilation. Another standard and mature technology we use is Debian packaging. After our CI has successfully run tests, it builds a package of the app and uploads it to a private APT repository. Deployment is then as easy as upgrading a package – and we can use all of the infrastructure around that to our benefit too. Production instances have to do less on a deployment, just move files into place and nudge the Rack server. A future post will look into how we create these packages.

It's given me a nice opportunity to write something non-trivial in Go, which is a language that has become rather popular recently. Enough other people have written about Go and my opinions aren't all that interesting, but essentially I like it for what it is (but I have my eye on Rust).

We're pleased to open up the source code to Moltar. It's installable through go get and Homebrew – details in the Readme on GitHub.

So there's a look into a part of how we're thinking about operations at the moment: simple interfaces, using agreed-upon concepts expressed with well-kept metadata, composing known-good technology. For the scale we work at (which can still power some pretty well-used sites), we don't need anything more complex.