Ansible from the Trenches

Although its name might suggest something different on Wikipedia, Ansible is an open source framework that is capable of automating various system tasks like setting up a server, deploying and configuring applications and command execution (See Ansibleworks.com). Ansible can run across many virtual machines without the need to augment it with other tools. For distribution on various machines Ansible by default relies on SSH as a transport layer. So aside from an SSH server, Ansible doesn’t require any additional daemons to be running on the remote servers. Also Ansible doesn’t leave any traces on remote systems after its done running. Ansible  depends on the python interpreter that comes with the majority of Linux distributions. And it uses a YAML format to describe configuration and execute commands, which makes it readable.

So why not use Ansible to deploy an application across several virtual machines?
For our project we needed to deploy and configure the following packages on Ubuntu 12.04:

• JDK 7
• Tomcat 7
• Elasticsearch
• Riak
• Application war’s

Our approach

As a non maintainer Ansible is a very lightweight framework that let’s you get started quickly with deployment and configuration. For our project we decided to use playbooks. Playbooks are the basis for configuration management and multi-machine deployment. In a playbook you can declare configurations, and orchestrate the set of steps a.k.a. tasks that need to be performed against all hosts. We started writing a playbook per package. So for our project we had to write 5 playbooks. Each playbook contains its own set of variables and tasks. Tasks are defined in a separate tasks module. Dependent configuration files like empty.policy for Tomcat are contained in a separate files folder. This keeps the playbook simple and maintainable.
For elasticsearch we’ve used an open source project on github as basis. This worked out pretty fine.

And then…

We managed to start fast with the playbooks, but it is the simplicity of the framework that is misleading; thinking you’ve done the steps necessary for deployment but in fact you haven’t.

Lack of feedback…

One of the first things we ran into is the fact that Ansible doesn’t warn if a host’s key changes. You’ll have to make sure that all machines are populated with the proper known-hosts file.

Another major drawback we experienced is the fact that Ansible doesn’t give much feedback when executing steps. So it is not very clear if steps have been performed properly or not. That you’ll have to check yourself. And if something goes wrong you barely have a clue what went wrong and where you should start looking making it very cumbersome.

We also had to download several packages like the Oracle tar.gz, a 90 Mb file, which takes quite some time to complete, even on a high speed network. Unfortunately Ansible doesn’t provide any feedback on the progress of the download.

It should come as no surprise then that we were not worried when a sub task in the Tomcat playbook could sometimes take quite some time to finish. But instead the sub task never completed because it was waiting for user input. This was not visible when running the playbook. It seems to be a conscious decision not to do this (see ansible issue 472)

Figure it out yourself….

Compared to frameworks like Puppet, with Ansible you have to write everything yourself. There is limited support from the framework for OS variables or settings. On one hand lots of freedom, but you do need to have a good and complete idea of what steps need to be performed and how. It also depends on the packages you install. Installing OpenJDK 7 with Ansible is much simpler than installing the Oracle JDK on Ubuntu. why? Because installing OpendJDK on Ubuntu already sets many environment variables as with Oracle you have to set all the variables with the Ansible script. So with the Oracle JDK you’ll have to set many variables manually. To get around this we’ve used the Oracle JDK installer from the WebUpd8 team.

In the end Ansible took us way more time than expected. Although it looks simple, the learning curve is quite high. It can also take quite some time to test and run a playbook, so don’t underestimate the amount of time necessary. We certainly won’t next time, also thanks to my co-writer and colleague in this project Thomas Zeeman. For those of you interested, we’ve also written some other blogs about Ansible in the past.