How to use Erlang application release with multiple configurations

On my opinion configuring and building releases for an Erlang applications has always been the most obscure and mis-understood part of Erlang ecosystem. For me personally, to build a release would always take a day or two because I could never set all the settings correctly in reltool.config and erlang release invariably failed with the most un-helpful error message.

The only Erlang book which explained in details how Erlang releases were supposed to work was Erlang and OTP in Action

Fortunately, one of the authors of this book, Eric Merritt, wrote a great tool relax and building releases became much simpler. Eric gave a talk about it. If you use Erlang you definitely need to check Relx, it will make you life much easier.

But some issues still remained. Relx.config allows you to specify sys.config and vm.args files for your release that set the configurations for VM (e.g. a name of the node , -sname or -name, cookie, etc) and some of these settings were not possible to change if you wanted to run multiple instances of your release. E.g. overriding -name parameter via command line argument wasn’t possible.

This was massively inconvenient  as you would need to build multiple releases of your application if, for example, you needed to run a cluster of several nodes. In this case you would use overlays in rebar.config to override some params of the release and generate a separate release for each node. Example of such configuration you can find here.

Tristan Sloughter fixed this issue with one of his many additions to relx.

His solution (which is unfortunately not documented in relx official documentation yet) makes use of additional env var $RELX_REPLACE_OS_VARS. If this var is set to true, the application start up script generated with relx will replace placeholders in vm.args and sys.config files with values of env variables which match the values of placeholders. It will then create vm2.args and sys2.config (with replaced placeholders values) and start your application with these config files instead of original ones. This allows you to run the same release with multiple configs and override settings for PROD and QA for example.

for example if you want to override the name of the node and the cookie:

is your vm.args you add placeholders that you want to replace:

-name ${NODE_NAME}
-cookie ${COOKIE}

then in your start-prod.sh file

export RELX_REPLACE_OS_VARS=true
export NODE_NAME=node_prod
export COOKIE=prod
exec _rel/your_app/bin/your_app foreground "$@"

and in start-qa.sh file

export RELX_REPLACE_OS_VARS=true
export NODE_NAME=node_qa
export COOKIE=qa
exec _rel/your_app/bin/your_app foreground "$@"

make sure you generate your release with at least 1.1.0 version of relx.

Kudos to Tristan for this great solution, I asked the question on this Issue, otherwise I might’ve never found this override.

Posted in Uncategorized | Leave a comment

How to learn to type on Kinesis Advantage

I have recently got myself a new keyboard – https://www.kinesis-ergo.com/shop/advantage-for-pc-mac/

advantage-black

I won’t go in length into advantages / disadvantage of using it – you can find all

these discussions somewhere else. So far I am loving it – mechanical keys are definitely a way to go. I am heavy emacs user and the ability to reprogram the key binding is a massive plus.

But this post is about a little LifeHack. This keyboard will make you cry for at least couple of weeks until you re-learn to type again as all the key chords that your fingers remember won’t work for you anymore.

You can find some Training material but they are in paper format. The better way to use them is to feed these excercises into some touch typing  application.

Fortunetely, TIPP10 if perfect app for this as it allows you to use your own exercises.

You can download the lesson files from here

Posted in Uncategorized | Leave a comment

How to manage public and private Puppet modules with Vagrant

If you use Puppet for provisioning Vagrant VMs, the chances are that you want to use some standard public Puppet modules which could be found at Puppet Forge and the question becomes how to get them installed as part of provisioning process? You see, you can’t just download them as part of your Puppet provisioning as Puppet catalog will not compile because the modules will be missing at the time of Puppet run. It is a chicken and egg situation.

You can, of course, manually download the modules from Forge and store them in the repository of your project in puppet/modules directory alongside with your own Puppet code. This is rather messy, you have to manually maintain the dependencies and it makes your repository bigger than necessary. There has to be a better way of doing this.

I did some reading and found few ideas how to deal with this issue. One possibility is to make use of Puppet module Librarian-puppet. Librarian-puppet, once installed, allows you to specify a list of Puppet modules that your infrastructure depends on in Puppetfile and it will manages the installation of these modules and their dependencies for you.

The trick is you have to install Librarian and make it install modules from Forge *before* you can run your Puppet provisioner, otherwise Puppet will complain about missing modules while compiling the catalog. So, we need to use shell provisioner, get Librarian-puppet installed and then use it to install modules from Forge. After that you can safely use Puppet provisioner to run your own Puppet code knowing that all the dependencies that your code needs have been already installed.

Here are the steps:

In your Vagrantfile add the following line:

  # install librarian-puppet and run it to install puppet common modules.
  # This has to be done before puppet provisioning so that modules are available
  # when puppet tries to parse its manifests
  config.vm.provision :shell, :path => "provision/shell/main.sh"
 

It should be added *before* any puppet provisioners.

This will run main.sh which for lives in /provision/shell directory and looks like this:
(I borrowed this script from this project , most of the ideas in this post come from it)

#!/usr/bin/env bash

# Directory in which librarian-puppet should manage its modules directory
PUPPET_DIR=/vagrant/provision/puppet

# NB: librarian-puppet might need git installed. If it is not already installed
# in your basebox, this will manually install it at this point using apt or yum

$(which git > /dev/null 2>&1)
FOUND_GIT=$?
if [ "$FOUND_GIT" -ne '0' ]; then
  echo 'Attempting to install git.'
  $(which apt-get > /dev/null 2>&1)
  FOUND_APT=$?
  $(which yum > /dev/null 2>&1)
  FOUND_YUM=$?

  if [ "${FOUND_YUM}" -eq '0' ]; then
    yum -q -y makecache
    yum -q -y install git
    echo 'git installed.'
  elif [ "${FOUND_APT}" -eq '0' ]; then
    apt-get -q -y update
    apt-get -q -y install git
    echo 'git installed.'
  else
    echo 'No package installer available. You may need to install git manually.'
  fi
else
  echo 'git found.'
fi

if [ "$(gem search -i librarian-puppet)" = "false" ]; then
  gem install librarian-puppet
  cd $PUPPET_DIR && librarian-puppet install --path modules-contrib
else
  cd $PUPPET_DIR && librarian-puppet update
fi

Main.sh cd-es into your provision/puppet directory which will be mounted to Vagrant VM as /vagrant/provision/puppet and installs Librarian. It then reads Puppetfile and installs the Puppet modules defined in Puppetfile into provision/puppet/modules-contrib directory on your host machine. Librarian-puppet installs modules to the directories relative to location of Puppetfile and you can specify the name of the root directory by passing –path param to librarian-puppet when it is run first time:

librarian-puppet install --path modules-contrib

Puppetfile just gives a location of forge and lists modules that you want to pull off forge (the format is flexible enough to be able to pull code repos from github as well) :

# Puppetfile
# Configuration for librarian-puppet. For example:
# 
forge "http://forge.puppetlabs.com"
mod "garethr/docker"
mod "camptocamp/archive"
mod "puppetlabs/vcsrepo"
mod "maestrodev/wget"
mod "puppetlabs/git"

And finally, in your Vagrantfile for Pupper provisioner configuration you need to define “module_path” and list the locations of your custom puppet modules and the public ones which you installed with Librarian:

  # Provide basic configuration, install git
  config.vm.provision "puppet" do |d|
    d.manifests_path = 'provision/puppet/manifests'
    d.manifest_file = 'site.pp'
    d.module_path = [ 'provision/puppet/modules-contrib', 'provision/puppet/modules' ]
    #d.options = "--verbose --debug"
  end

make sure that empty “modules-contrib” folder exists in your project, it needs to be there at the time when you do ‘vagrant up’ otherwise Vagrant won’t be able to mount this folder to your VM.

git doesn’t allow to keep empty directories but you can get around this by adding the following .gitignore into modules-contrib folder.

# Ignore everything in this directory
*
# Except this file
!.gitignore
Posted in Puppet, Vagrant | Tagged , | 2 Comments

How to make Vagrant and Puppet to clone private github repo

I have been playing with Vagrant and Puppet lately to automate my home development environment and got to the point when I needed to clone / build code from several private github repos. I ended up spending two days to make this work as it turned out to be more difficult then I expected.

1. To start with you need to have working SSH private / public keys on your host computer. Public key needs to be added to your github private repo. Follow this instructions to setup and test.

2. Now you have a private key on your machine and public key sitting on github side. You also want to enable SSH agent forwarding to save you time of typing in the passphrase associated with your private key every time you login to github. Here are the instructions how to enable ssh forwarding.

3. At this point you probably want to get you Vagrant provisioning process to be able to login to your private github repo as well. There are two ways of doing this:

  1. you can setup a Vagrant task which will find your private and public keys and copy them over to Vagrant VM (into /root/.ssh because Vagrant runs provisioning using root account). This is possible but not very convenient.
  2. much better way is to enable Vagrant VM to use already existing ssh keys from your host machine. This will allow you to share your Vagrantfile with other developers who might have access to your private repo and they will be able to use their SSH keys

Add the following to your Vangrantfile:

 config.ssh.private_key_path = [ '~/.vagrant.d/insecure_private_key', '~/.ssh/id_rsa' ]
 config.ssh.forward_agent = true

You also need to add “github.com” hostname into a list of ssh known hosts in your Vagrant VM.

The problem is even if you enable ssh forwarding from your Vagrant VM, when cloning job makes a first time connection to github it will get the following message and fail

RSA key fingerprint is 16:27:ac:a5:7c:28:2d:36:63:2b:56:4d:eb:df:a6:48.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'github.com,205.232.175.93' (RSA) to the list of known hosts.

One way around this is to set option StrictHostKeyChecking=no for ssh forwarding agent but this opens you up to certain security risk. Better way is to add github.com to /root/.ssh/known_hosts as part of provisioning. You can do it with the following rule in Vagrantfile which needs to be executed before you make ssh connection to github:

  # add github to the list of known_hosts
  config.vm.provision :shell do |shell|
    shell.inline = "mkdir $1 && touch $2 && ssh-keyscan -H $3 >> $2 && chmod 600 $2"
    shell.args = %q{/root/.ssh /root/.ssh/known_hosts "github.com"}
  end

or ( better! ) use the following Puppet module:

 # -*- mode: ruby -*-
# vi: set ft=ruby :

class known_hosts( $username = 'root' ) {
    $group = $username
    $server_list = [ 'github.com' ]

    file{ '/root/.ssh' :
      ensure => directory,
      group => $group,
      owner => $username,
      mode => 0600,
    }

    file{ '/root/.ssh/known_hosts' :
      ensure => file,
      group => $group,
      owner => $username,
      mode => 0600,
      require => File[ '/root/.ssh' ],
    }

    file{ '/tmp/known_hosts.sh' :
      ensure => present,
      source => 'puppet:///modules/known_hosts/known_hosts.sh',
    }

    exec{ 'add_known_hosts' :
      command => "/tmp/known_hosts.sh",
      path => "/sbin:/usr/bin:/usr/local/bin/:/bin/",
      provider => shell,
      user => 'root',
      require => File[ '/root/.ssh/known_hosts', '/tmp/known_hosts.sh' ]
    }
}

this rule will create /root/.ssh directory and execute the following known_hosts.sh

#!/bin/bash
array=( 'github.com' )
for h in "${array[@]}"
do
    #echo $h
    ip=$(dig +short $h)
    ssh-keygen -R $h
    ssh-keygen -R $ip
    ssh-keyscan -H $ip >> /root/.ssh/known_hosts
    ssh-keyscan -H $h >> /root/.ssh/known_hosts
 done

you need to add this bash script into ‘files’ directory of your module, e.g. for me it lives in “puppet/modules/known_hosts/files/known_hosts.sh”

All you need now is the rule to clone the github repo, I am using Puppet module vcrepo and it looks like this:

    vcsrepo { "/opt/code/${repo}":
      ensure => latest,
      owner => $username,
      group => $username,
      provider => git,
      require => [ Package[ 'git' ] ],
      source => "git@github.com:<your account name>/<your project name>.git",
      revision => 'master',
    }

Here we clone from github into local Vagrant directory “/opt/code/repo” and this directory will be owned by username that you can define.

You can also add a rule to build your code:

    exec{ 'make': 
      command => "make",
      environment => "HOME=/home/${username}",
      cwd => "/opt/code/${repo}",
      path => "/sbin:/usr/bin:/usr/local/bin/:/bin/",
      require => [ Vcsrepo[ "/opt/code/${repo}" ], Package[ 'erlang' ] ],
    }

Hopefully these instructions will save you some time.

Posted in Vagrant | Tagged , , , | 2 Comments

Docker, Puppet and taking care of chaos

Out of many technical talks I have recently watched I am finding the one delivered by Tomas Doran at Puppet Camp the most impressing. You can find it here.

Despite the flamboyant presentation style (and silly hair!) – the topic of the talk is dead serious : what is the sane way to *consistently* rebuild your production environment which is made up out of many services? And by extension : how to make your Dev environment to match your production?

Tomas’s answer to this question is Docker and Puppet. I have already briefly touched Vagrant and Docker.

Docker is the new technology which is quickly getting momentum and it allows you to take all the code for your application and all the dependencies and build this mess into a Linux container which is portable across the platforms. From practical perspective you will have a single file (Docker Image) which you can create on your laptop and then deploy at thousands of nodes in the cloud as a service. If you have to learn one technical thing this year – make it to be Docker : Introduction to Docker

Puppet is extremely powerful and flexible management and configuration tool. It allows you to script up all the steps required to build your VM – all the packages that needs to be installed, all the directories which need to be created, all the sources that need to be pulled from github and built and deployed, all the config values that need to be set on your box. It allows you to define *everything*.

Tomas talks about how to combine Puppet and Docker together and never again have a need to log on to your Prod box to make a config change but instead make a change in Puppet script and re-deploy your Prod VM, in his words the VMs should be immutable.

And of course if you add Vagrant in this mix, you can also have you Dev VM to be built from the same Puppet scripts as your Prod making you Dev highly consistent with Prod.

Posted in Vagrant | Tagged , , | Leave a comment

How to swap Option (Alt) and Command keys on Mac

If you are (like I) an emacs user on MacOS X you might have this problem : how to re-assign Meta key from Option (Alt) to Cmd key in Terminal (if you don’t know what the heck I am talking about – congratulations! you are free of emacs virus).

For some mysterious reason this seems to be a huge problem, I tried few things (e.g. remapping with keyremap4macbook and cmd-key-happy) but the only reliable solution which worked so far is this:

1. Install iTerm2
2. In the “Keys” tab in configuration change “Left Command” to map to the “Right Option” key, and then under the “Profiles” tab changed “Right Option” key to act as Esc. And then you will have Cmd as Meta

I found this suggestion here

Posted in Uncategorized | Tagged | Leave a comment

How to configure RabbitMQ to run as Docker container in Vagrant

Vagrant is one of those technologies that hit you like a truck and you don’t understand anymore how you lived without it before.

If you haven’t heard about Vagrant yet – it is a simple mechanism to automatically build a complete development environment which is sandboxed in a VM. By default it uses VirtualBox but could be used with other providers, including VMware, AWS, Google Compute Engine, etc. It seamlessly integrates with things like Chef, Puppet and Docker.

From practical perspective it works like this: you add to your git repo a Vagrantfile which is a configuration of VM image and software that you want to have installed for your project. When you clone this repo to your local machine you need to run a single command: “vagrant up” and Vagrant will setup a VM and install all the dependencies. Depending on the complexity of configuration this might take few mins but when Vagrant is done, you will have completely isolated dev environment which you can start, halt, destroy, etc. The main idea is that you have exactly the same development setup as you use in production so entire class of bugs (“… but it worked on my laptop!”) is eliminated and Vagrant takes away all the pain usually associated with repeatable env setup.

Vagrant is platform independent so you can continue using your favorite OS (MacOs, Linux or Windows) to run Ubuntu, CentOS or any other VM image, you of course can continue using your text editor and Vagrant will take care of file synch to VM.

There are plenty of resources on Vagrant but you probably should start with talk by Vagrant developer Mitchell Hashimoto and with Vagrant book.

For one of my recent projects I wanted to use RabbitMQ Docker container and here are the steps how Vagrant, RabbitMQ and Docker could be used together.

First you need RabbitMQ Docker image, you can build one yourself but there are quite a few of them already available at Docker index, I picked up the most recent one built by
Mikael Gueck

2. Now you need to install VirtualBox and Vagrant at your machine.
Install virtualbox from https://www.virtualbox.org/ (or use your package manager)
Install vagrant from http://www.vagrantup.com/ (or use your package manager)

3. Next step you need Vagrantfile which describes VM configuration:


# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  # All Vagrant configuration is done here. The most common configuration
  # options are documented and commented below. For a complete reference,
  # please see the online documentation at vagrantup.com.

  # Every Vagrant virtual environment requires a box to build off of.
  config.vm.box = "precise64"

  # The url from where the 'config.vm.box' box will be fetched if it
  # doesn't already exist on the user's system.
  config.vm.box_url = "http://files.vagrantup.com/precise64.box"

  # Create a forwarded port mapping which allows access to a specific port
  # within the machine from a port on the host machine. In the example below,
  # accessing "localhost:8080" will access port 80 on the guest machine.
  #config.vm.network :forwarded_port, guest: 80, host: 8080
  config.vm.network :forwarded_port, guest: 15672, host: 15672, auto_correct: true 
  config.vm.network :forwarded_port, guest: 4369, host: 4369,   auto_correct: true
  config.vm.network :forwarded_port, guest: 5672, host: 5672,   auto_correct: true

  # Create a private network, which allows host-only access to the machine
  # using a specific IP.
  config.vm.network :private_network, ip: "192.168.33.10"

  # Load and start rabbitmq image
  config.vm.provision "docker" do |d|
    d.pull_images "mikaelhg/docker-rabbitmq"
    d.run "mikaelhg/docker-rabbitmq",
      args: "-h rabbithost -p :5672 -p :15672"
  end
end

this config will build Ubuntu based VM, will install RabbitMQ Docker container and start it with ports 5672 and 15672. The docker image will also have RabbitMQ management console plugin installed. The image will add NAT port mappings and also expose IP “192.168.33.10” for this VM.

4. Now you can run : “vagrant up” command and Vagrant will build and start the VM.

with “vagrant ssh” you can log into the VM console and with “docker ps -a” you can inspect running Docker containers.

here is the screenshot with the steps:
Screen Shot 2014-01-12 at 11.20.15

5. the final step: run http://192.168.33.10:49153/ from your browser (RabbitMQ login / password is “guest” / “guest”)

if everything has started correctly you should get RabbitMQ Management Console in your browser.

In conclusion: both Vagrant and Docker are absolutely amazing technologies and you should be aware of their usage.

Here is the gitbub repo with already preset project.

run:
git clone https://github.com/RomanShestakov/docker-rabbitmq.git
cd docker-rabbitmq
vagrant up
go to http://192.168.33.10:49153/

Hopefully this post would be helpful to somebody.

Posted in Uncategorized | Tagged , , | 2 Comments