Docker, Puppet and taking care of chaos

Out of many technical talks I have recently watched I am finding the one delivered by Tomas Doran at Puppet Camp the most impressing. You can find it here.

Despite the flamboyant presentation style (and silly hair!) – the topic of the talk is dead serious : what is the sane way to *consistently* rebuild your production environment which is made up out of many services? And by extension : how to make your Dev environment to match your production?

Tomas’s answer to this question is Docker and Puppet. I have already briefly touched Vagrant and Docker.

Docker is the new technology which is quickly getting momentum and it allows you to take all the code for your application and all the dependencies and build this mess into a Linux container which is portable across the platforms. From practical perspective you will have a single file (Docker Image) which you can create on your laptop and then deploy at thousands of nodes in the cloud as a service. If you have to learn one technical thing this year – make it to be Docker : Introduction to Docker

Puppet is extremely powerful and flexible management and configuration tool. It allows you to script up all the steps required to build your VM – all the packages that needs to be installed, all the directories which need to be created, all the sources that need to be pulled from github and built and deployed, all the config values that need to be set on your box. It allows you to define *everything*.

Tomas talks about how to combine Puppet and Docker together and never again have a need to log on to your Prod box to make a config change but instead make a change in Puppet script and re-deploy your Prod VM, in his words the VMs should be immutable.

And of course if you add Vagrant in this mix, you can also have you Dev VM to be built from the same Puppet scripts as your Prod making you Dev highly consistent with Prod.

Advertisements
Posted in Vagrant | Tagged , , | Leave a comment

How to swap Option (Alt) and Command keys on Mac

If you are (like I) an emacs user on MacOS X you might have this problem : how to re-assign Meta key from Option (Alt) to Cmd key in Terminal (if you don’t know what the heck I am talking about – congratulations! you are free of emacs virus).

For some mysterious reason this seems to be a huge problem, I tried few things (e.g. remapping with keyremap4macbook and cmd-key-happy) but the only reliable solution which worked so far is this:

1. Install iTerm2
2. In the “Keys” tab in configuration change “Left Command” to map to the “Right Option” key,

Screen Shot 2015-12-11 at 20.43.31

and then under the “Profiles” tab changed “Right Option” key to act as Esc. And then you will have Cmd as Meta

Screen Shot 2015-12-11 at 20.47.58

I found this suggestion here

Posted in Uncategorized | Tagged | Leave a comment

How to configure RabbitMQ to run as Docker container in Vagrant

Vagrant is one of those technologies that hit you like a truck and you don’t understand anymore how you lived without it before.

If you haven’t heard about Vagrant yet – it is a simple mechanism to automatically build a complete development environment which is sandboxed in a VM. By default it uses VirtualBox but could be used with other providers, including VMware, AWS, Google Compute Engine, etc. It seamlessly integrates with things like Chef, Puppet and Docker.

From practical perspective it works like this: you add to your git repo a Vagrantfile which is a configuration of VM image and software that you want to have installed for your project. When you clone this repo to your local machine you need to run a single command: “vagrant up” and Vagrant will setup a VM and install all the dependencies. Depending on the complexity of configuration this might take few mins but when Vagrant is done, you will have completely isolated dev environment which you can start, halt, destroy, etc. The main idea is that you have exactly the same development setup as you use in production so entire class of bugs (“… but it worked on my laptop!”) is eliminated and Vagrant takes away all the pain usually associated with repeatable env setup.

Vagrant is platform independent so you can continue using your favorite OS (MacOs, Linux or Windows) to run Ubuntu, CentOS or any other VM image, you of course can continue using your text editor and Vagrant will take care of file synch to VM.

There are plenty of resources on Vagrant but you probably should start with talk by Vagrant developer Mitchell Hashimoto and with Vagrant book.

For one of my recent projects I wanted to use RabbitMQ Docker container and here are the steps how Vagrant, RabbitMQ and Docker could be used together.

First you need RabbitMQ Docker image, you can build one yourself but there are quite a few of them already available at Docker index, I picked up the most recent one built by
Mikael Gueck

2. Now you need to install VirtualBox and Vagrant at your machine.
Install virtualbox from https://www.virtualbox.org/ (or use your package manager)
Install vagrant from http://www.vagrantup.com/ (or use your package manager)

3. Next step you need Vagrantfile which describes VM configuration:


# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  # All Vagrant configuration is done here. The most common configuration
  # options are documented and commented below. For a complete reference,
  # please see the online documentation at vagrantup.com.

  # Every Vagrant virtual environment requires a box to build off of.
  config.vm.box = "precise64"

  # The url from where the 'config.vm.box' box will be fetched if it
  # doesn't already exist on the user's system.
  config.vm.box_url = "http://files.vagrantup.com/precise64.box"

  # Create a forwarded port mapping which allows access to a specific port
  # within the machine from a port on the host machine. In the example below,
  # accessing "localhost:8080" will access port 80 on the guest machine.
  #config.vm.network :forwarded_port, guest: 80, host: 8080
  config.vm.network :forwarded_port, guest: 15672, host: 15672, auto_correct: true 
  config.vm.network :forwarded_port, guest: 4369, host: 4369,   auto_correct: true
  config.vm.network :forwarded_port, guest: 5672, host: 5672,   auto_correct: true

  # Create a private network, which allows host-only access to the machine
  # using a specific IP.
  config.vm.network :private_network, ip: "192.168.33.10"

  # Load and start rabbitmq image
  config.vm.provision "docker" do |d|
    d.pull_images "mikaelhg/docker-rabbitmq"
    d.run "mikaelhg/docker-rabbitmq",
      args: "-h rabbithost -p :5672 -p :15672"
  end
end

this config will build Ubuntu based VM, will install RabbitMQ Docker container and start it with ports 5672 and 15672. The docker image will also have RabbitMQ management console plugin installed. The image will add NAT port mappings and also expose IP “192.168.33.10” for this VM.

4. Now you can run : “vagrant up” command and Vagrant will build and start the VM.

with “vagrant ssh” you can log into the VM console and with “docker ps -a” you can inspect running Docker containers.

here is the screenshot with the steps:
Screen Shot 2014-01-12 at 11.20.15

5. the final step: run http://192.168.33.10:32768/ from your browser (RabbitMQ login / password is “guest” / “guest”)

if everything has started correctly you should get RabbitMQ Management Console in your browser.

In conclusion: both Vagrant and Docker are absolutely amazing technologies and you should be aware of their usage.

Here is the gitbub repo with already preset project.

run:
git clone https://github.com/RomanShestakov/docker-rabbitmq.git
cd docker-rabbitmq
vagrant up
go to http://192.168.33.10:32768/

Hopefully this post would be helpful to somebody.

Posted in Uncategorized | Tagged , , | 3 Comments

Resource Discovery in Erlang

It is interesting and surprising how some of the fundamental things are missing from Erlang OTP lib. Of course it is impossible to include everything into OTP but some of the things are almost obvious. On my opinion Ulf Wiger’s Gproc lib is one of such examples, I use it all the time to give a process a name and refer to it by a name rather by PId later on. It is easy to do with Gproc (without having to make a process into registered one) and project code quality is superb (not surprising given who the author is). Hopefully GProc will make it into OTP one day.

But another example is resource discovery problem in Erlang cluster. It is not easy, if you have a system made up from multiple components you kind of need to hardcode the names of the nodes that provide a service of a given type. And what if you have new nodes entering and exiting your cluster all the time? I don’t think OTP addresses this problem easily. Fortunately, Martyn Logan showed the possible solution in “Erlang and OTP in Action” book. In chapter 8 there is an example of a simple resource discovery protocol. Apart from the fact that “Erlang and OTP in Action” is a great book in its own right, it is worth buying just for this chapter alone.

Martyn implemented the initial idea and I and other people made few additions to it and now it is available on github resource_discovery .

I think it is as useful as Ulf’s GProc. The idea is simple, you have nodes in your cluster which provide services (e.g. logger or webserver or task worker, etc) and there are nodes which need to consume such services (e.g. a send a log message to a one of 10 different loggers). This is where you need automatic resource discovery mechanism, so instead of picking a name of the node from some config file you can ask a question : give me a resource of type ‘logger’ or whatever. And the system will reply with a list of all resources of this type. The resource could be a name of the node or PID of the process, it doesn’t matter. The important thing it is all dynamic, so if you need to add extra task workers to your cluster, you do it and then the resource discovery protocol will know that you have new nodes which provide ‘worker’ service. And the same thing happens when services are dropping off – one of 10 loggers could disappear from the cluster and it will be purged from the resource discovery automatically.

If you decide to use it you need to add “Resource Discovery” as a dependency to your project:

{deps, [
	{'resource_discovery', ".*",{git, "git@github.com/erlware/resource_discovery.git", "master"}}
       ]}.

you need to start ‘resource_discovery’ application, I usually do it as part of my start/0 function in _app.erl module:

-module(example_app).
-behaviour(application).
-define(APPS, [lager, resource_discovery, example]).

%% Application callbacks
-export([start/0, start/2, stop/1]).

%% ===================================================================
%% Application callbacks
%% ===================================================================

start() ->
    [begin application:start(A), io:format("~p~n", [A]) end || A 
<- ?APPS].

start(_StartType, _StartArgs) ->
    lager:info("starting example on a node ~p", [node()]),
    example_sup:start_link().

stop(_State) ->

then in the init/0 function of the process which provides the service, you announce that you have a service of the given type by adding it to resource discovery:

resource_discovery:add_local_resource_tuple({worker, self()}),

You can also register your interest to the service of another type that some other resource in the cluster provides and trigger resource synchronization:

 resource_discovery:add_target_resource_types([?LOGGER]),
 resource_discovery:trade_resources(),

here is possible example for init function:

init([]) ->
    process_flag(trap_exit, true),
    lager:info("starting task server on: ~p", [node()]),
    %% announce via resource_discovery that we have available resource
    resource_discovery:add_local_resource_tuple({worker, self()}),
    %% add request for logger
    resource_discovery:add_target_resource_types([?LOGGER]),
    %% synch resources
    resource_discovery:trade_resources(),
    {ok, #state{}}.

Now, if you need to find a PID of resource ‘worker’ from another nodes and use it (e.g. by sending a message with a task to it), you can ask ask how many such resources exist or get all resources or get a single resource:

 NofResource = resource_discovery:get_num_resource('worker'),
 AllWorkers = resource_discovery:get_resources('worker'),
 SingleWorker  = resource_discovery:get_resource('worker')

and when your worker leaves the cluster, you can cleanup the resource from the global resource_registry:

terminate(_Reason, _State) ->
    %% make resource 'worker' unavailable for other clients
    resource_discovery:delete_local_resource_tuples(resource_discovery:get_local_resource_tuples()),
    resource_discovery:trade_resources(),
    lager:info("worker is shutting down on node ~p", [node()]),
  ok.

I find resource_discovery app hugely useful and hope that somebody else feels the same.

Posted in Erlang, Resource Discovery | Tagged , | 9 Comments

How to make closable tabs with Nitrogen and jQuery UI 1.9

A minor annoyance with using jQuery UI lib is that some of the controls are missing obvious features. I don’t want to call the lib “half-baked” – jQuery UI is of good quality but sometimes you can’t help but to be irritated by it.

One of the obvious misses is the absence of ability to add “Close” button to a tab in Tabs control. All the tabs in all the browsers always have an icon in top right corner to close the tab but jQuery UI designers for some mysterious reasons decided to leave it out.

Fortunately, it is possible to extend jQuery controls and add your own functionality. In this post I will give the example of how to add Close button to Tabs Element in Nitrogen_Element lib and have both Closable and Unclosable tabs:

Screen Shot 2013-04-13 at 19.56.38

Andrew Watts wrote a great post a while back showing how to make closable tabs for jQuery 1.8 but this method doesn’t work anymore for jQuery 1.9.

First thing you need to be aware of is that jQuery UI 1.9 has been completely re-designed in order to pave the way for the next major release jQuery 1.10. This means that prototype based extension method which worked for jQuery UI 1.8 is no longer working.

Instead, you need to use the widget factory method ($.widget()). Please refer to official Widget documentation for more details but in short, this is the method used for creation of brand new widget as well as for extension of the existing ones. Here is the useful Gist which compares the extension differences between jQuery 1.8 and 1.9.

To extend Tabs all you need to do is to call widget factory and pass the name of your widget and the base class to inherit from.
Defining a widget with the same name as the one you inherit from allows to extend the widget in place. The third parameter is the prototype object which implements the override methods of the base class implementation or adds new methods:

$.widget( "ui.tabs&amp", $.ui.tabs, { ... } );  

I re-used some of the Andrew’s code and re-wrote it to be compatible with jQuery 1.9 and added some of my own features. Here is the full ui.tabs.closable.js module and I will explain what it does and how to use it.


(function() {
    $.widget( "ui.tabs", $.ui.tabs, {
	options: {
	    spinner: "<em>Loading…</em>",
	    closable: false
	},
	_removeTab: function( index ) {
	    index = this._getIndex( index );
	    tab = this.tabs.eq( index ).remove();
	    // permanently remove the tab
	    this.tabs.splice( index, 1 );
	    // remove a panel
	    panel = this._getPanelForTab( tab ).remove();
	    // select a tab
	    if( tab.hasClass( "ui-tabs-active" ) && this.tabs.length > 2 ) {
	    	this._activate( index + ( index + 1 < this.tabs.length ? 1 : -1 ));
	    };
	    this._refresh();
	    return this;
	},

	_processTabs: function() {
	    this._super( "_processTabs" );
	    var self = this;
	    var lis = this.tablist.children( ":has(a[href])" );
	    // remove annoying link outline at tabs title
	    lis.children("a").css('outline', 'none');

	    if (this.options.closable === true) {
	    	var unclosable_lis = lis.filter(function() {
		    // return tabs which don't have '.ui-closable-tab' yet and also not marked with '.ui-unclosable-tab'
                    return ($('.ui-closable-tab', this).length === 0 && $('.ui-unclosable-tab', this).length === 0);
	    	});

		// append the close button and associated events
		unclosable_lis.each(function() {
                    $(this)
			.append('<a href="#"><span class="ui-icon ui-icon-circle-close ui-closable-tab"></span></a>')
			.css('outline', 'none')
			.find('a:last .ui-closable-tab')
                        .hover(
                            function() {
				$(this).addClass('ui-icon-circle-triangle-e');
                                $(this).css('cursor', 'pointer');
                            },
                            function() {
				$(this).removeClass('ui-icon-circle-triangle-e');
				$(this).css('cursor', 'default');
                            }
                        )
                        .click(function(e) {
                            // don't follow the link
			    e.preventDefault();
			    // get fresh state of the tabs list
			    var lis = self.tablist.children( ":has(a[href])" );
			    // get index of the tabs
                            var index = lis.index($(e.delegateTarget).parent().parent());
                            if (index > -1) {
                                // remove this tab
                                self._removeTab(index);
                            }
                        })
			.end();
		});
	    }
	}
    });
})(jQuery);


First you need to add a reference to ui.tabs.closable.js on your page, depending on where ui.tabs.closable.js lives in your directory structure:

<script src='/plugins/ui.tabs.closable/js/ui.tabs.closable.js' type='text/javascript' charset='utf-8'></script>

Then you need to add a html markup for Tabs to your page and init tabs control. Nitrogen_Elements Tabs control will create a html markup and instantiate jQuery Tabs for you if you add the following record to you Nitrogen page:

#tabs{
   id = tabs,
   options = [{selected, 0}, {closable, true}],
   tabs = [
	#tab{title = "Tab 1", url = "/content/tabs2.htm", closable=false},
	#tab{title = "Tab 2", body = ["Tab two body..."]}
    ]
}

Notice that you need to add option {closable, true} to tell the extension method to add Close link to your tabs. If you want some of your tabs *not to have* close button you need to add option “closable=false” for the Nitrogen record for a given tab. Under the covers Nitrogen Tabs control will add “.ui-unclosable-tab” class to the tab anchor and this will tell tabs extension method to skill the tab.

Extension module overrides _processTabs() method, first we call _processTabs() from base class, then select a list of tabs anchors – this is the list of tabs titles and remove outlines for anchors which some browsers render by default.

this._super( "_processTabs" );
var self = this;
var lis = this.tablist.children( ":has(a[href])" );
// remove annoying link outline at tabs title
lis.children("a").css('outline', 'none');

then, if Option closable=true was passed to the control, we get a list of tabs which don’t yet have class ‘.ui-closable-tab’ and also haven’t been explicitly excluded from having Close button with class ‘.ui-unclosable-tab’.

if (this.options.closable === true) {
     var unclosable_lis = lis.filter(function() {
	// return tabs which don't have '.ui-closable-tab' yet and also not marked with '.ui-unclosable-tab'
        return ($('.ui-closable-tab', this).length === 0 && $('.ui-unclosable-tab', this).length === 0);
});
...
}

After that for each element of this list we append a new anchor and set an iron for a span element and also mark this span with ‘ui-closable-tab’ class:

.append('<a href="#"><span class="ui-icon ui-icon-circle-close ui-closable-tab"></span></a>')

We also remove outline for this anchor and add action for Hover event : we want the icon to change when we pass a mouse over the Close icon.

.css('outline', 'none')
.find('a:last .ui-closable-tab')
.hover(
  function() {
	$(this).addClass('ui-icon-circle-triangle-e');
        $(this).css('cursor', 'pointer');
       },
  function() {
	$(this).removeClass('ui-icon-circle-triangle-e');
	$(this).css('cursor', 'default');
     }
)

We also attach to Click event on the span to detect the index of the tab whose Close button was clicked and remove the tab:

.click(function(e) {
   // don't follow the link
  e.preventDefault();
  // get fresh state of the tabs list
  var lis = self.tablist.children( ":has(a[href])" );
  // get index of the tabs
  var index = lis.index($(e.delegateTarget).parent().parent());
  if (index > -1) {
      // remove this tab
      self._removeTab(index);
  }
})

When we click Close icon – this will call private _removeTab(index) and in this method we remove the title link and tab panel and also activate the closest remaining tab:

	_removeTab: function( index ) {
	    index = this._getIndex( index );
	    tab = this.tabs.eq( index ).remove();
	    // permanently remove the tab
	    this.tabs.splice( index, 1 );
	    // remove a panel
	    panel = this._getPanelForTab( tab ).remove();
	    // select a tab
	    if( tab.hasClass( "ui-tabs-active" ) && this.tabs.length > 2 ) {
	    	this._activate( index + ( index + 1 < this.tabs.length ? 1 : -1 ));
	    };
	    this._refresh();
	    return this;
	}

This is all the code you need to implement closable Tabs control if you are using jQuery 1.9 or higher.

The full Nitrogen example is here.

But if you just need to use closable Tabs in your pure javascript code here is the separate project just for ui.tabs.closable.js module.

Posted in custom Nitrogen elements, Erlang, Nitrogen, Nitrogen_Elements, Uncategorized | Tagged , , , , , | Leave a comment

How to build a live datagrid with Nitrogen

One of the most useful element in Nitrogen Elements project is jqGrid control.
jqGrid is a Nitrogen wrapper for a popular javasript library which allows to represent and manipulat tabular data on the web. There are lots of examples for this control which are available here.

The nice thing about most of the data grid controls (jqGrid is one of many available) is that they usually works with XML or JSON data sources. So, you can initialize the datagrid on the client side and populate it with the data which are retrieved from the URL which returns JSON or XML string.

Here is the example to add #jqgrid{} to your page:

	#jqgrid{
	    id = jqgrid,
	    options=[
		{url, 'get_jqgrid_data'},
		{datatype, <<"json">>},
		{colNames, ['ID', 'Name', 'Values']},
		{colModel, [
		    [{name, 'id'}, {index, 'id'}, {width, 55}],
		    [{name, 'name'}, {index, 'name1'}, {width, 80}],
		    [{name, 'values1'}, {index, 'values1'}, {width, 100}]
		]},
		{rowNum, 10},
		{rowList, [10, 20, 30]},
		{sortname, 'id'},
		{viewrecords, true},
		{sortorder, <<"desc">>},
		{caption, <<"JSON Example">>},
		{groupColumnShow, false},
		{loadonce, false},
		{scrollOffset, 0}, %% switch off scrollbar
		{autowidth, true} %% fill parent container on load
	    ]

You will need to specify the data type and url from which the control gets the data:

{url, 'get_jqgrid_data'},
{datatype, <<"json">>},

You also need to specify the data model: column names for the table view:

	{colNames, ['ID', 'Name', 'Values']},
		{colModel, [
		    [{name, 'id'}, {index, 'id'}, {width, 55}],
		    [{name, 'name'}, {index, 'name1'}, {width, 80}],
		    [{name, 'values1'}, {index, 'values1'}, {width, 100}]
		]},

The datasource URL point to the REST handler which returns JSON string:

-module(get_jqgrid_data).

-export([init/3, content_types_provided/2, to_json/2]).

init(_Transport, _Req, []) ->
    {upgrade, protocol, cowboy_rest}.

content_types_provided(Req, State) ->
    {[
      {<<"application/json">>, to_json}
     ], Req, State}.

to_json(Req, State) ->
    Data = {struct, [{<<"total">>, 1},
		     {<<"page">>, 1},
		     {<<"records">>, 2},
		     {<<"rows">>, [{struct, [{<<"id">>, 1}, {<<"cell">>, [<<"1">>, <<"cell11">>, <<"values11">>]}]},
				   {struct, [{<<"id">>, 2}, {<<"cell">>, [<<"2">>, <<"cell15">>, <<"values22">>]}]}
				  ]}
		    ]},
    Data1 = iolist_to_binary(mochijson2:encode(Data)),
    {Data1, Req, State}.

Here is how the #jqgrid{} defined with the above code will look at your page:
Screen Shot 2013-03-24 at 22.26.15

With a bit of extra code you can turn this datagrid live and make it auto-update from the continuous data stream by hooking it up via Websocket to the module which will push JSON updates from the server side.

It works very similar to the viz.js example.

We add a button and on_click event open a Websocket connection to “ws://localhost:8000/jqdata”, Websocket connection is served by Jqdata.erl module which will generate and push json updates to the browser:

-module(jqdata).
-behaviour(cowboy_websocket_handler).

-export([init/3]).
-export([websocket_init/3]).
-export([websocket_handle/3]).
-export([websocket_info/3]).
-export([websocket_terminate/3]).

init({tcp, http}, _Req, _Opts) ->
    {upgrade, protocol, cowboy_websocket}.

websocket_init(_TransportName, Req, _Opts) ->
    erlang:start_timer(1000, self(), get_data()),
    {ok, Req, undefined_state}.

websocket_handle({text, Msg}, Req, State) ->
    {reply, {text, << "", Msg/binary >>}, Req, State};
websocket_handle(_Data, Req, State) ->
    {ok, Req, State}.

websocket_info({timeout, _Ref, Msg}, Req, State) ->
    erlang:start_timer(1000, self(), get_data()),
    {reply, {text, Msg}, Req, State};
websocket_info(_Info, Req, State) ->
    {ok, Req, State}.

websocket_terminate(_Reason, _Req, _State) ->
    ok.

get_data() ->
    Data = {struct, [{<<"total">>, 1},
		     {<<"page">>, 1},
		     {<<"records">>, 4},
		     {<<"rows">>, [{struct, [{<<"id">>, 1}, {<<"cell">>, [<<"1">>, <<"cell11">>, random:uniform(100)]}]},
				   {struct, [{<<"id">>, 2}, {<<"cell">>, [<<"2">>, <<"cell15">>, random:uniform(50)]}]},
				   {struct, [{<<"id">>, 3}, {<<"cell">>, [<<"3">>, <<"cell55">>, random:uniform(20)]}]},
				   {struct, [{<<"id">>, 4}, {<<"cell">>, [<<"4">>, <<"cell66">>, random:uniform(10)]}]}
				  ]}
		    ]},
    iolist_to_binary(mochijson2:encode(Data)).

We also add a callback to the “on_message” event, with it we set jqgrid datatype to “jsonstring” and datastr to the json string sent from the server. We then trigger jqgrid update event:

event({Tag, connect}) ->
    Server = wf:q(server_txt),
    %?PRINT({gqgrid_server, Server}),
    wf:wire(#ws_open{server = Server, func = "function(event){console.log('open')};"}),
    wf:wire(#ws_message{func = wf:f("function(event){$(obj('~s')).jqGrid('setGridParam',
	                 {datatype:\"jsonstring\", datastr:event.data}).trigger(\"reloadGrid\")};", [jqgrid])}),
    wf:wire(#ws_error{func = "function(event){console.log('error')};"}),
    wf:replace(conn, #button{id = conn, text = "Disconnect", actions = [#event{type = click, postback = {Tag, disconnect}}]});

With all of this we will have the following ticking datagrid, if you run the example you will see the changing values in “Values” column:

Screen Shot 2013-03-24 at 23.25.53

The full example is here.

Posted in Cowboy, custom Nitrogen elements, Erlang, Nitrogen, Nitrogen_Elements, Uncategorized | Tagged , , , , | Leave a comment

Extended example with graphviz updates via Websockets

Native Websockets support is a very compelling reason to use Cowboy. (Yaws also supports Websockets natively as well so it is another alternative).

I will give a simple example how to use Websockets with Nitrogen and Cowboy by extending on my blog post which showed how to render static graphviz graphs.

This time we will use Websockets to push the graph updates to the browser from the server back-end.

First, you need to tell Cowboy what handler will be processing your Websocket requests. You can do this with adding a route to the dispatch table:

dispatch_rules() ->
    cowboy_router:compile(
	%% {Host, list({Path, Handler, Opts})}
	[{'_', [
	    {["/favicon.ico"], cowboy_static, [{directory, {priv_dir, ?APP, [<<"static">>]}}, {file, "favicon.ico"}]},
	    {["/content/[...]"], cowboy_static, [{directory, {priv_dir, ?APP, [<<"content">>]}},
		{mimetypes, {fun mimetypes:path_to_mimes/2, default}}]},
	    {["/static/[...]"], cowboy_static, [{directory, {priv_dir, ?APP, [<<"static">>]}},
		{mimetypes, {fun mimetypes:path_to_mimes/2, default}}]},
	    {["/plugins/[...]"], cowboy_static, [{directory, {priv_dir, ?APP, [<<"plugins">>]}},
		{mimetypes, {fun mimetypes:path_to_mimes/2, default}}]},
	    {["/doc/[...]"], cowboy_static, [{directory, {priv_dir, ?APP, [<<"doc">>]}},
		{mimetypes, {fun mimetypes:path_to_mimes/2, default}}]},
	    {["/get_jqgrid_data/[...]"], get_jqgrid_data, []},
	    {["/websocket"], ws_handler, []},
	    {["/jqdata"], jqdata, []},
	    {'_', nitrogen_cowboy, []}
    ]}]).

In this case we are saying that requests to “websocket”, e.g. like “ws://localhost:8000/websocket”, will be handled by erlang module ws_handler.erl. Here is this module:

-module(ws_handler).
-behaviour(cowboy_websocket_handler).

-export([init/3]).
-export([websocket_init/3]).
-export([websocket_handle/3]).
-export([websocket_info/3]).
-export([websocket_terminate/3]).

-define(DATA, <<"digraph G {subgraph cluster_0 {style=filled;color=lightgrey; node [style=filled,color=yellow]; a0 -> a1 -> a2 -> a3;label = \"process #1\";} subgraph cluster_1 {node [style=filled]; b0 -> b1 -> b2 -> b3; label = \"process #2\";color=red} start -> a0; start -> b0; a1 -> b3; b2 -> a3; a3 -> a0; a3 -> end; b3 -> end; start [shape=Mdiamond]; end [shape=Msquare];}">>).

-define(DATA1, <<"digraph G {subgraph cluster_0 {style=filled;color=lightgrey; node [style=filled,color=blue]; a0 -> a1 -> a2 -> a3 -> a4;label = \"process #1\";} subgraph cluster_1 {node [style=filled]; b0 -> b1 -> b2 -> b3 -> b4; label = \"process #2\";color=red} start -> a0; start -> b0; a1 -> b3; b2 -> a3; a3 -> a0; a3 -> a4; a4 -> end; b3 -> end; start [shape=Mdiamond]; end [shape=Msquare];}">>).

-define(DATA2, <<"digraph G {subgraph cluster_0 {style=filled;color=lightgrey; node [style=filled,color=red]; a0 -> a1 -> a2 -> a3 -> a4;label = \"process #1\";} subgraph cluster_1 {node [style=filled]; b0 -> b1 -> b2 -> b3 -> b4; label = \"process #2\";color=red} start -> a0; start -> b0; a1 -> b3; b2 -> a3; a3 -> a0; a3 -> a4; a4 -> end; b3 -> end; start [shape=Mdiamond]; end [shape=Msquare];}">>).

-define(G, [?DATA, ?DATA1, ?DATA2]).

init({tcp, http}, _Req, _Opts) ->
    {upgrade, protocol, cowboy_websocket}.

websocket_init(_TransportName, Req, _Opts) ->
    erlang:start_timer(1000, self(), lists:nth(random:uniform(3), ?G)),
    {ok, Req, undefined_state}.

websocket_handle({text, Msg}, Req, State) ->
    {reply, {text, << "", Msg/binary >>}, Req, State};
websocket_handle(_Data, Req, State) ->
    {ok, Req, State}.

websocket_info({timeout, _Ref, Msg}, Req, State) ->
    erlang:start_timer(1000, self(), lists:nth(random:uniform(3), ?G)),
    {reply, {text, Msg}, Req, State};
websocket_info(_Info, Req, State) ->
    {ok, Req, State}.

websocket_terminate(_Reason, _Req, _State) ->
    ok.

here, for example purposes, we define three dot binary strings that specify the possible states that our graph could be in. Then we randomly select a state and push it to the client side in websocket_info function.

websocket_info/3 – is used to handle the events coming from the server side.
websocket_handle/3 – is used to handle messages coming from the browser side.

On the browser side we need to implement the callbacks to WebSocket API.
In general case, in javascript, you need to create an instance of WebSocket object and pass a connection string to it (e.g. “ws://localhost:8000/websocket”):

 websocket = new WebSocket(wsHost);
 websocket.onopen = function(evt) { onOpen(evt) }; 
 websocket.onclose = function(evt) { onClose(evt) }; 
 websocket.onmessage = function(evt) { onMessage(evt) }; 
 websocket.onerror = function(evt) { onError(evt) }; 

You can find a full pure javascript websocket example in the “Examples” dir which comes with Cowboy repo.

For Nitrogen I added a simple action_ws_api.erl module which implements a simple API for onopen / onclose / onmessage / onerror Websockets calls in Nitrogen style:

% simple websocket api
-module(action_ws_api).
-include("nitrogen_elements.hrl").
-compile(export_all).

render_action(#ws_open{server = Server, func = OnOpen}) ->
    [
     wf:f("$(function() { var websocket;
           websocket = new WebSocket('~s');window.websocket = websocket;", [Server]),
           on_open_script(OnOpen),
           ";})"
    ];
render_action(#ws_message{func = OnMessage}) -> on_message_script(OnMessage);
render_action(#ws_error{func = OnError}) -> on_error_script(OnError);
render_action(#ws_close{func = OnClose}) -> on_close_script(OnClose).

on_open_script("") ->
    "websocket.onopen = function(event){console.log('close');};";
on_open_script(OnOpen) ->
    wf:f("websocket.onopen = ~s", [OnOpen]).

on_close_script("") ->
    wf:f("$(function(){window.websocket.close();console.log('close');});");
on_close_script(OnClose) ->
    wf:f("websocket.onclose = ~s", [OnClose]).

on_message_script("") ->
    "window.websocket.onmessage = function(event){console.log(event.data)};";
on_message_script(OnMessage) ->
    wf:f("websocket.onmessage = ~s", [OnMessage]).

on_error_script("") ->
    "websocket.onerror = function(event){console.log(event.data)};";
on_error_script(OnError) ->
    wf:f("websocket.onerror = ~s", [OnError]).

The final part is adding all this to the erlang module which implements Nitrogen page:

-module(viz).

-include_lib("nitrogen_elements/include/nitrogen_elements.hrl").
-compile(export_all).

-define(DATA, <<"digraph G {subgraph cluster_0 {style=filled;color=lightgrey; node [style=filled,color=white]; a0 -> a1 -> a2 -> a3;label = \"process #1\";} subgraph cluster_1 {node [style=filled]; b0 -> b1 -> b2 -> b3; label = \"process #2\";color=blue} start -> a0; start -> b0; a1 -> b3; b2 -> a3; a3 -> a0; a3 -> end; b3 -> end; start [shape=Mdiamond]; end [shape=Msquare];}">>).

body(_Tag) ->
    %% output html markup
    [
	#viz{id = viz, data = ?DATA}
    ].

control_panel(Tag) ->
    #panel{id = control_panel, body = [
	#textbox{id = server_txt, style = "width: 100%", text = "ws://localhost:8000/websocket"},
	#p{},
	%% postback is to the index page, event from index.erl will call event(connect)
	#button{id = conn, text = "Connect", actions = [#event{type = click, postback = {Tag, connect}}]}
    ]}.

event({Tag, connect}) ->
    Server = wf:q(server_txt),
    ?PRINT({viz_server, Server}),
    wf:wire(#ws_open{server = Server, func = "function(event){console.log('open')};"}),
    wf:wire(#ws_message{func = wf:f("function(event){var g = jQuery(obj('~s'));
                                               g.html(Viz(event.data, \"svg\"));
	                                       g.find(\"svg\").width('100%');
	                                       g.find(\"svg\").graphviz({status: true});};", [viz])}),
    wf:wire(#ws_error{func = "function(event){console.log('error')};"}),
    wf:replace(conn, #button{id = conn, text = "Disconnect", actions = [#event{type = click, postback = {Tag, disconnect}}]});
event({Tag, disconnect}) ->
    wf:wire(#ws_close{}),
    wf:replace(conn, #button{id = conn, text = "Connect", actions = [#event{type = click, postback = {Tag, connect}}]});
event(Event) ->
    ?PRINT({viz_event, Event}).

Here, we are opening Websocket connection to the server and adding javascript onmessage handler which updates the graph:

  wf:wire(#ws_open{server = Server, func = "function(event){console.log('open')};"}),
    wf:wire(#ws_message{func = wf:f("function(event){var g = jQuery(obj('~s'));
                                               g.html(Viz(event.data, \"svg\"));
	                                       g.find(\"svg\").width('100%');
	                                       g.find(\"svg\").graphviz({status: true});};", [viz])}),

This is it, if you run this example you will see the initial graph state:

Screen Shot 2013-03-17 at 11.48.38

but if you press “Connect” button, you will open Websocket connection to the server and server will start generating new states for the graph and pushing them to the browser. This will trigger onMessage event for which we have a handler. This handler will get a dot string, calculate a new graph layout with Viz.js, generate a new svg from it and update a “svg” element with a new state. It will continue doing this until you press “Disconnect” which will close Websocket connection.

Screen Shot 2013-03-17 at 11.53.26

The full example is here.

Posted in Cowboy, custom Nitrogen elements, Erlang, Nitrogen, Nitrogen_Elements | Tagged , , , , | Leave a comment