Munin

I'll just focus on Munin for this post.

It is quite simple to explain what Munin does. It fetches regularly some stats from your registered hosts, collect them into a simple time series db and plot them in a minimal website you can visit with your browser.

What's really nice of Munin is that is very simple to spin it up and keep it running. You just need to install it using probably the default package of your distribution, enlist the hosts in a config file and configure nginx to serve the static site that Munin regularly updates.

Then you setup the munin-node service to the hosts you want to monitor, enable the plugins for the services you want to monitor and you are done.

Collect your data

Once you are ok with the basic setup, you'll soon probably need to keep track of some stats of your application. For instance a service like the adserver behind Viralize gathers stats about user requests, transactions, average bid, its own analytics timing, etc...

Unfortunately Munin is not that great when it comes to integrate with applications. Ideally you would like to send your events and forget about them. Munin instead wants to pull the data from your app.

Here is where StatsD comes to play. StatsD is a nice service that runs in the background and collects any events you like with a very simple interface. You just need to send a text message via UDP like this:

echo 'app.mycnt:1|c' | nc -u -w0 127.0.0.1 8125

and StatsD will do the rest. (Python has a nice library called python-statsd)

StatsD plugin

Now you probably think that, as Munin seems to be quite popular and StatsD is very popular as well, you can just search for "plug statsd and munin" on Google and get the answer. I did the search and I've just got thousands of flames about "statsd/graphite vs munin", "collectd vs munin" and many other useless results. The answer is this post.

You can achieve the goal in two ways:

1. You can connect to the TCP stats interface of StatsD, pull the data and transform it in Munin plugin format.
2. You can write a StatsD backend that stores the data at each flush interval in a place reachable by your plugin.

Even if the first method looks simpler, the output of StatsD interface is not easily parsable by a shell script without writing much code.

I preferred instead to write a backend that outputs the stats in a JSON file. You can find the plugin here: https://gist.github.com/duilio/20e71c7fba278cce6f2e

Here is a sample config for statsd:

{
  jsonout: {output: "/tmp/statsd.json"},
  port: 8125,
  flushInterval: 5*60*1000, // set to the same interval of Munin (5 mins by default)
  backends: [ "./backends/jsonout" ] // reminds to copy there the jsonout backend
}

Then with jq it is quite easy to write Munin plugins using bash:

#!/bin/bash
# -*- sh -*-

# statsd output json file and namespace of the app stats
STATSDJSON=/tmp/statsd.json
STATSDAPPNS=app

if [ "$1" = "autoconf" ]; then
   echo yes
   exit 0
fi

if [ "$1" = "config" ]; then
    echo 'graph_title Graph Title Here'
    echo 'graph_vlable Graph Label'
    echo 'graph_category Category'

    echo 'mycnt.label mycnt'
    echo 'mycnt.draw LINE1'

    exit 0
fi

function print_counter {
    echo ${1}.value `jq ".counters[\"${STATSDAPPNS}.${1}\"]" $STATSDJSON | sed s/null/0/`
}

print_counter mycnt

Now you are ready to plot all the stats of your apps ;-)