Using Hubot for Home Server Monitoring

I set up Hubot to do some simple server administration – this allows me to do a few common operations tasks while I’m travelling. I’ve previously tried SSH on my phone, but that seems unsafe in general, and it’s hard to use – this fixes a big frustration for me, if this site goes down and I’m not able to do anything.

Installation
Hubot responds to chat messages and calls into custom Javascript classes, so you first need to install Node / npm:

sudo apt-get update
sudo apt-get install nodejs npm

Then follow the hubot install instructions:

npm install -g yo generator-hubot
useradd hubot
sudo su - hubot

Chat

Since I want this for travelling, I need a chat integration that works on my phone, that is also relatively secure.

The Hubot documentation recommends several options.  One caution from what I learned doing this,”Let’s chat” doesn’t seem to allow you to lock down who can create accounts, and SSL between my Linode Server and Heroku is completely broken (it seems to be a defect in Socket.io). For the time being I landed on Campfire, which works on a phone (barely). It looks like a lot of people use Hipchat, but for now I’m trying to run this on the cheap.

Plugins

I like the reminder plugin – this has a semi-natural language interface to remind yourself of things later. To add this, add “hubot-remind-her”: “^0.4.1” to package.json, and do npm.install. Even though it matches a lot of date formats, it’s still pretty flaky and you have to get used to what it wants.

For simple administration, I added a memory check plugin that I wrote:

module.exports = (robot) ->
 robot.respond /memory( (.+))?/i, (msg) ->
    q = msg.match[1]
    @exec = require('child_process').exec
    command = "top -b -n 1 "
    if (!!q && q.length > 0)
      command = command + ' | grep "' + q.match(/[^_\W]+/g).join(' ') + '"

    msg.send "Memory usage:"

    @exec command, (error, stdout, stderr) ->
      msg.send error
      msg.send stdout
      msg.send stderr

As well as a disk space checker:

module.exports = (robot) ->
 robot.respond /disk (usage|use|space)/i, (msg) ->
    q = msg.match[1]
    @exec = require('child_process').exec
    command = "df -h "

    msg.send "Disk free space:"

    @exec command, (error, stdout, stderr) ->
      msg.send error
      msg.send stdout
      msg.send stderr

To be able to effect change, I made it so I can restart mysql:

module.exports = (robot) ->
 robot.respond /restart mysql/i, (msg) ->
    hostname = msg.match[1]
    @exec = require('child_process').exec
    command = "sudo /etc/init.d/mysql restart"

    msg.send "Restarting mysql service on linode"

    @exec command, (error, stdout, stderr) ->
      msg.send error
      msg.send stdout
      msg.send stderr

You can do some entertaining things to make Hubot feel more like interacting with a person:

 robot.respond /thanks(.*)/i, (msg) ->
    msg.send "you're welcome!"

 robot.respond /sorry(.*)/i, (msg) ->
    msg.send "no problem!"

 robot.respond /(status|how are you)(.*)/i, (msg) ->
    msg.send "outstanding!"

To show how this looks, this is a conversation I had with hubot:

 2973 mysql     20   0  306m 123m 9.9m 

mysql stop/waiting
mysql start/running, process 14261

Security

The code for the above commands shows some of the risks of this – you don’t want Hubot to be able to do everything, even though it should only be getting commands through the chat, because you’re constantly a step away from giving someone full control of the box that runs hubot, and anything it can access.

To make this safer, I escaped the grep input in the memory command, and require hubot to sudo to restart mysql. This specific command is granted to the hubot user in sudoers, so at least it limits it’s ability to cause damage:

hubot        ALL = NOPASSWD: /etc/init.d/mysql restart

Generally I would audit any code you install for Hubot, because the script-writing community seems more interested in meme-generation than secure code. Even though the default meme/image generation plugins are cute, I would remove them.

Daemonizing Hubot

Follow these instructions:
https://gist.github.com/1394520/ed2c32663ae61038d85b1cae1e2f10c105f07576

All of the chat settings should go in a configuration file – the trick to this is getting the paths all correct.

Once you get it to work, make it start on boot:

update-rc.d Hubot defaults

Future Ideas

In the future, I’m planning to extend this setup so that each of my machines (laptop, desktop) has a Hubot as well. This will for instance allow me to search for vacation photos or specific files while travelling, or to see if the power is on at home / in the office.

I’m also looking to add some database commands (tell me the status of a user, extend an account, etc) to see how the experience works. I suspect that if you use a Google docs integration for Hubot, you could get access to a lot of applications that have Zapier APIs, but this will require some investigation.

I also need to find a more suitable chat client (hopefully installable on a server), but security is challenging.