If you followed my previous post on setting up the ELK stack on Azure, you can configure AppHarbor to use it fairly easily.
You need to open a port on Azure (I’ve chosen 9000). You can set the inbound IP range to AppHarbors IPs1.
Once you do this, make a new file in /etc/logstash/conf.d called AppHarbor.conf, and enter the following into it’s contents:
input {
tcp {
port => 9000
type => syslog
}
udp {
port => 9000
type => syslog
}
}
filter {
grok {
match => [ "message", "%{GREEDYDATA:syslog_message}" ]
}
mutate { replace => { type => "appharbor" } }
}
output {
elasticsearch {
hosts => "localhost"
}
}
While the AppHarbor logs supposedly conform to “syslog” format, I did not find this to be true – as of this time, I haven’t established the best log format yet, but this is easier to control than using “syslog” as an input type.
If you don’t do this, you will get a lot of messages tagged with “grokparsefailure_sysloginput”.
Adding this to AppHarbor is super-easy, start by selecting a logging configuration:
Select the option to add a new log drain:
Then, add the URL. This url needs to be in the format of “syslog://test.domain.com:9000”. If you use HTTP instead of “syslog”, you will get errors that say “(output buffer overflow)”, and lose all the useful data2
Once you finish, this is what you’ll see:
And you should be all set.
If you want to get the contents of these log messages into specific fields, I recommend the grok debugger.