On Centralizing Logs at Monitorama EU

I’m really excited to be attending Monitorama EU this week! I’ll give a talk about centralizing logs on Friday at 15:15 . You can see the full schedule here.

Please tweet about On Centralizing Logs at Monitorama EU.

The talk is mainly about centralizing logs and storing them in Elasticsearch. It will begin with tips about using Elasticsearch for logs in production, so it runs fast and stable. There will be an introduction to Kibana 3 and then we’ll move to indexing.

We’ll start the indexing part by clarifying the term syslog: is it about a daemon? is it about a log message format? is it about a protocol for transferring logs? Spoiler alert: it can be any of the three, and there are options at every level. And one of them is to write syslog messages to Elasticsearch.

Then, we’ll move to Logstash and describe a few typical deployments. We’ll end by introducing alternatives to the setup described above: from our Logsene to the Flume + Solr combination.

If there’s enough time, I have some extra slides with tips about configuring rsyslog for processing lots of messages. Think 100K+ or even 1M+ messages per second, depending on the hardware and configuration.

Below is a sketchnote of the whole talk, which will be printed and given to participants. Click on the image get the full resolution.

For the occasion, Sematext is giving a 20% discount for all SPM applications. The discount code is MONEU2013

Also, Manning is giving a 44% discount for Elasticsearch in Action and all the other books from their website. The discount code is mlmoneu13cf