Sending your Windows Event Logs to Logsene using NxLog and Logstash

There are a lot of sources of logs these days. Some may come from mobile devices, some from your Linux servers used to host data, while other can be related to your Docker containers. They are all supported by Logsene. What’s more, you can also ship logs from your Microsoft Windows based hosts and visualize them using Logsene. In this blog post we’ll show how to send your Windows Event Logs to Logsene in a way that will let you build great visualizations and really see what is happening on your Windows-based systems.

How to forward CloudTrail (or other logs from AWS S3) to Logsene

This recipe shows how to send CloudTrail logs (which are .gz logs that AWS puts in a certain S3 bucket) to a Logsene application, but should apply to any kinds of logs that you put into S3. We’ll use AWS Lambda for this, but you don’t have to write the code. We’ve got that covered.

The main steps are:
0. Have some logs in an AWS S3 bucket 🙂
1. Create a new AWS Lambda function
2. Paste the code from this repository and fill in your Logsene Application Token
3. Point the function to your S3 bucket and give it permissions
4. Decide on the maximum memory to allocate for the function and the timeout for its execution
5. Explore your logs in Logsene 🙂

Docker Events and Docker Metrics Monitoring

[ Note: Click here for the Docker Monitoring webinar video recording and slides. And click here for the Docker Logging webinar video recording and slides. ]

——-

Docker deployments can be very dynamic with containers being started and stopped, moved around the YARN or Mesos-managed clusters, having very short life spans (the so-called pets) or long uptimes (aka cattle).  Getting insight into the current and historical state of such clusters goes beyond collecting container performance metrics and sending alert notifications.  If a container dies or gets paused, for example, you may want to know about it, right?  Or maybe you’d want to be able to see that a container went belly up in retrospect when troubleshooting, wouldn’t you?

Just two weeks ago we added Docker Monitoring (docker image is right here for your pulling pleasure) to SPM.  We didn’t stop there — we’ve now expanded SPM’s Docker support by adding Docker Event collection, charting, and correlation.  Every time a container is created or destroyed, started, stopped, or when it dies, spm-agent-docker captures the appropriate event so you can later see what happened where and when, correlate it with metrics, alerts, anomalies — all of which are captured in SPM — or with any other information you have at your disposal.  The functionality and the value this brings should be pretty obvious from the annotated screenshot below.

Like this post?  Please tweet about Docker Events and Docker Metrics Monitoring

Know somebody who’d find this post useful?  Please let them know…

Here’s the list of Docker events SPM Docker monitoring agent currently captures:

• Version Information on Startup:
  • server-info – created by spm-agent framework with node.js and OS version info on startup
  • docker-info – Docker Version, API Version, Kernel Version on startup
• Docker Status Events:
  • Container Lifecycle Events like
    • create, exec_create, destroy, export
  • Container Runtime Events like
    • die, exec_start, kill, oom, pause, restart, start, stop, unpause

Every time a Docker container emits one of these events spm-agent-docker will capture it in real-time, ship it over to SPM, and you’ll be able to see it as shown in the above screenshot.

Oh, and if you’re running CoreOS, you may also want to see how to index CoreOS logs into ELK/Logsene. Why? Because then you can have not only metrics and container events in one place, but also all container and application logs, too!

If you’re using Docker, we hope you find this useful!  Anything else you’d like us to add to SPM (for Docker or any other integration)?  Leave a comment, ping @sematext, or send us email – tell us what you’d like to get for early Christmas!

Beyond POC: Processing Metrics, Logs and Traces … at Scale

For those of you attending next week’s DevOps Summit event in New York City (part of the larger Cloud Computing Expo) with an interest in topics like performance monitoring and processing metrics, log management, and distributed transaction tracing — at scale, no less! — then Sematext founder Otis Gospodnetić will be speaking your language on Wednesday, June 10.

Talk Summary

Application metrics, logs, and business KPIs are a goldmine. It’s easy to get started with the ELK stack (Elasticsearch, Logstash and Kibana) — you can see lots of people coming up with impressive dashboards, in less than a day, with no previous experience. Going from proof-of-concept to production tends to be a bit more difficult, unfortunately, and it tends to gobble up our attention, time, and money. In this talk Otis will share the architecture and decisions behind our services for handling large volumes of performance metrics, traces, logs, anomaly detection, alerts, etc. Attendees will follow data from its sources, its collection, aggregation, storage, and visualization. The talk will also cover the overview of some of the relevant technologies and their strengths and weaknesses, such as HBase, Elasticsearch, and Kafka.

• Date: Wednesday, June 10
• Time: 3:30 pm to 4:30 pm

Panel Discussion: Microservices and IoT Power

Otis will also be participating in a lunchtime panel discussion, also on June 10 (from 12:45 pm to 1:45 pm) with other tech industry experts called “Microservices and IoT Power” that dives deep into the important architectural principles behind implementing IoT solutions for the enterprise. Let’s face it, as remote IoT devices and sensors become increasingly intelligent, they become part of our distributed cloud environment, and we must architect and code accordingly.  It promises to contain buzzwords galore!

Let’s Talk About Elasticsearch, ELK Stack, Solr, Spark, Kafka, APM, Centralized Log Management, and…

We’ll be at Booth #230 in the DevOps Summit section of the floor, so stop by and say hello.  We’ll be demo-ing SPM performance monitoring, Logsene Log Management and Analytics and Site Search Analytics, along with our usual interest in discussing Search and Big Data consulting topics and more.  Or just drop us an email or DM us if you’re not going to be in the Big Apple from June 9-11 but have interest in chatting.

Hope to see you in NYC next week!

Solr Presentations from Lucene/Solr Revolution 2014

Thanks to everyone who stopped by the Sematext booth at last week’s Lucene/Solr Revolution event in Washington, DC and attended our two talks:

• Tuning Solr for Logs by Radu Gheorghe
• Solr Anti-Patterns by Rafal Kuc

The attendance, questions and interest are very much appreciated.  As a company that prides itself on its Solr expertise (and Elasticsearch expertise too, for that matter), it was nice to spend a couple days talking about search and Big Data challenges, performance monitoring and logging with fellow experts from around the world. Here are the slides for the two talks we gave (summaries of the talks can be found here):

Tuning Solr for Logs from Sematext Group, Inc.

 

Solr Anti Patterns from Sematext Group, Inc.

  Videos of the talks will be posted here soon.  Hope to see everyone again next year!

Sematext at Lucene/Solr Revolution 2014

Going to Lucene/Solr Revolution next week — November 11-14 — in Washington, DC?  If so…Sematext will be there exhibiting AND giving two talks!  If you are going, stop by our table to say hello.  We can show you the latest versions of SPM Performance Monitoring, Logsene Log Management and Analytics, Site Search Analytics, and, of course, talk about metrics, centralized log management, Lucene, Solr, Elasticsearch, and just about any other search-related topic you might be interested in.  After all, not only have we blogged, given talks and spread the word in all sorts of ways, we’ve also written books on these subjects!

Both of the Sematext engineer talks take place on Friday, November 14.  They are:

Radu Gheorghe will talk about “Tuning Solr for Logs” at 10:15 am

Summary:  Performance tuning is always nice for keeping your applications snappy and your costs down. This is especially the case for logs, social media and other stream-like data that can easily grow into terabyte territory. While you can always use SolrCloud to scale out of performance issues, this talk is about optimizing. The following questions about Solr settings will be answered. How often should you commit and merge? How can you have one collection per day/month/year/etc? What are the performance trade-offs for these options?  There will also be a discussion around choosing the appropriate hardware.  Radu will talk about optimizing the infrastructure when pushing logs to Solr. This includes tuning Apache Flume to handle large flows of logs and overall design options that also apply to other shippers, like Logstash.

Rafal Kuc will talk about “Solr Anti-Patterns” at 10:55 am

Summary:  Working as a consultant, software engineer and helping people in various ways, Rafał has seen multiple patterns in how Solr is used and how it should be used. Consulting on best practices is common, but talking about what NOT to do is not. This talk will point out common mistakes and roads that should be avoided at all costs, covering use cases and guidelines around general configuration pitfalls, data modeling and what to avoid when making your data indexable, and mistakes made when it comes to queries and searching for indexed data. Each use case will be illustrated by a before and after analysis where changes in metrics will be shown to bring a know-how worth remembering.

20% Discount Code

If you currently use a Sematext product or have been a client in the past and want to go, drop us a line for more info.

Hope to see you in DC!

Talk: Using Logstash and Elasticsearch Together

Sematext engineer and Elasticsearch / Logstash expert Rafal Kuc is giving a talk about using Logstash and Elasticsearch together at DevOps Days Warsaw on September 26.  The talk is titled “From zero to hero: easy log centralization with Logstash and Elasticsearch” and it will be livestreamed and available afterward as a video. Details will be posted here and @sematext in late September.

Here is the common logging problem that Rafal is going to address: digging through logs to find one particular event — or group of them. And going even further into this pain point — what if you have lots of servers and you don’t have a single place to look for logs?  Let’s face it, you can end up spending hours digging through log files to find events you are interested in and manually correlating them with events from different sources.  If you have read this far into this post then chances are you have been there, done that.

And here is how Rafal’s talk could benefit people working with Logstash and Elasticsearch: he is going to guide attendees/viewers through the basics of using Logstash and Elasticsearch together as the perfect combination for handling logs from multiple applications.  So if you want to see how Logstash + Elasticsearch can help you, this is a great opportunity to do so.  Attendees/viewers will learn how to set up Logstash, how to configure it to parse logs and, finally, sending them to an Elasticsearch cluster.

Rafal will also discuss tuning Elasticsearch for log management and centralized logging purposes, and show how to easily switch between shipping logs to a self-hosted solution like Elasticsearch / Logstash / Kibana (ELK) and instead ship logs to Logsene Log Management and Analytics by changing a single line in Logstash configuration.

Last but not least — Rafal should be at the event both days (September 25 & 26).  So if you are going to attend in person and have some Logstash and/or Elasticsearch topics you’d like to discuss with an expert, then reach out to him on Twitter – @kucrafal

Logging Expertise If You Need It

Logging is in our DNA.  If you could use some help with log management and centralized logging, Sematext engineers provide logging expertise with Elasticsearch, Logstash, Syslog, Flume and other logging-related tools. Drop us a line if you could use a logging expert!

Two Lucene/Solr Revolution 2014 Talks Accepted!

We recently got word from Lucene/Solr Revolution 2014 (in Washington, DC from Nov. 11-14) that talks submitted by two Sematext engineers were accepted as part of the Tutorial track!  They are:

• Tuning Solr for Logs by Radu Gheorghe
• Solr Anti-Patterns by Rafal Kuc

In “Tuning Solr for Logs” Radu will discuss Solr settings, hardware options and optimizing the infrastructure pushing logs to Solr.

In “Solr Anti-Patterns” Rafal will point out common Solr mistakes and roads that should be avoided at all costs.  Each of the talk’s use cases will be illustrated with a before and after analysis — including changes in metrics.

You can see more details about both talks in this recent blog post.

The full agenda, including dates and times for the talks, will be available soon on the Lucene/Solr Revolution 2014 web site.

If you do attend one of these talks please stop by and say hello to Radu and Rafal.  Not only do they know Solr inside and out, but they are good guys as well!

Love Solr Enough to Even Want to Attend One of These Talks?

If you enjoy Solr enough to even think of attending these talks — and you’re looking for a new opportunity — then Sematext might be the place for you.  We’re hiring planet-wide and currently looking for Solr and Elasticsearch Engineers, Front end and JavaScript Developers, Developer Evangelists, Full-stack Engineers, and Mobile App Developers.

Join Sematext at AWS Summit in New York City on July 10

Live, work or occasionally travel to New York City?  Going to be in our sleepy little town next Thursday, July 10th?  If the answer is “yes” (coupled with the fact that you’re reading our blog) then you might be interested to know that Sematext will be exhibiting at the AWS Summit in New York City on Thursday, July 10.  We’ll be in booth 323 AND we’re holding an iPad giveaway!

We encourage you to stop by, say hello, and check out the new goodies in SPM and the new Logsene Log Management and Analytics UI and functionality.   Now that SPM and Logsene are seamlessly integrated, you can not only find out that SOMETHING happened with AWS (or just about any other app), but also exactly WHAT happened.

If you are going to AWS Summit and want to set up a personal demo on site, drop us a line at mick.emmett@sematext.com.  Hope to see you there!

Not Going to AWS Summit in NYC?  You Can Still Try SPM and Logsene for Free

Try Logsene and/or SPM Performance Monitoring for Free for 30 days by registering here.  There’s no commitment and no credit card required.  You can also check out a live demo and see Storm, Kafka, Solr, Elasticsearch, Hadoop, HBase, MySQL, and other types of apps being monitored.

We’re Hiring!

If you are planning to attend AWS Summit New York — and even if you’re not — and you enjoy performance monitoring, log analytics, or search analytics, working with projects like Elasticsearch, Solr, HBase, Hadoop, Kafka, and Storm, then stop by our booth and meet us in person (or drop us a line)!  We’re hiring planet-wide!  Front end and JavaScript Developers, Developer Evangelists, Full-stack Engineers, Mobile App Developers…get in touch!

Event Stream Processor Matrix

We published our first ever UI-focused post on Top JavaScript Dynamic Table Libraries the other day and got some valuable feedback – thanks!

We are back to talking about the backend again.  Our Search Analytics and Scalable Performance Monitoring services/products accept, process, and store huge amounts of data.  One thing both of these services do is process a stream of events in real-time (and batch, of course).  So what solutions are there that help one process data in real-time and perform some operations on a rolling window of data, such as the last 5 or 30 minutes of incoming event stream?  We know of several solutions that fit that bill, so we decided to put together a matrix with essential attributes of those tools in order to compare them and make our pick.  Below is the matrix we came up with.  If you are viewing this on our site, the table is likely going to be too wide, but it should look find in a proper feed reader.

If you like working on systems that handle large volumes of data, like our Search Analytics and Scalable Performance Monitoring services, we are hiring world-wide.

Matrix part 1:

	License	Language	Scaling	Add or change rules on the fly	Other infra needed	Rule types
Esper	GPL2, commercial	java	Scale up	yes	none	Declarative, query-based
Drools Fusion	ASL 2.0	java	Scale up	yes	none	Declarative, mostly rule based, but support queries too
FlumeBase	ASL 2.0	java	Horizontal: natural sharding on top of Flume	yes	Flume	Declarative, query-based
Storm	EPL 1.0	clojure	Horizontal	Can be implemented on top of Zookeeper	ZeroMQ, Zookeeper	Provides only low level primitives(like grouping). Rule engine should be implemented manually on top.
S4	ASL 2.0	java	Horizontal	Can be implemented on top of Zookeeper	Zookeeper	Provides set of low level primitives. Somehow correlation support via joins. Documentation have a “windowing” section, but it empty.
Activeinsight	CPAL 1.0, commercial	java	Horizontal	yes		Declarative, Query-like
Kafka	APL 2.0	java	Horizontal		Zookeeper	Set of low level primitives

Matrix part 2:

	Docs / examples	Maturity	Community	URL	Notes
Esper	very good	mature, stable	medium	esper.codehaus.org
Drools Fusion	good	3 years, stable	small	jboss.org/drools/drools-fusion.html
FlumeBase	good	alpha	small	flumebase.org
Storm	exists	used in production	growing very fast	tech.backtype.com	good deployment features
S4	average	alpha, butused in production	medium (will grow under ASF)	s4.io
Activeinsight	poor	unknown	unknown	activeinsight.org
Kafka	good	used in production	small (will grow under ASF)	incubator.apache.org/kafka

So there you have it – we hope you find this useful.  If you have any comments or questions, tweet us (@sematext) or leave a comment here.  If you like working on systems that handle large volumes of data, like our Search Analytics and Scalable Performance Monitoring services, we are hiring world-wide.