esp

We published our first ever UI-focused post on Top JavaScript Dynamic Table Libraries the other day and got some valuable feedback – thanks!

We are back to talking about the backend again. Our Search Analytics and Scalable Performance Monitoring services/products accept, process, and store huge amounts of data. One thing both of these services do is process a stream of events in real-time (and batch, of course). So what solutions are there that help one process data in real-time and perform some operations on a rolling window of data, such as the last 5 or 30 minutes of incoming event stream? We know of several solutions that fit that bill, so we decided to put together a matrix with essential attributes of those tools in order to compare them and make our pick. Below is the matrix we came up with. If you are viewing this on our site, the table is likely going to be too wide, but it should look find in a proper feed reader.

If you like working on systems that handle large volumes of data, like our Search Analytics and Scalable Performance Monitoring services, we are hiring world-wide.

Matrix part 1:

	License	Language	Scaling	Add or change rules on the fly	Other infra needed	Rule types
Esper	GPL2, commercial	java	Scale up	yes	none	Declarative, query-based
Drools Fusion	ASL 2.0	java	Scale up	yes	none	Declarative, mostly rule based, but support queries too
FlumeBase	ASL 2.0	java	Horizontal: natural sharding on top of Flume	yes	Flume	Declarative, query-based
Storm	EPL 1.0	clojure	Horizontal	Can be implemented on top of Zookeeper	ZeroMQ, Zookeeper	Provides only low level primitives(like grouping). Rule engine should be implemented manually on top.
S4	ASL 2.0	java	Horizontal	Can be implemented on top of Zookeeper	Zookeeper	Provides set of low level primitives. Somehow correlation support via joins. Documentation have a “windowing” section, but it empty.
Activeinsight	CPAL 1.0, commercial	java	Horizontal	yes		Declarative, Query-like
Kafka	APL 2.0	java	Horizontal		Zookeeper	Set of low level primitives

Matrix part 2:

	Docs / examples	Maturity	Community	URL	Notes
Esper	very good	mature, stable	medium	esper.codehaus.org
Drools Fusion	good	3 years, stable	small	jboss.org/drools/drools-fusion.html
FlumeBase	good	alpha	small	flumebase.org
Storm	exists	used in production	growing very fast	tech.backtype.com	good deployment features
S4	average	alpha, butused in production	medium (will grow under ASF)	s4.io
Activeinsight	poor	unknown	unknown	activeinsight.org
Kafka	good	used in production	small (will grow under ASF)	incubator.apache.org/kafka

So there you have it – we hope you find this useful. If you have any comments or questions, tweet us (@sematext) or leave a comment here. If you like working on systems that handle large volumes of data, like our Search Analytics and Scalable Performance Monitoring services, we are hiring world-wide.