announcement

Apache Spark Monitoring in SPM

Apache Spark is an open-source, large-scale data processing engine built on top of the Hadoop Distributed File System (HDFS) and enables applications in Hadoop clusters to run up to 100x faster in memory, and 10x faster even when running on disk. So it’s not surprising the usage of Spark is booming as this Google Trends graph shows.

And while Spark usage has been going through the roof, Engineers and DevOps handling Spark have not had a good monitoring tool at their disposal. Well, that is, until now. By releasing the first Spark monitoring product to market Sematext has, with the addition of Spark monitoring to SPM Performance Monitoring, Alerting and Anomaly Detection, just filled a big hole in the Spark ecosystem.

Having just been added — along with other goodies — to the latest SPM release, SPM for Spark monitors all Spark metrics. It includes alerting, anomaly detection, log correlation, custom dashboards, events graphing, custom metrics, and a ton more. SPM can be installed On Premises or one can use the Cloud version run by Sematext, in which case the setup takes less than 5 minutes before graphs with performance metrics start appearing in real-time.

Enough with the words – Show me what Spark Monitoring looks like!

Have a look at a few screenshots to see how we graph Spark metrics in SPM. While we don’t use Spark at Sematext at this time and thus don’t have a live demo to show you, you can check out SPM’s live demo and see some other types of apps we monitor, such as Hadoop, HBase, Cassandra, Kafka, Storm, ZooKeeper, Elasticsearch, Solr, NGINX and NGINX Plus, Apache, MySQL, Redis, Java webapps and generic Java applications, as well as custom metrics.

Screenshot – Spark Executor metrics [click to enlarge]

Screenshot – Spark Worker metrics [click to enlarge]

And One More Thing…

SPM now works hand-in-hand with Logsene Log Management and Analytics. This makes the integration of performance metrics, logs, events and anomalies more robust for those of you looking to combine performance monitoring and centralized log management in one place — not only knowing that SOMETHING affected performance of your Spark cluster when you look at your performance metrics graphs or get an alert, but also exactly WHAT happened with the cluster by having immediate access to all relevant Spark event logs right there!

Take a Test Drive — It’s Easy and Free to Get Started

Like what you see here? Sound like something that could benefit your organization? Then try SPM and/or Logsene for Free for 30 days by registering here. There’s no commitment and no credit card required.

Announcement: New Functionality in SPM and Logsene

Summer is all but officially over, yet our work with SPM Performance Monitoring, Alerting and Anomaly Detection and Logsene Log Management and Analytics is not. While lots of us took a well-deserved break over the last 1-2 months, we added a few goodies to both SPM and Logsene. More interesting stuff is coming in the next release.

New in SPM

With SPM, the most notable addition is monitoring for Apache Spark. We’ll have a separate post about Spark monitoring with SPM next week with all the details, including screenshots. But that’s not the only new goodness; other additions include:

Integration with Nagios

You can now tell SPM where your Nagios lives and SPM will push all your Alerts to Nagios. If you use PagerDuty, SPM can push your Alerts there, too.

Lowered SPM agent overhead

Those sending large volumes of metrics will see the most benefit. The new agent makes use of Apache Flume to transport metrics.

Switched to sending metrics over HTTPS by default

You can still use HTTP if you want, of course.

These additions to SPM, along with recently announced monitoring support for NGINX Plus and NGINX make it an even more effective solution for organizations who are paying the unfortunate price of having a mish-mash of monitoring and alerting tools bolted together in an uneasy coexistence.

If you haven’t seen SPM yet, we have a live SPM demo so you can see it for yourself. The demo shows Hadoop, HBase, Kafka, Elasticsearch, Solr, MySQL, Redis, and other types of apps being monitored.

New in Logsene

Until now you could create an unlimited number of Dashboards with SPM graphs, and now you can do that with Logsene graphs, too. Moreover, you can place Logsene log graphs alongside SPM’s performance graphs, on the same Dashboard, and correlate your performance with your application logs!

This makes the integration of performance metrics, logs, events and anomalies more robust for those of you looking to combine performance monitoring and centralized log management in one place — not only knowing that SOMETHING happened when you look at your performance metrics graphs, but also exactly WHAT happened by having immediate access to relevant logs right there!

Screenshot – Dashboard with SPM Performance Graphs & Logsene Log Graphs [click to enlarge]

Take a Test Drive — It’s Easy and Free to Get Started

Like what you see here? Sound like something that could benefit your organization? Then try SPM or Logsene for Free for 30 days by registering here. There’s no commitment and no credit card required.

Announcement: NGINX and NGINX Plus Monitoring in SPM

The list of monitored stacks keeps growing! SPM Performance Monitoring, Alerting and Anomaly Detection just added support for NGINX and NGINX Plus. Now you can capture metrics like total requests and connections along with an overview report and a whole lot more.

NGINX is a high performance, open source web application accelerator that helps millions of the world’s busiest websites deliver more content, faster, to its users. It is the #1 web server for the top 10k busiest websites in the world. NGINX Plus is the commercial version which adds advanced features, professional services, and shows more juicy metrics.

NGINX and NGINX Plus monitoring in SPM complements recently announced support for apps like MySQL, Cassandra, Memcached, Apache and AWS CloudWatch. You can see a more complete list here (scroll down to “Monitored Apps”)

Here’s a glimpse into what SPM for NGINX and NGINX Plus provides – click on an image to see the full view or you can look at the actual SPM live demo showing SPM monitoring NGINX and NGINX Plus and their metrics.

Screenshot – NGINX Plus Overview (click to enlarge)

Screenshot – NGINX Plus Server Status (click to enlarge)

Screenshot – NGINX Plus Status Zones (click to enlarge)

Screenshot – NGINX Plus Caches (click to enlarge)

Screenshot – NGINX Plus Upstreams (click to enlarge)

Screenshot – NGINX Overview (click to enlarge)

Screenshot – NGINX Status (click to enlarge)

Live Demo — See SPM for Yourself

Check out SPM’s live demo to see NGINX and NGINX Plus monitoring for yourself. You’ll also be able to poke around and see Kafka, HBase, Elasticsearch, Solr, MySQL, and other types of apps being monitored.

Love the Idea of Monitoring NGINX and NGINX Plus? Take a Test Drive — It’s Easy to Get Started.

Try SPM Performance Monitoring for Free for 30 days by registering here. There’s no commitment and no credit card required.

Announcement: What’s New in SPM Performance Monitoring

A new SPM Performance Monitoring release was just pushed to production and it’s chock full of great new stuff to complement its proactive performance monitoring, alerting, anomaly detection, etc., available in the Cloud or On Premise. Here is a run-down of the juicier additions. The slightly longer version can be found in the SPM Changelog.

Integration with Logsene Log Management and Analytics

SPM performance monitoring now gives users access to even more metrics by seamlessly integrating with event data and logs via Logsene Log Management and Analytics. This enables correlation across performance metrics, alerts, anomalies, events, logs, and provides a single pane of glass across any organization’s IT infrastructure.

Monitoring Support for More Applications

We’ve added native monitoring support for the following applications to complement monitoring for Solr, Elasticsearch, Hadoop, HBase, Storm, Redis, Kafka, ZooKeeper and many others.

Screenshots

Eager to see pictures instead of reading content? Then jump below to see screenshots of these apps being monitored.

UI/UX Improvements

UI/UX improvements include: zooming and panning, client-side caching, wider and simpler metric charts, new filter slide-out panels with search capabilities, quick access to all dashboards and easier dashboard creation, and more.

Event Graphs

Events and event graphs are now integrated into SPM Performance Monitoring. You can now correlate various types of events, such as alerts, anomalies, application deployments, restarts, releases, server reboots, etc., with performance metrics graphs, as well as with logs and log graphs. Many of you will also be happy to hear that SPM can now turn Alerts into Events, and graph them as well. Check out Event Integration if you want to publish your own Events.

More Powerful Storm Monitoring

SPM Storm monitoring now serves up more metrics, more graphs and provides more granular details. This includes the addition of metric filters and the ability to monitor not just Spouts and Bolts, but also monitor Storm Workers.

Dashboard Enhancements

Creating and working with dashboards just got a lot more intuitive and flexible. This includes:

creating new dashboards via an intuitive “build your own dashboard” tool
easier navigation via Miller Columns (think column-oriented view in OSX Finder)
adding whatever graphs you want to an existing or brand new dashboard from within that dashboard
a pull down menu to select specific dashboards for much quicker access to a specific dashboard

Screenshot – SPM Dashboard (one of many possible views; click to enlarge)

Screenshot – Cassandra Overview (click to enlarge)

Screenshot – MySQL Overview (click to enlarge)

Screenshot – Memcached Overview (click to enlarge)

Screenshot – Apache Monitoring Overview (click to enlarge)

Screenshot – AWS CloudWatch EBS Read/Write Bandwidth (click to enlarge)

Live Demo

Check out SPM’s live demo to see it for yourself. You won’t find any demo apps showing Cassandra or Memcached metrics because we don’t use them at Sematext yet, but you’ll be able to poke around and see other types of apps being monitored — like Solr, Kafka, Hadoop and HBase, for example — along with MySQL, AWS, and Apache.

Consolidate Your App Monitoring — It’s Easy!

Many organizations tackle performance monitoring with a mish-mash of different monitoring and alerting tools cobbled together in an uneasy coexistence that is often far from seamless. SPM takes all that hassle away and makes it easy and comprehensive in one step.

Try SPM for Free for 30 Days

Try SPM Performance Monitoring for Free for 30 days by registering here. There’s no commitment and no credit card required.

We’re Hiring!

If you enjoy performance monitoring, log analytics, or search analytics, working with projects like Elasticsearch, Solr, HBase, Hadoop, Kafka, and Storm, then drop us a line. We’re hiring planet-wide! Front end and JavaScript Developers, Developer Evangelists, Full-stack Engineers, Mobile App Developers…get in touch!

Announcement: Cassandra Performance Monitoring in SPM

Cassandra is a distributed database management system built to handle massive data sets while providing high availability without compromising performance. That’s why many organizations use it for their mission-critical data. That being said, if you are running Cassandra then you’ll want to keep close tabs on it. And now SPM Performance Monitoring can help you do just that as the latest release supports Cassandra performance monitoring. Not to mention that all the usual SPM alerting, anomaly detection, etc., can be used with any of the Cassandra metrics.

Why Not Monitor More Than Just Cassandra?

Unlike some of the tools that only monitor Cassandra and nothing else, SPM Performance Monitoring covers Cassandra, the competing database HBase, and a lot more: Solr, Elasticsearch, Hadoop, MySQL, AWS CloudWatch, Memcached, Apache, and just about any other app you want to monitor. SPM also monitors Storm and Kafka, which are often used together with Cassandra (considered their most popular data store). Now all three can be monitored together!

Have a look at a few of the screenshots to see how we graph Cassandra metrics in SPM (a list of Cassandra metrics we monitor is listed further below). You can also check out SPM’s live demo. You won’t find any demo apps showing Cassandra metrics because we don’t use Cassandra at Sematext yet, but you’ll be able to poke around and see other types of apps being monitored, like Solr, Kafka, Hadoop and HBase, for example.

Overview (click to enlarge)

Compactions (click to enlarge)

Pending Writes (click to enlarge)

Write Requests (click to enlarge)

SSTable (click to enlarge)

Bloom Filter (click to enlarge)

SPM now monitors Cassandra metrics like:

Write Requests (rate, count, latency)
Read Requests (rate, count, latency)
Pending Write Operations (flushes, post flushes, write requests, replication of write)
Pending Read Operations (read requests, read repair tasks, compactions)
Pending Cluster Operations (manual repair tasks, gossip tasks, hinterd handoff, internal responses, migrations, misc tasks, request responses)
Compactions
Row/Key Cache (hit ratio, requests count)
Local Writes (rate, count, latency)
Local Reads (rate, count, latency)
SSTable (size, count)
Bloom Filter (space used, false positives ratio)

Please tell us what you think – @sematext is always listening! Is there something SPM Performance Monitoring doesn’t monitor that you would really like to monitor?

Consolidate Your App Monitoring — It’s Easy!

Many organizations tackle performance monitoring with a mish-mash of different monitoring and alerting tools cobbled together in an uneasy coexistence that is often far from seamless. Think Graphite+Nagios, for example. SPM takes all that hassle away and makes it easy and comprehensive in one step.

Try SPM for Free for 30 Days

Try SPM Performance Monitoring for Free for 30 days by registering here. There’s no commitment and no credit card required.

We’re Hiring

Announcement: Apache Monitoring in SPM

Responsible for your organization’s Apache servers? If so, you’ll be happy to hear that you can now capture metrics about your Apache web servers — along with alerting and anomaly detection, traffic spikes, dips/gaps, etc. — with SPM Performance Monitoring. This includes system-wide visibility into resource utilization, application performance, and operational health.

We’re also announcing other great new additions to SPM in the coming days to complement just-released monitoring support for MySQL, Cassandra, Memcached and AWS CloudWatch. Watch this space for details…

Here’s a glimpse into what SPM for Apache provides – click on an image to see the full view or you can look at the actual SPM live demo showing some of our own Apache servers and their metrics.

Overview (click to enlarge)

Workers (click to enlarge)

Traffic Rate (click to enlarge)

Scoreboard Serving (click to enlarge)

CPU Details (click to enlarge)

Memory Details (click to enlarge)

IO Read/Write (click to enlarge)

Network Traffic (click to enlarge)

Please tell us what you think – @sematext is always listening! Is there something SPM Performance Monitoring doesn’t monitor that you would really like to monitor?

Why Have Something to Just Monitor Apache?

…When you can monitor almost everything with one solution — SPM. Many organizations tackle performance monitoring with a mish-mash of different monitoring and alerting tools cobbled together in an uneasy coexistence that is often far from seamless. Think Graphite and Nagios just for Apache, and numerous other tools for…everything else. SPM takes all that hassle away and makes it easy and comprehensive in one step.

Try SPM for Free for 30 Days

Try SPM Performance Monitoring for Free for 30 days by registering here. There’s no commitment and no credit card required.

We’re Hiring!

Announcement: AWS CloudWatch Metrics in SPM

Wouldn’t it be great to have metrics for your AWS resources captured in one place? And beyond just capturing them, to also have the ability to do useful things with those metrics like create custom dashboards, detect and get alerted on metric anomalies, correlate them with other application events and logs, etc., all in a single pane of glass? Well…you can! SPM Performance Monitoring Alerting and Anomaly Detection now captures metrics about your AWS resources via AWS CloudWatch. This includes system-wide visibility into resource utilization, application performance, and operational health.

Why is this important?

AWS shows metrics for various AWS resources in CloudWatch available via AWS Management Console. This is nice, but it is not very practical if you already use and prefer SPM for your non-AWS resources (e.g. servers and applications running in your data center) or if you are already shipping your logs to Logsene. Do you really want to use another, separate UI for monitoring just your AWS resources? It’s also not practical to use alerting in CloudWatch if you already use alerting and anomaly detection functionality in SPM. Now that SPM gathers metrics for your AWS resources you can have a single place to see all your metrics, alerts, and anomalies.

Today we are exposing all Elastic Cloud Compute (EC2), Elastic Load Balancer (ELB), and Elastic Block Store (EBS) metrics in SPM. We will continue to add other AWS services to this list. Having AWS metrics in SPM means that you can apply not only threshold-based alerting to your AWS metrics, which AWS itself provides, but also SPM Anomaly Detection which is much more useful and which AWS CloudWatch does not offer.

Headache-relieving and Time-saving Benefits for Your Organization

Major benefits to using SPM to monitor AWS CloudWatch include:

there is nothing to install (i.e., it’s all agentless)
AWS cost and performance optimization
increase in transparency (i.e., now that AWS metrics are in a common monitoring app more people in your organization can see what you have running in AWS)

Have a look at a few of the screenshots to see some of the AWS metrics SPM graphs. You can also check out SPM’s live demo. Or, if you prefer to see the full list of AWS metrics SPM captures, just jump down past the screenshots to see them listed below.

EBS Read/Write Bandwidth (click to enlarge)

EBS Read/Write Latency (click to enlarge)

EC2 CPU Utilization (click to enlarge)

EC2 Read/Write Operations per Second (click to enlarge)

EC2 Network In/Out (click to enlarge)

ELB Backend 2XX/3XX/4XX/5XX Response Counts (click to enlarge)

ELB Healthy and Unhealthy Instance Counts (click to enlarge)

ELB Request Count (click to enlarge)

AWS Metrics List

Here is the complete list of AWS metrics that SPM gathers as of today:

EC2:

CPU Utilization
Disk Read Operations
Disk Write Operations
Disk Read Bytes
Disk Write Bytes
Network In
Network Out
Status Check Failed
Status Check Failed (Instance)
Status Check Failed (System)

ELB:

Healthy Host Count
UnHealthy Host Count
Request Count
Latency
ELB 4XX Responses Count
ELB 5XX Responses Count
Backend 2XX Responses Count
Backend 3XX Responses Count
Backend 4XX Responses Count
Backend 5XX Responses Count
Backend Connection Errors Count
Surge Queue Length
Spillover Count

EBS:

Volume Read Bytes
Volume Write Bytes
Volume Read Ops
Volume Write Ops
Volume Total Read Time
Volume Total Write Time
Volume Idle Time
Volume Queue Length

Metrics available for IOPS provisioned instances:

VolumeThroughputPercentage
VolumeConsumedReadWriteOps

Please tell us what you think – @sematext is always listening! Is there something SPM Performance Monitoring doesn’t monitor that you would really like to monitor?

Try SPM for Free for 30 Days

Try SPM Performance Monitoring for Free for 30 days by registering here. There’s no commitment and no credit card required.

We’re Hiring!

Announcement: MySQL Performance Monitoring in SPM

We live in the “data era”. A database of one kind or another lives at the core of virtually every application on the planet. As such, you better monitor your database performance and availability metrics and you better have alerting and anomaly detection mechanisms to notify you when things go awry before your users start calling you and the spiky-haired boss storms in.

We’re happy to announce MySQL performance monitoring being added to our SPM Performance Monitoring platform. The addition of MySQL monitoring complements Memcached performance monitoring we announced earlier today, HBase monitoring as well as the Redis monitoring we added earlier this year. We’re also announcing other great new additions to SPM in the coming days. Watch this space for details…

The latest release of SPM covers over 150 different MySQL metrics you can monitor, such as:

Availability
Replication
Connections
Slow queries
Cache
Data in/out
…and many more

Here’s a glimpse into what SPM for MySQL provides – click on an image to see the full view or look at the actual SPM live demo:

MySQL Overview

MySQL Runtime

MySQL Queries/Questions Rate

MySQL Cache Usage

MySQL Table Stats

Please tell us what you think – @sematext is always listening! Is there something SPM doesn’t monitor that you would really like to monitor?

Why Have Something to Just Monitor MySQL?

…When you can monitor almost everything with one solution — SPM. Many organizations tackle performance monitoring with a mish-mash of different monitoring and alerting tools cobbled together in an uneasy coexistence that is often far from seamless. Think Ganglia+Nagios. SPM takes all that hassle away and makes it easy and comprehensive in one step.

Try SPM Today for Free for 30 Days

Try SPM Performance Monitoring for Free for 30 days by registering here. There’s no commitment and no credit card required.

We’re Hiring!

Announcement: Memcached Performance Monitoring in SPM

Memcached is a high-performance, distributed memory object caching server that, despite being over 10 years old, is still used by great many organizations around the planet. While we don’t use it at Sematext, we are happy to announce our SPM Performance Monitoring platform can now monitor Memcached performance and graph Memcached stats. Of course, all the usual alerting, anomaly detection, etc. can be used with any of the Memcached metrics.

SPM currently monitors and graphs 15 different Memcached metrics, plus over 20 system metrics. Here is a partial list of the Memcached metrics SPM can monitor:

Number of cached objects
Number of evictions
Number of cache hits
Number of cache misses
Cache hit rate
Get/Set Rate
Touch/Flush Rate
…

Have a look at a few of the screenshots to see how we graph Memcached metrics in SPM. You can also check out SPM’s live demo. You won’t find any demo apps showing Memcached metrics, but you’ll be able to poke around and see other types of apps being monitored, like the MySQL DB monitoring demo.

Memcached Overview

Memcached Size – number of objects, evictions, used and free memory

Memcached Request Rates – rate of gets, sets, hits, and misses, plus the hit percentage

Memcached Touch and Flush Rate

Memcached Threads and Connections

Memcached Network Traffic

Please tell us what you think – @sematext is always listening! Is there something SPM doesn’t monitor that you would really like to monitor?

Why Have Something to Just Monitor Memcached?

…When you can monitor almost everything with one solution — SPM. Many organizations tackle performance monitoring with a mish-mash of different monitoring and alerting tools cobbled together in an uneasy coexistence that is often far from seamless. Think Ganglia+Nagios. SPM takes all that hassle away and makes it easy and comprehensive in one step.

Try SPM Today for Free for 30 Days

Try SPM Performance Monitoring for Free for 30 days by registering here. There’s no commitment and no credit card required.

We’re Hiring!

Announcement: Coming Up in Site Search Analytics

Have you checked out Site Search Analytics yet? If not, and if you think that gaining insight into user search behavior and experience is valuable information, then we’ve got something for you that’s battle-tested and ready to go.

This year we are adding some killer new features that will make SSA even more useful. So, if you want to be enjoying benefits like:

Viewing real-time graphs showing search and click-through rates
Awareness of your top queries, top zero-hit queries, most seen and clicked on hits, etc.
Having a mechanism to perform search relevance A/B tests and a relevance feedback mechanism
Not having to develop, set up, manage or scale all the infrastructure needed for query and click log analysis
And many others — here is a full list of features and benefits

…then you will love the new functionality we have on the way. After all, how can you improve search quality if you don’t measure it first and keep track of it?

Sound interesting? Then check out a live demo. SSA is 100% focused on helping you to improve the search experience of your customers and prospects. And a better search experience translates into more traffic to your web site and greater awareness of your business.