G1GC vs. Concurrent Mark and Sweep Java Garbage Collector

Lately, inquiries about the G1GC Java Garbage Collector have been on the rise (e.g., see this search-lucene.com graph).  So last Friday we took one of our Search Analytics servers running HBase RegionServer and a semi-recent version of Oracle Java (build 1.7.0_07-b10) and switched it to G1 (-XX:+UseG1GC and no other G1-specific parameters).  Below are some before and after graphs taken from SPM, which is what we use to monitor all our servers, of course.  Click on them for full size.

G1GC Switch from CMS

Can you tell what time we made the switch?  If you cannot tell, have a look at the graph below for a more obvious and more dramatic before and after.

G1GC vs. CMS, GC Details

Bases on the above graphs, we can see that, at least in this particular workload, G1GC does more collections, but they are smaller and thus take less time.  We’ll keep monitoring JVMs using G1, and if they continue to perform well, we’ll switch to them wherever applicable.  Here is one more look at the situation:

G1. vs. CMS Dashboard

On this SPM Dashboard you should compare the panel/graph on top (using CMS) with the panel/graph directly below it (G1).

In short, we are seeing some more GC work done on the Young generation with G1, but a lot less work being done with the Old generation and thus we see far fewer long pauses than with CMS garbage collector!

If performance and/or monitoring excites you, come join us! If you want to learn more about Java GC and live in or near New York, register for Living with Garbage by Gregg Donovan, a Senior Software Engineer at Etsy.

Please tweet this post about G1GC.

Have you tried G1 Garbage Collector lately?  How did it work for you?

– @sematext

What’s New in SPM 1.13.0

We pushed a new SPM release to production this morning and it’s loaded with goodies.  Here is a quick run-down of a few interesting ones. The slightly longer version can be found in SPM Changelog:

PagerDuty integration. If you are a PagerDuty user, your alerts from SPM can now go to your PagerDuty account where you can handle them along with all your other alerts.

Ruby & Java libraries for Custom Metrics.  We open-sourced sematext-metrics, a Ruby gem for sending Custom Metrics to SPM as well as sematext-metrics for doing the same from Java.

Coda Metrics & Ruby Metriks support.  We open-sourced sematext-metrics-reporter, a Coda’s Metrics reporter for sending Custom Metrics to SPM from Java, Scala, Clojure, and other JVM-based apps, and we’ve done the same for Metriks – the Ruby equivalent of Coda’s Metrics library.

Puppet metrics. We begged James Turnbull to marry Puppet and SPM and write a Puppet report processor thats sends each of the metrics generated by a Puppet run to SPM, which he did without us having to buy him drinks….yet.

Performance.  We’ve done a bit of work in the layer right behind the UI to make the UI a little faster.

CentOS 5.x support.  Apparently a good number of people still use CentOS 5.x, so we’ve update the SPM client SPM to work with it.  You can grab from SPM Client page.

– @sematext

What’s New in Sematext Search Analytics

We’ve been busy with adding functionality and improving SPM, our Performance Monitoring service, but we’ve also been quietly working on our free Search Analytics service (internally known as SA).  As a matter of fact, not coincidentally, SPM and SA share a lot of backend components, as well as UI-level pieces.  This, of course, allows a good amount of software reuse and lets us improve both services without double the effort.

Here are some of the new things in Search Analytics:

Live Demo. Before you create your Sematext Apps account (it’s free, no need to take out your credit card) you can check out the live demo and see both SPM and SA in action.

Real-time. Previously, SA used MapReduce jobs to process the collected data and make them available as reports.  That is no longer the case. We’ve put SA on the same real-time OLAP engine that powers SPM.  This means you’ll see your graphs refresh and change before your eyes.

Dashboards. Just like we’ve added Dashboards to SPM, we’ve added them to SA, too.  You can now create custom Dashboards, pick which graphs you want on them, and where you want to put them on a Dashboard.  This is great if you want to display your search stats and trends on a large office monitor, as some SA users are already doing.  Moreover, you can put widgets from multiple SA and SPM Apps all on the same Dashboard, so you can see your performance metrics, SPM custom metrics (e.g. your KPIs), and your SA metrics on a single Dashboard, side by side.

Report/URL Sharing. Just like in SPM, we’ve made it possible to copy the URL from the browser, give it to anyone who has access to your SA reports.  When this URL is opened any filters or time selection will be automatically applied.  This makes it very easy for multiple people to easily share their “SA view” by sharing the URL instead of having to tell others which report they should look at, what time and what filters they should select, etc.

Graph Embedding and Sharing. Similar to URL Sharing (but different!), you can now share and embed individual SA graphs.  Each graph has a short URL that you can Tweet or share elsewhere.  You can also get a URL/HTML snippet and embed SA graphs in your blog, wiki, web site, etc.

User Sessions. We’ve added a User Sessions report. This report shows you the number of search sessions over time, the number of queries per session, as well as the number of distinct users using your search.  If anyone asked you to provide these numbers for your site, would you know them?  Most people would say no.  These metrics are good to know and with SA everyone will now be able to say yes to that question.

Distinct Queries. We’ve added the number of Distinct Queries to the Rate & Volume report.  Another nice metric.

Hourly Granularity. All graphs in SA now go down to hourly granularity.  This let’s you see how trends change over the course of each day.  This can lead to insights around differences in how your users use your search in the morning vs. during work hours vs. evening.

HTTPS/SSL. The SA JavaScript beacon can now use HTTPS.  This is important if your site uses HTTPS when displaying search results.  To send your search and clickstream data VIA HTTPS just replace http:// with https:// in SA JavaScript beacon.

We hope you like these changes.  Please leave a comment or let us know if you have suggestions for other improvements or new features you would like to see in Sematext Search Analytics.

– @sematext

What’s new in SPM 1.12.0

We’ve been very heads down since our last official release of SPM, as you can see from our SPM Changelog.  Here is some more info about new things you’ll find in SPM:

• For the impatient – there is now a demo user (click on this and you will be logged in as the demo user).  This lets you into both SPM and Search Analytics even if you don’t have an account, so you can see  reports for various types of SPM Apps as well as what Search Analytics reports are like.

• The SPM Client (aka SPM Agent) can now be installed as an RPM or a Debian packages – check SPM client page.  Until now, the installer was completely written in Bash, but using Jordan Sissel’s fpm we were able to easily put together SPM Client packages.  Moreover, you can now easily install SPM Client on Redhat, CentOS, Fedora, Debian, Ubuntu, SuSE, Amazon Linux AMI, and maybe some other smaller distros we didn’t get to test.  If you try it on some  other distro, please let us know if it worked or if you had issues, so we can help!

• SPM for Hadoop can now be used to monitor Hadoop HDFS, MapReduce, and YARN clusters.  We have tested SPM for Hadoop with CDH (Cloudera Hadoop Distribution), but it should work just as well with Apache Hadoop, or HDP (Hortonworks Data Platform), and perhaps MapR’s Hadoop distros as well.  Of course, you can use the same Sematext Apps account to monitor any number of your clusters and clusters of any type, so it’s extremely easy to switch from looking at your Hadoop metrics, to HBase metrics, to Solr or ElasticSearch metrics.  We are working on expanding monitoring support to other technologies.  Tell us what you would like SPM to monitor for you!

• SPM users already loved the Overview panel in SPM (see Screenshots), but the new Custom Dashboards are even cooler!  Here are some things to know about the new SPM Dashboards:
  • You can have any number of them and you can name them – there are no limits
  • You can add any SPM graph to any of your existing dashboards and it’s super easy to create a new dashboard when you realize you want the graph you are looking at on a new dashboard
  • You can drag and drop dashboard panels wherever you want, resize them, and they’ll nudge other panels around and snap onto a neat, invisible grid
  • You can add graphs from any number of different SPM applications to the same dashboard.  So you can create a dashboard that has all the graphs that are important to you and your application(s), and combine metrics from different (types of) apps in a single view.  For example, you can have a dashboard that shows the number of HBase compactions as well as ElasticSearch index merging stats on the same dashboard, right next to each other.
  • Not only can you mix and match graphs from different SPM Apps on the same dashboard, but if you are a Search Analytics user you can also have Search Analytics graphs right next to your performance graphs!  That’s powerful!
  • And for those who discovered Logsene, our soon to be announced Data & Log Analytics service, you can imagine how eventually SPM & Logsene will be able to play together and share the same dashboards!

• SPM Clients do a good job of gathering server and application metrics for applications covered by SPM.  But what if you want to monitor something else in addition to what SPM collects and graphs?  Or what if you want to feed in some non-performance data – say some user engagement metrics or some other Business KPI?  Well, you can do that now!  We’ve added Custom Metrics to all plans and this addition is free of charge!  And guess what?  You can build graphs with your custom metrics and put them on any and however many of your Dashboards you want, so you could potentially have your KPIs be shown next to your Performance Metrics, next to your Log Analytics, next to your Search Analytics!

• To help SPM users feed custom metrics into SPM we’ve released and open-sourced the first Sematext Metrics library for Java, with libraries for other languages to follow. Improvements welcome, pull requests welcome, support for other languages super welcome!

• You can now share, tweet, and embed your SPM graphs as you please.  Next to each icon you will see a little “share” icon that will open up a dialog window and there you can save the displayed short link, which you can tweet, or email.  You’ll also see an HTML snippet that you can copy and paste into your blog or your Wiki.  The shared or embedded widget is cool because you can select very specific filters in SPM and the shared/embedded graph will remember them.  Furthermore, you can choose to have your graph show a specific, fixed time range or, alternatively, you can choose to always show the last N minutes, or hours, or days…. This can be quite handy for sharing with team members who may not have access to SPM for one reason or another, but do want access to these graphs – think C*Os or anyone else who needs to be fed graphs and graphs, and some more graphs.

• Here is a cool and very handy feature that was suggested by George Stathis from Traackr, one of our early SPM users:  If you are a developer in charge of your application, your servers, your cluster, what happens when something breaks or just starts working poorly?  Many people turn to mailing lists to seek help from the open source community.  Experts on these mailing lists often ask for more information so they can provide better help.  Very often this information is in the form of graphs of various performance metrics.  So if you look at SPM you will see a little ambulance icon that, when clicked, puts together an email template that includes a short, public link to whichever graph’s ambulance icon you clicked on.  You can then incrementally keep adding links to other graphs to this email and, when you are ready, email it without leaving SPM.  This is incredibly easy and it makes it a breeze to put together an email that has a bunch of informative graphs in it.  This means you don’t have to go open your email client, you don’t have to type in the mailing list address, and you don’t have to jump between the email you are composing and SPM while you copy and paste links from SPM into your email client.  And while this functionality is really handy when you need help from experts and you want to give them as much information about your system as you can so they can help you better, you can also simply change where this email will get sent and send it to whoever you want.

• We listen to our user really carefully (see above!).  Some SPM users would like to keep their data longer, so we’ve added Pro Silver – a new plan with longer data retention, more metrics, more everything.

• We’ve improved SPM for ElasticSearch so it can now collect metrics even from non-HTTP nodes.

• We have made SPM Sender, the component responsible for sending all the metrics over to us for processing, more resilient and even lighter than before.

 

We want to hear what you think and what you want!  Please use comments to leave suggestions, requests, and feedback, or simply email us or tweet @sematext.