When optimizing your web application or server, the process can be a bit daunting.  In fact, its usually something that is never fully checked off your to-do-list.  The reason being is that you’ll never really know how your application will scale until it starts to show its growing pains.  Just when you thought that you’ve addressed the performance issues, a few weeks or months go by and new ones will crop up.

Before New Relic came along, I used to cat the slow-query-log.  I would address any queries that I saw took a while to execute.   Then there is the MySQL Performance Tuning Primer Script.  Also a great tool and will give you a good idea on how MySQL is performing as a whole.  I would continue to look at these things, but the load averages kept climbing and I could not figure out what the problem was.  I also couldn’t figure out which pages took the longest to load.

Enter New Relic

I actually considered throwing more hardware at the problem, since I could not figure it out through the usual means.  Then New Relic came along and I decided to give it a try. Here is my site performance graph since I first started using New Relic:

You’ll notice that when it was first turned on, the database was clearly the issue. The problem was UPDATING a table that has almost 2 million rows. (Duh.  Apparently, it was way more costly than I ever imagined)  Incrementing the view count on one of the main tables was the root cause to 90% of my problems.  I went ahead and created dedicated tables to hold these stats  and load averages eventually settled to around .20 across the board.  MySQL was no longer an issue.  FantasySP was silky smooth, once again!

The best part about New Relic is that once you set it up, you don’t have to spend time SSH’ing your box to find performance issues.  Everything you need is much easier to read, not to mention there is a performance breakdown based on each page/script.  Now I can easily see the pages, ajax requests, and background tasks that take the longest to complete.  All of these can be easily optimized now.  Their interface isn’t perfect, but it’s improving every day.

Impact on Google Bot & Organic Search

Of course when you optimize your website you do so for the end user experience.  The added plus to this is that Google Bot gets a huge benefit from it too.  I’ve been reading over and over about how load time is now a ranking factor.  Sure, but it’s not that big of a deal, right?  Well, lets take a look at how Google Bot responded to these tweaks:

Clearly, the faster the response time, the more pages that can be crawled in a given day.  The average response time for FantasySP.com is now at around 100ms.  The average response time for Mobile FantasySP is at 72ms.  Google Bot quickly realized that the site is performing faster and ramped up its crawl rate.

So you might be wondering, has the faster and more responsive site resulted in more search engine traffic from Google?  So far, the answer is yes.  By about 5 – 10% the past 7 straight days. Coincidence?  I doubt it.  Getting a bump in SERPs for being a considerably faster site is very real.  There is no question that the easier it is for Google Bot to crawl, the better off I’ll be in the long run.

Conclusion

New Relic has just released a whole bunch of new changes such as Real User Monitoring , an improved install process, and a much more affordable pricing structure.  It doesn’t matter if your a company of 15 developers or just one developer working on a side project. . . New Relic will be worth it.  In fact, for smaller sites you should be able to take advantage of New Relic’s 14 day professional trial and be done optimizing by the time your trial is finished.  However, I’ve been addicted to the stats and will eventually subscribe to their “Standard Package”.

New Relic essentially takes the guesswork out of optimization.  Now you can easily track down costly plugins in WordPress and even compare performance numbers from week to week.  This is by far the best tool you can use to make your site or app run faster.  Spend the time and install it, or submit a support ticket and have your host do it.

I can go on and on about New Relic and you will probably get a bit lost in the data for a few days.  So go ahead and give it a try.