Performance Testing Rails Applications: How To?

This guest post is by Gonçalo Silva, who is a full-time Ruby on Rails developer at escolinhas.pt and has participated in the Ruby Summer of Code 2010. He loves and contributes to many open-source projects, being a fan of Linux, Ruby and Android. He likes to call himself a hacker, but that’s just an excuse for being in front of the computer all the time. Oh, and he tweets at @goncalossilva.

Rails 3.1 is just around the corner, and it brings enhanced performance testing tools. Let’s have a look at this often overlooked feature of our web application framework of choice.

This isn’t new

Rails has had built-in performance testing tools since version 2.2. Originally developed by Jeremy Kemper, these allowed developers to test the performance of their applications by writing integration tests which could be benchmarked and profiled under MRI. He later introduced two scripts – benchmarker and profiler – which were great to quickly benchmark or profile small snippets of code.

Actually, this is kind of new

I came across these tools during last year’s Ruby Summer of Code. I remember feeling astonished and bit ashamed about not having played with them before. I couldn’t use them at their full potential because of the lack of full support for YARV (or MRI 1.9), so I set off fixing that. While working on it, I’ve made a list of other things these tools lacked, that I wanted to implement after RSoC, namely: – Rubinius support – JRuby support – Test configurability – Decoupling benchmarker and profiler from RubyProf

Everything listed above is now implemented. Rails 3.1 will ship with these improvements and we’ll no longer have excuses for not using these great tools Rails provides for all of us.

Why you should care

The web should be fast. Response times are a key factor in user experience and there is very limited patience for slow websites. Ruby interpreters aren’t famous for being performant and our beloved framework isn’t known for getting faster with new releases. Nevertheless, we want our websites to be fast and responsive, and buying tons of hardware isn’t always an available choice – we need our code to be fast. We should care.

How does this work?

Rails’ performance testing tools allow you to quickly detect performance bottlenecks. As a rule of thumb, use benchmarking to detect the problem and then use profiling to understand it. Profiling provides in-depth information about your code and what it’s doing, but it lacks the speed and simplicity of benchmarking.

Patching your Ruby interpreter

You can skip this section if you’re a Rubinius/JRuby/REE user.

If you’re an MRI/YARV user, you’ll need a patched interpreter to access all available metrics. Before you run off, let me tell you that it’s very simple to install a patched Ruby interpreter nowadays. Thanks to Wayne, the author of RVM, all you need to do is to specify an additional flag when installing your interpreter, like this: rvm install 1.9.2 --patch gcdata Or, if you’re still using 1.8 (really?): rvm install 1.8.7 --patch ruby187gc

That’s all, folks. You now have a patched Ruby interpreter. If you want, you can have your patched interpreter side by side with your regular one, by simply assigning a name to it:

rvm install 1.9.2 --patch gcdata --name perf
rvm 1.9.2-perf  # my patched interpreter
rvm 1.9.2       # my regular interpreter

And that’s it.

Editing your Gemfile

You can skip this section if you’re using Rubinius/JRuby.

If you’re not, you’ll need to add RubyProf to your Gemfile:

gem 'ruby-prof', :git => 'git://github.com/wycats/ruby-prof.git'

Don’t forget to remove this from your Gemfile and re-run bundle install if you intend to switch to Rubinius or JRuby.

Performance tests

In order to use these tools, you’ll need to write performance tests. These tests are just like integration tests, except that the point is not to assert anything. They’ll just run the code that you want to see benchmarked/profiled.

Generating

As expected, Rails does this stuff for you. Just run:

script/rails generate performance_test example

And a new file will be placed in test/performance/example_test.rb containing the default test:

require 'test_helper'
require 'rails/performance_test_help'
class ExampleTest < ActionDispatch::PerformanceTest
  # Refer to the documentation for all available options
  # self.profile_options = { :runs => 5, :metrics => [:wall_time, :memory]
  #                          :output => 'tmp/performance', :formats => [:flat] }

  def test_homepage
    get '/'
  end
end

Editing

Since ActionDispatch::PerformanceTest inherits from ActionDispatch::IntegrationTest, you can use all available helpers for integration tests in your performance tests. For instance, if you wanted a test for your login action you could use:

class LoginTest < ActionDispatch::PerformanceTest
  fixtures :users
  self.profile_options = { :metrics => [:wall_time, :memory] }

  def test_login
    post_via_redirect "/login", :username => users(:youruser).username, :password => users(:youruser).password
  end
end

Tweaking

Starting with Rails 3.1, performance tests can be configured. As you’ve probably figured out from the aforeshown LoginTest, all you need to do is to specify an optional hash of options to use when benchmarking/profiling. You can use one set of options for each class. Not all options are available to all interpreters, especially the ones related with profiling. Metric/output availability for each interpreter will be shown below. You can skip this section and come back later, after grasping the whole concept. You’ll also be able to check it out on Rails’ performance testing guide once 3.1 comes out.

Metric availability

Benchmarking

Profiling

Output availability

Running

Finally, it’s time to run your tests. Let’s start with benchmarking:

rake test:benchmark

And the output should be similar to this:

ExampleTest:
ExampleTest#test_homepage (16 ms warmup)
           wall_time: 0 ms
              memory: 17 KB
             objects: 195
             gc_runs: 0
             gc_time: 0 ms
 homepage (0.75s)

LoginTest:
LoginTest#test_login (92 ms warmup)
           wall_time: 10 ms
              memory: 180 KB
 login (0.44s)

Finished in 1.193759 seconds.

If any result disappoints you, profile it:

rake test:profile TEST=test/performance/login_test.rb

And you should get a similar output:

LoginTest:
LoginTest#test_login (105 ms warmup)
           wall_time: 69 ms
              memory: 2.4 KB
 login (5.02s)

Profiling will give you much more information than what’s printed on your terminal.

Reviewing results

By default, performance tests store their results in tmp/performance (although it can be changed by specifying a value for :output in the profile_options hash). For benchmarks, this is pretty straightforward: it stores one CSV per metric (LoginTest#test_login_memory.csv, for instance) with the results as time goes by.

measurement,created_at,app,rails,ruby,platform
183222,2011-08-10T18:15:09Z,,3.1.0.rc5,ruby-1.9.2.290,i686-linux
216344,2011-08-11T14:37:59Z,,3.1.0.rc5,ruby-1.9.2.290,i686-linux
(...)

When profiling, however, the result files are extremely important. They contain the juicy details of your test runs. Similarly to benchmarking results, there will be one file per metric. There are, however, multiple formats available, specially if you’re using RubyProf (and consequently MRI/REE/YARV). These formats can range from messy flat text files to awesome HTML stack traces, and they will provide valuable input when spotting bottlenecks.

The scope of this article is not to explore RubyProf’s available output formats, but you should have a look at the available printers. However, keep in mind that RubyProf supports more metrics and output formats than Rubinius/JRuby‘s profilers. These can only measure wall time when profiling, and will only print their results in Flat/Graph text formats.

RubyProf’s HTML stack printer

Quick tests

Performance tests are great, but they can be inconvenient when all you want is to quickly test a small snippet of code. For this, Rails provides two command line tools: benchmarker and profiler.

Open your terminal and run:

rails benchmarker 'User.all'

And it will work as if you had created a performance test and put that code in it. Very simple, right? Another example:

rails profiler 'User.all' 'User.find_by_login("goncalossilva")' --runs 3 --metrics cpu_time,memory # profiling memory won't work under Rubinius/JRuby (benchmarking memory will!)

Two things pop up from this code snippet: you can run multiple tests in a single command and you can specify options as you would with normal performance tests.

To get a glimpse at all available options, run:

rails benchmarker --help rails profiler --help

What can be done with this?

A lot of things can be accomplished with these tools. First and foremost, you can assess the performance of your application by benchmarking certain parts, either through tests or simple snippets of code. After finding potential bottlenecks, you can use profiling to gain a greater insight into what’s happening and how it can be improved.

There are other useful tasks that can be done with these tools. You could, for instance, compare the performance of different interpreters on your application:

    rvm 1.9.2
    rails benchmarker 'MyModel.slow_method' 'get "/"' --metrics wall_time,memory
    rvm ree
    rails benchmarker 'MyModel.slow_method' 'get "/"' --metrics wall_time,memory
    rvm rubinius
    rails benchmarker 'MyModel.slow_method' 'get "/"' --metrics wall_time,memory
    rvm jruby
    rails benchmarker 'MyModel.slow_method' 'get "/"' --metrics wall_time,memory

Now you’ll know which interpreter takes less/more time/memory when it’s opening your homepage/running MyModel.slow_method.

Giving it a try

If you’ve come this far, now you know how to use these powerful tools. Try playing with them: I’m sure you’ll find valuable information about your applications’ performance, and potentially spot some easily fixable bottlenecks. With little effort, your application will be faster, you will be prouder and your users will be happier!

Feel free to ask questions and give feedback in the comments section of this post. Gonçalo has also written a guest blog post for RubyLearning before, titled – “Ruby gems — what, why and how“. Fellow Rubyists, if you would like to write a guest blog post for RubyLearning write to satish [at] rubylearning.org