Most of the time, a failing test is a good thing. It points to something wrong in your assumptions about your code, and can even guide you to a fix. The worst kind of failing test, though, is one that only fails sometimes. Say your test suite takes ten minutes to run, and one intermittent failure breaks the build. Congratulations, now your test suite takes twenty minutes to run, maybe longer.
Now, the ideal thing to do here would be to figure out why the test fails sometimes, and address the issue so the test is reliable all the time. But as developers, we don’t always have the luxury to set aside working on new features to deal with a flaky test. That’s where rspec-retry comes in.
rspec-retry is my favorite Band-Aid solution to intermittently failing tests in Rails, for those times when you can’t really dig into the issue. If a test fails, rspec-retry will attempt it again, until it passes or it reaches a set number of attempts. I’ve used it in a number of Rails applications, with good success. Even though each retry attempt adds some time to your overall test suite run, unless your suite is in really bad shape, then there’s a really good chance that it’ll be less time overall than running your whole suite twice (or more).
To start, add
rspec-retry to the
test group in your Gemfile, then
bundle install to add it. Then, configure RSpec to use it–I prefer to
do this in a separate file like spec/support/rspec_retry.rb, but you
can also include it in your main spec/rails_helper.rb file, as shown
in rspec-retry’s README. Whatever works for you. Here’s the
configuration that’s made me happy so far:
Let’s walk through these configurations:
verbose_retryindicates whether to log the retry attempt in test output; this is useful for seeing which tests are intermittently failing.
display_try_failure_messageslogs each failure with the usual stack trace to help with troubleshooting and, ideally, fixing the ailing test.
default_sleep_intervalis optional, but useful to configure if things need a moment to breathe between retries–for example, sometimes waiting a second or two gives an external service at the root of a problem time to correct itself. rspec-retry doesn’t offer an environment configuration variable to set this, so I created my own, as shown in this sample configuration.
retry_callbackaccepts a block and runs between each retry. So far, all I’ve needed to do here is ensure Capybara is reset between attempts.
Finally, I use the
RSPEC_RETRY_RETRY_COUNT environment configuration
to set the number of times to try a failing test, as documented in
rspec-retry’s README. If that value is let unset, then a test will only
be tried once. It’s important to point out here that the second
is a misnomer; this environment variable indicates the total number of
attempts, including the first failure. In other words, if
RSPEC_RETRY_RETRY_COUNT is set to three, then RSpec will attempt to
pass the test three times, not once plus three times.
Even though you can configure rspec-retry on a per-test basis, it’s counterintuitive to me–I’d expect a setting on an individual test to override any project-wide configurations, but that’s not the case. If you set rspec-retry to try a specific test five times, but your project environment configuration is set for three times, then it’ll try three times. With that said, I’ve been content to just stick to the project-wide settings. If you’ve found something different, please let me know how you’ve got rspec-retry configured.
What’s the best setting for
RSPEC_RETRY_RETRY_COUNT? That’s really up
to you and your team. Start with two (remember, that’s the first failure
and one retry) and work up from there. If you find yourself setting the
number much higher than, say, five, then consider addressing the flaky
tests in your suite sooner rather than later. Each of those retries will
add seconds to your test runs, and those seconds add up!
rspec-retry has been a major win for my testing toolbox, and I hope you find it helpful, too. If you do, try your best to treat it as a temporary solution, and create an item in your issue tracker of choice to get to the bottom of the intermittent failure you’re working around, and correct it!
If you liked my series on practical advice for adding reliable tests to your Rails apps, check out the expanded ebook version. Lots of additional, exclusive content and a complete sample Rails application.
Ruby on Rails news and tips, and other ideas and surprises from Aaron at Everyday Rails. Delivered to your inbox on no particular set schedule.