Blog (protip) - Fred Wu's Tech

Protip: Faster Ruby Tests with DatabaseCleaner and DatabaseRewinder

2013-09-18T06:50:00.000000Z

Please also see this blog post on tweaking your ruby GC settings.

I use and love DatabaseCleaner, although historically I had never paid too much attention on the performance of its varies cleaning strategies - I’d always used truncation.

We use Postgres, and after digging around and finding out the difference between DELETE and TRUNCATE, I ended up improving our test suite speed by about 30-40% simply by tweaking the cleaning strategies.

RSpec.configure do |config|
  config.before :suite do
    DatabaseCleaner.clean_with :truncation
    DatabaseCleaner.strategy = :transaction
  end

  config.before do
    if example.metadata[:js] || example.metadata[:type] == :feature
      DatabaseCleaner.strategy = :deletion
    else
      DatabaseCleaner.strategy = :transaction
      DatabaseCleaner.start
    end
  end

  config.after do
    DatabaseCleaner.clean
  end
end

Essentially, we want to truncate the DB only once before the whole suite runs to ensure a clean slate DB, then we only want to use deletion on Capybara tests, everything else should just use transaction which is the fastest strategy.

Now, as a bonus, I have just discovered @amatsuda’s DatabaseRewinder which is a lightweight alternative that supports only ActiveRecord. It offers comparable performance with a much similar API.

RSpec.configure do |config|
  config.before :suite do
    DatabaseRewinder.clean_all
  end

  config.after do
    DatabaseRewinder.clean
  end
end

By the way, we also use parallel_tests to scale our test suite to multiple processes, even on Travis CI and Wercker.

Hooray to faster tests! :)

]]>

Protip: Ruby Devs, Please Tweak Your GC Settings for Tests!

2013-09-06T11:13:37.000000Z

It was made apparent to me that many ruby devs either aren’t aware or couldn’t be bothered to tweak their ruby garbage collector settings.

Well, if you are using MRI, please start tweaking your GC settings. Here’s what I use (on my 15” Macbook Pro Retina):

export RUBY_GC_MALLOC_LIMIT=90000000
export RUBY_FREE_MIN=200000

Not only can you tweak it for your local dev machine, you can also tweak Jenkins, Travis CI, Wercker and other CI solutions, making instant speed gain for your test suite!

Here’s what we get:

            Before After
Local:      8m22s 5m52s
Travis CI:  10m10s 6m0s
Wercker:    8m45s 6m24s

YMMV depending on your system and available RAM.

]]>

Gotchas in the Ruby Sequel Gem

2013-08-21T12:39:00.000000Z

I haven’t really used Sequel much therefore I am definitely a newbie. However, after days and nights of frustration, endless debugging and some search-fu during the development of Datamappify, I have finally arrived at the conclusion that Sequel is a capable library, as long as you are aware of the gotchas.

Gotcha 1: Always use “`select`“/“`select_all`“, or your data records will mysteriously have wrong IDs!

In ActiveRecord, joining an associated model couldn’t be simpler:

Post.joins(:author)

In Sequel, despite having a similar API for models to declare associations and their corresponding primary and foreign keys, you cannot do a join without specifying the keys:

Not good:

Post.join(:authors)
# or
Post.join(Author)

Better:

Post.join(:authors, :id => :author_id)

You would think the version above works - it doesn’t. Even worse, the above example will give you incorrect data - the IDs of the Post records will now contain the IDs from their corresponding Author records! This is because upon a join, Sequel merges attributes from both models into a single hash.

The correct version:

Post.join(:authors, :id => :author_id).select(:posts __id, :posts__ title, :posts__body)
# or
Post.join(:authors, :id => :author_id).select_all(:posts)

Gotcha 2: Always call “`all`“ at the end of the chain, or the chain will present data in a different format.

In ActiveRecord, all of the below examples return an ActiveRecord::Relation collection:

Post.where(:title => 'Hello world')
Post.joins(:author)
Post.includes(:author)

And indeed, calling first on any of them returns an object of class Post (assuming the result collection is not empty).

Post.where(:title => 'Hello world').first.class #=> Post
Post.joins(:author).first.class #=> Post
Post.includes(:author).first.class #=> Post

In Sequel, the below examples all return a Sequel::DataSet collection:

Post.where(:title => 'Hello world')
Post.eager(:author)
Post.eager_graph(:author)

But let’s see what we get from calling first.class on them:

Post.where(:title => 'Hello world').first.class #=> Post
Post.eager(:author).first.class #=> Post
Post.eager_graph(:author).first.class #=> Hash

Huh? Last one is a Hash? It turns out, if you call all at the end of chains to convert them to Arrays, then the returned collections are consistent:

Post.where(:title => 'Hello world').all.first.class #=> Post
Post.eager(:author).all.first.class #=> Post
Post.eager_graph(:author).all.first.class #=> Post

]]>