Skip to content

Latest commit

 

History

History
147 lines (103 loc) · 4.19 KB

flaky.md

File metadata and controls

147 lines (103 loc) · 4.19 KB
Raw

Fight the flakiness

Tips on detecting and solving flaky tests in Rails apps.

❗️Reproduce before fixing

This is the most important rule: make sure you can reproduce the flakiness before starting to fix it. Flakiness is the bug, and like any other bug, it should be first identified. Find a root cause, and fix it. Even though the flakiness reasons could be identified by eyes, you still need to prove it with your code.

Make sure tests run in random order

For example, config.order :random for RSpec.

Make sure you use transactional tests

For example, config.transactional_tests = true for RSpec.

Avoid before(:all) (unless you sure it's safe)

  • Use rubocop-rspec RSpec/BeforeAfterAll cop to find before(:all) usage
  • Consider replacing with before or before_all

Travel through time and always return back

  • Find leaking time traveling with TimecopLinter
  • Add config.after { Timecop.return }
  • If you rely on time zones in the app, randomize the current time zone in tests (e.g. with zonebie) to make sure your tests don't depend on it.

Clear cache / in-memory stores after each test

For example, for ActiveJob (to avoid have_enqueued_job matcher catching jobs from other tests):

RSpec.configure do |config|
  config.after do
    # Clear ActiveJob jobs
    if defined?(ActiveJob) && ActiveJob::QueueAdapters::TestAdapter === ActiveJob::Base.queue_adapter
      ActiveJob::Base.queue_adapter.enqueued_jobs.clear
      ActiveJob::Base.queue_adapter.performed_jobs.clear
    end
  end
end

Generated data must be random enough

  • Respect DB uniqueness constraint in your factories (check with FactoryLinter)

Make sure tests pass offline

Tests should not depend on the unknown outside world.

  • Wrap dependencies into testable modules/classes:
# Make Resolv testable
module Resolver
  class << self
    def getaddress(host)
      return "1.2.3.4" if test?
      Resolv.getaddress(host)
    end

    def test!
      @test = true
    end

    def test?
      @test == true
    end
  end
end

# rspec_helper.rb

Resolver.test!
  • Provide mock implementations:
# App-specific wrapper over S3
class S3Object
  attr_reader :key, :bucket
  def initialize(bucket_name, key = SecureRandom.hex)
    @key = key
    @bucket = bucket_name
  end

  def get
    S3Client.get_object(bucket: @bucket, key: @key).body.read
  end

  def put!(file)
    S3Client.put_object(bucket: @bucket, key: @key, body: file)
  end
end

# Mock for S3Object to avoid calling real AWS
class S3ObjectMock < S3Object
  def get
    @file.rewind
    @file.read.force_encoding(Encoding::UTF_8)
  end

  def put!(file)
    @file = file
  end
end

# in test
before { stub_const "S3Object", S3ObjectMock }

Do not sleep in tests

When writing System Tests avoid indeterministic sleep 1 and use have_xyz matchers instead–they keep internal timeout and could wait for event to happened.

Remember: Time is relative (Einstein).

Match arrays with match_array

If you don't need to the exact ordering, use match_array matcher instead of eq([...]).

Testing NotFound with random IDs

If you test not-found-like behaviour you can make up non-existent IDs like this:

expect { User.find(1234) }.to raise_error(ActiveRecord::RecordNotFound)

There is a change that the record with this ID exists (if you have before/before(:all) or fixtures).

A better "ID" for this purposes is "-1":

expect { User.find(-1) }.to raise_error(ActiveRecord::RecordNotFound)

Read more