index.textile.in

<notextile>
---
layout: default
title: Rust Mock Shootout!
---
</notextile>
h1. Rust Mock Shootout!

Rust has several different mocking libraries.  None is clearly superior to all
of the others.  This project compares their feature sets, using an
apples-to-apples approach.  Each of several dozen features is implemented with
multiple mocking libraries, if possible.  The test results then show which
features work with which library.

h2. The Contenders

- Mockers := This is the oldest library in our shootout.  Inspired by
GoogleMock, it has an elegant syntax, powerful sequence matching, and a rich
variety of helper methods.  It works on stable Rust, though generic methods
require nightly.

- "Mock_Derive":https://github.com/DavidDeSimone/mock_derive := This was the
first Rust mocking library that supported automatically @derive@ ing the Mock
object from the target trait.  That saves a lot of typing!  Mock_Derive is
still very easy to use, though it lacks any ability to validate method
arguments.  It also can't work with generic traits, traits with generic
methods, traits defined in external crates, or multiple traits (like @T: A +
B@).  However, it does have a few rare features, like the ability to mock
foreign free functions, or traits with static methods. Unfortunately, it's
maintenance has fallen behind and it *no longer compiles on recent toolchains*.

- "Galvanic-mock":https://github.com/mindsbackyard/galvanic-mock := This is
part of a suite of testing libraries.  Along with galvanic-test and
galvanic-assert, it provides a comprehensive set of testing functionality for
Rust programs.  Galvanic-mock itself takes a behavior-driven approach to
testing.  It tries to separate the specification from what a mock does, from
how that mock is expected to be used.  It's a good all-around library: good
feature set, good documentation, good ergonomics.

- "Pseudo":https://github.com/iredelmeier/pseudo := All of the previous
libraries had one thing in common: they all require the nightly compiler. 
That allows them to do some pretty cool stuff, like tweak the language's
syntax, but it's also inherently unstable.  No code that relies on nightly can
be guaranteed to work with future compilers.  Pseudo is different.  It eschews
nightly-dependent features like @derive@ so that it can work on stable Rust.
Unfortunately, that also makes it pretty verbose.

- "Double":https://github.com/DonaldWhyte/double :=  Like Pseudo, Double runs
on stable Rust.  However, it uses a few macros to reduce the verbosity.  The
feature set is pretty similar.  In fact, the entire API is eerily similar.  I
think one of these crates must've copied from the other (perhaps they both
did).

- "Simulacrum":https://github.com/pcsm/simulacrum := This is a bit of a
different beast.  Whereas other mock libraries try to provide a clean API,
Simulacrum actually provides 3 different APIs.  That can be confusing at first,
but the result is great power.  Unusual requirements, impossible to meet with
the highest-level and most conveient API, can be satisfied (at greater effort)
with the lower-level APIs.  Simulacrum also runs on stable Rust, and manages
to do it with less verbosity than Double. Optional, nightly-dependent, support
for @derive@ is a work-in-progress.

- "Mock-it":https://github.com/nathanielsimard/mock-it := Mock-it is one of
Rust's simplest mocking libraries.  Its chief advantage is that its simplicity
allows it to run on stable Rust, though that's fast becoming a less unique
feature.  However, the lack of a high-level API also gives Mock-it some of the
power of Simulacrum; it can mock a @struct@, for example.  Overall, Mock-it
has few practical advantages, but it's a good starting point for someone
looking to build something bigger.

- "Mocktopus":https://github.com/CodeSandwich/Mocktopus := Mocktopus is a bit
  of a different beast.  Whereas every other library focuses on mocking
  @trait@ s, Mocktopus focuses on free functions.  In fact, the only way that
  it can mock a @trait@ is by manually creating an implementation of that
  @trait@, and then mocking every single function. Mocktopus also has only the
  most rudimentary support for expectations.  Its advantages are that it
  requires very little boiler plate, it can mock @struct@ s and free functions,
  and it handles generic functions well.

- "Mockiato":https://github.com/myelin-ai/mockiato := A fairly new library.
  The syntax is simple and terse, but it has few distinct features.

- "Mockall":https://github.com/asomers/mockall := Mockall is the newest
  contender.  It aims to have the best feature set of all of the above
  libraries, with the most ergonomic interface.  It runs on stable Rust, and
  uses no @unsafe@ code at all.  I wrote it myself, after writing this
  shootout and being unsatisfied with all of the other libraries.

h2. Features

I evaluated about three dozen features evaluated for each library.  The first
group are the essential features.  These determine the library's overall
capabilities.  While they may not be important to all users, they're considered
"essential" because a user can't implement any of these if the library doesn't
intrinsically support them.

The second group are the convenience features.  These are features that a user
can implement in terms of other essential features, for example, "Match
constant" can be implemented by matching with a method).  The lack of any of
these features shouldn't preclude the use of a certain library, but may cause
some annoyance.

The third group of "features" are really more informational in nature:

h3. Feature Matrix

%%TABLE%%

h3. Feature Definitions

- Associated types := Can the library mock a trait with associated types, like
@Iterator@?

- Checkpoints := When validating sequences of method calls, can the library
create checkpoints (aka Eras)?  A checkpoint divides expectations
chronologically.  All expectations created before the checkpoint must be
satisfied before it, and all expectations created afterwards must be satisfied
after it.

- Closures := Can the library mock a method that takes a closure argument, and
  execute that closure when checking call arguments and calculating return
  values?

- Reference parameters := Can a mocked method take its parameters by reference?

- Consume parameters := Can a mock method consume its parameters, passing them
by value to an arbitrary function?  This is important, for example, to keep the
parameters from @drop@ ping after the mocked method call.

- Consume self := Can the library mock a consuming method?  A consuming method
is one that takes the @self@ parameter by value, rather than by reference.
@into_*@ methods are a common example.

- Doctest := Can the library be used in doc tests?  The key difference here is
that doc tests are compiled with @cfg=false@.

- External traits := Can the library mock a trait defined in another module or
crate?

- Fallback := Can a mock object proxy certain method calls to a real object?

- Foreign := Can the library mock static external functions?

- Generic methods := Can the library mock traits with generic methods that
have parameterized arguments, and set expectations for those methods?  For
example, a method like @fn foo<T>(&self, t: T) -> u32@.

- Generic method with lifetime parameters := Can the library mock
functions or methods that have generic lifetime parameters, and set
expectations for those methods?  For example, a method like
@fn foo<'a>(&self, t: T<'a>)@.

- Generic return := Can the library mock traits with generic methods that have
parameterized return values, and set expectations for those methods?  For
example, a method like @fn foo<T>(&self, x: u32) -> T@.

- Generic structs := Can the library mock generic structs, like
@std::convert::Mutex<T>@?

- Generic traits := Can the library mock generic traits, like
@std::sync::Into<T>@?

- Impl Trait := Can the library derive mocks for methods that use
@-> impl Trait@ syntax?

- Inherited traits := Can the library mock inherited traits like @pub trait B:
A@?

- Match function := Can an expectation validate arguments with an arbitrary
function?

- Structs := Can the library mock a concrete @struct@ instead of just
@trait@ s?  This requires altering the module's namespace during unit tests.
The @test_double@ crate can do that.

- Modules := Can the library mock every function in a module?

- Multiple traits := Can the library create a mock that satisfies multiple traits,
so it can be passed to a function like @fn foo<T: A + B>(t: T)@?

- Traits := Can the library create a mock object that implements a @trait@?

- Return call with args := Can a mocked method return a value computed from the
arguments by an arbitrary function?

- Return reference := Can a mocked method return a reference with the lifetime
  of the mock object?

- Return mutable reference := Can a mocked method return a mutable reference
  with the lifetime of the mock object?

- Return owned := Can a mocked method return ownership of a value that does not
implement @Clone@?

- Return parameters := Can a mocked method modify method arguments provided as
mutable references?

- Sequence := Can the library assert that methods are called in a particular
order?  This feature is implemented to different degrees by different libraries.
One library only supports validating call order on a method-by-method basis.
Another can validate the call order of different methods of the same mock
object.  Only one library can validate the call order of methods of different
objects.

- Send := Are mock objects @Send@?  If not, then the library cannot mock a
  trait that is.

- Static methods := Can the library mock a trait that has a static method?  A
static method, also called an "associated function" is one that does not receive
any form of @self@ as a parameter.  In Rust, they must be called using
@Trait::function()@ syntax, rather than @object.function()@.  That makes it
impossible for a mock library to set an expectation on such a method.  However,
mocking such a trait is still useful for setting expectations on other methods.

- Times range := Can the library expect a method to be called a variable number
of times, bounded by a range?

- Where clauses := Can the library mock generic traits and methods with where
clauses, and will the expectation still satisfy those where clauses?

- Derive := Can the library automatically generate the Mock struct by
@derive@ ing on the trait?  This feature can save a lot of typing, but it
invariably requires nightly Rust.  It also can't be used when mocking external
traits.

- Match combinattions := Expectations can match boolean combinations of other
validators.

- Match constant := Can an expectation match arguments with constants?

- Match operator := Can an expectation match arguments with common operators,
like @eq@ , @gt@ , etc?

- Match pattern := Can an expectation match enum arguments using patterns?

- Match range := Can an expectation match arguments with ranges?

- Match wildcard := Can an expectation match any argument?

- Return constant := Can an expectation return a constant?

- Return default := Can mocked methods automatically return the
@Default::default()@ value?

- Return panic := Can expectations @panic@ instead of returning?

- Times once := Can an expectation assert that it's called exactly once?

- Times any := Can a mocked method be called any number of times?

- Times n := Can an expectation assert that it's called an arbitrary number of
times?

- Times never := Can a mocked method expect to never be called?

- Maximum arguments := The maximum number of arguments for a mocked method.

- Rustc := Minimum required compiler version.  None of these six crates
guarantee a specific version, just "stable", or "nightly".  Typically, the
crates that require "nightly" will only work with a narrow range of nightly
versions.

- First release := Date of the first release on crates.io .

h2. Conclusion

The best mocking library is ... none of them.  No one library is clearly
superior to all of the others.  Every project will need to choose a different
mocking library with the features needed for that particular project.  Some
projects may event need to use multiple libraries in combination.  But a few
things stand out:

* Now that proc macros have been available on stable Rust for awhile, there
isn't any good reason to require nightly anymore.  However, a few libraries
have optional features that are only available when built with nightly
compiler.
* However, not all libraries make use of proc macros.  Those that don't, like Mock-it, are more verbose to use than those that do.
* On the other hand, proc macros can be difficult to debug when they fail to compile.  Galvanic-mock is probably the worst in that respect.  A non-proc-macro based library may be easier to debug.
* Validating call sequences is an underappreciated aspect of mocking.  Only
Mockers and Mockall have full support for validating the sequences of multiple
objects' methods.  However, Simulacrum's sequence syntax is more elegant.
* Mocking is complicated, and none of these libraries' authors anticipated every
possible situation.  However, Mock-it and Simulacrum proved surprisingly
versatile, since they allow the user relatively low-level access into the
library itself.
* Mockall is the clear winner on features, as it should be, because I wrote it
with prior knowledge of all the other libraries.
* Mockall and Mockers are probably the easiest libraries to use.

h2. Discussion

Tear me a new one at the "Rust Forums":https://users.rust-lang.org/t/rust-mock-shootout/17700