Is Your Test Suite Acceptable?

Posted by Jeremy Voorhis Fri, 25 Aug 2006 23:43:00 GMT

Is your test suite acceptable? I mean acceptable in the Chomsky sense of the word. Here are a few heuristics:

  1. Do you understand your test at first glance?
  2. Do you understand the code that is being exercised after reading your test?
  3. Do your tests tend to have 1 assertion or 10 assertions? More?
  4. Do your tests have cryptic names, like test_component?
  5. Are you using external test fixtures? If so, how much time do you spend reading and cross-referencing them?
  6. If you are using external test fixtures, how well do they serve your needs? Are you creating tests that require you to cross reference external fixtures and temporary objects?

Here are some design principles that are useful for lowering the mental overhead your test suite requires:

Give your tests meaningful names

If you can accurately infer what your test is trying to accomplish by reading its method name, you can understand it much faster. It’s even more helpful when that name appears in a report generated by a test runner!

Favor acceptability over DRYness

Which is worse: repeating a string literal two or three times within a test, or losing time reading the test later? Of course, this is a tradeoff, and both sides have cumulative effects.

Group your tests according to their context

Just because all of your tests exercise the same class, they don’t all have to live together. If you are testing a Stack class, don’t be afraid to separate your tests for an empty stack and your tests for a full stack. You can then use the setup method to instantiate the kind of stack you are testing, and your tests will be easier to understand. Conversely, when tests that are grouped together rely on different contexts, you will spend longer establishing your mental model when you read these tests later.

Write individual tests for a method’s individual behaviors

Rather than writing one test for one method, try analyzing the expectations you are placing on your method. Let’s use a method that accepts a blob, generates a filename and saves the blob to the named file as an example. Write one test to ensure the file name is generated correctly, and write another test to ensure the contents on disk are correct.

Keep your tests short

If you can use setup or some other means to create all the context you need, and your tests target individual behaviors, you can write a suite of tests that is very scannable by human eyes.

Avoid external fixtures when feasible

Relying on external fixtures may cause you to spend much time cross-referencing your test against many fixture files. What’s worse, changing an external fixture for one test may have the side effect of breaking other tests!

Comments

Leave a response

  1. James Mead said about 10 hours later:

    I totally agree. Avoiding external fixtures is one of the main drivers in the development of Mocha which allows you to mock or stub the ActiveRecord methods like find, create & save.

  2. Luke Redpath said about 12 hours later:

    Great post – its things like context-based tests that makes RSpec so appealing (not to mention its syntax).

    Its time to stop encouraging things like:

    class PersonTest < Test::Unit::TestCase def test_create end end

  3. Aslak Hellesoy said about 19 hours later:

    Good stuff.

    A common antipattern with tests (or in RSpec lingo – specs or examples) is actually overusing the DRY (Don’t Repeat Yourself) principle. To quote David Chelimsky’s Agile spec manifesto> we prefer CLEAN SPECS over well factored specs That is to say that while we value well factored specs, we value clean specs more

    It’s OK to have duplication in tests! -Because it often makes them more readable – you don’t have to hunt around and cross-reference.

    Sure, sometimes when you refactor you’ll have several tests failing, and you’ll have to do the same changes in many places. The important difference with duplication (non-DRYness) in tests and ‘code’ is that with the former, you know immediately where to go and fix things. So it’s not as bad as duplication in the ‘code’.

  4. Stéphane Tavera said about 21 hours later:

    I almost agree with your principles. But I would question the “Write individual tests for a method’s individual behaviors”. Shouldn’t we try instead to split any method in small unit-testable sub-methods ? OK, one drawback of this approach (in Java for example) is that you may It also leads to very small methods (some like it , some don’t but seems to be best for maintenance though). My 2 cts.

  5. JV said about 24 hours later:

    @Aslak

    I admit, some of my principles are derived from Dave Astels and the BDD community at large. Rspec is a good tool, and it is hard not to follow the principles I stated when using it. When I am mindful, however, I find I can accomplish the same thing with Test::Unit and it’s worked well for me.

    @Stéphane

    Small methods are good for maintenance in my experience, provided their responsibilities are clear and you are writing OO code, not procedural code. There are certain bits in Rails, however, where that kind of decomposition is more difficult. For example, I wrote a declarative-style file management plugin for ActiveRecord called asset_field. The plugin works by adding behavior to ActiveRecord::Base#save via callbacks. I can only invoke #save on my model, and I want to test the resultant behavior. The plugin should have its own tests for its behaviors, but I find it necessary to also test my integration of the plugin by making sure the file is saved according to the parameters I have given it. I may have more than one test which calls #save and tests the model’s behavior, but I target only one behavior per test.

RSS feed for this post trackback uri

(leave url/email »)