FitNesse or NUnit?

As a relative newcomer to the arena of .NET test tools, FitNesse inevitably gets compared to NUnit. So let's tackle this issue now.

The primary target of FIT and FitNesse are customer-oriented acceptance tests, and that is where these tools really excel. NUnit and similar tools are aimed at code-oriented tests, verifying the functionality and the design of software from the developer perspective. However, the ease of writing and managing complex tests with FitNesse makes it also attractive as a tool for code-oriented tests.

The most important technical difference between NUnit tests and FitNesse (FIT) tests is that FitNesse tests are, for the most part, not in the code. They are described with HTML tables and run from an external server. This coin has two sides: it is easy to write FitNesse tests even before we start coding (so tests can truly guide the code), and half-done tests will not break the compilation. On the other side, FitNesse tests are somewhat harder to debug, and are not automatically refactored with the code.

Unit tools are excellent for testing code, but they suffer from a domain mismatch when we try to describe something outside of their basic language. Writing database or UI tests in C# can be quite inconvenient. With FIT/FitNesse, database tests can be described in a tabular form and UI tests in a story-like list of instructions.

Instead of splitting tests between NUnit and FitNesse by whether they are code-oriented or customer-oriented, I think that a more useful criterion is the area of coverage.

Quick basic tests: use NUnit

All developers should run basic tests (and make sure that they work) before committing code to the main branch. The basic test suite is normally executed a few times until all the obvious bugs are solved. So these basic tests have to run as fast as lightning, and they have to run on developer machines. Such tests typically do not connect to real services, but use mock objects to simulate the workflow. They should test small parts of the code, focusing on mistake-proofing in the small. In two words: unit tests.

From my experience, any unit test suite that runs longer than a minute is more of an obstacle than an aid. People will start skipping tests, which pretty much defeats the whole point of having them. This does not mean that we should not write tests that run longer, just that people should not be made to run them every time (see Don't mix quick and slow tests ). Michael Feathers summarised a discussion on the XP mailing list on a similar subject in this way:[2]

 

A test is not a unit test if

  • It talks to the database

  • It communicates across the network

  • It touches the file system

  • It can't run at the same time as any of your other unit tests

  • You have to do special things to your environment (such as editing config files) to run it

Tests that do these things aren't bad. Often they are worth writing, and they can be written in a unit test harness. However, it is important to keep them separate from true unit tests so that we can run the unit tests quickly whenever we make changes.

 
 --Michael Feathers

It works well to keep true unit tests in a tool like NUnit, so that we can run them from within the IDE. Note the word true: component and integration tests in disguise are not welcome here. Using NUnit makes basic tests easier to debug and troubleshoot, giving us a quicker turnaround time between spotting a problem and fixing it.

Manageable larger tests: use FitNesse

FitNesse has quite a few useful features that make tests easier to write and manage than with a unit-test tool. This is why I recommend keeping larger code-oriented tests in FitNesse, in addition to acceptance tests. Categorising tests like this also enables us to execute component and integration tests separately from the basic test suite, and not worry too much about their speed. They can then connect to real services, a proper database, and check larger and longer workflows.

FitNesse is miles better then unit-test tools for regression tests (see Chapter 13, Testing legacy code). The tabular language for describing tests in FitNesse makes it a good choice for relational data tests and database testing (covered in Chapter 12, Testing database code). Also, FitNesse integrates nicely with various libraries, like Selenium for web user interface testing (covered in Chapter 11, Testing web interfaces).

Because of its descriptive language, FitNesse can help to turn e-mails about bug reports into automated tests quickly. It is also a good tool for getting non–developers involved in the process of testing; it is much easier to get support people to write a FitNesse test than a NUnit test.

Not a silver bullet

FitNesse is not a general solution to all testing problems. For example, FitNesse does not support record-and-replay operations, which are a very effective way of automating GUI tests. It is also not a good tool for load testing and performance testing.

As explained in the previous section, FitNesse is not a replacement for unit testing tools, but is an addition to them. Think of it as a bigger hammer, which can also work on smaller nails, but is better used when you need more power and best combined with other tools when you need more precision.

Having said all this, FitNesse is an extremely useful utility in its own domain. It is an ideal tool for writing and managing story tests, the testing complement of user stories, which have become the preferred way of collecting requirements in agile teams. FitNesse truly helps in setting the target for development, making sure that everyone involved agrees what the target is, and automating verification to check how the development is going.