I just wanted your opinion on SLIM as opposed to standard FIT/Fitnesse. Are there things that can only be done via the FIT/Fitnesse route that cannot be done via SLIM? So for acceptance tests and integration tests can I just use SLIM?

We want to exploit the BDD abilities of Scenario tables in SLIM. Ideally I would like to use SLIM to undertake all kinds of tests. I assume it has all the same capabilities? Are there any issues?

On the face of it, SLIM seems to be the preferred way forward for new FitNesse implementations as it is being actively developed and maintained by the same folks who develop FitNesse. FIT is a bit of an orphan at the moment, Rick Mugridge and I were talking about taking over that integration and enhancing it.

In terms of features, SLIM gives you better compatibility across platforms because a lot of the test system responsibility has been taken over by FitNesse itself (parsing HTML, deciding how to interpret a table, storing and reading symbol values). Watch this video for more information about the differences in responsibilities.

On the other hand, because of the way SLIM works, fitlibrary flow-mode interaction is practically impossible. Most of my clients still use flow mode tests as that is a great way to write and maintain a very thin fixture layer for complex tests.

SLIM also allows you to use Scenario tables, as you mentioned. Scenario tables give testers a lot more power as they can script multi-step execution and compose lower level fixtures into higher level tables, without involving developers to do the same in fixtures. Depending on your environment and team, this might or might not make sense. For covering and existing system with regression tests, it probably does. For using acceptance tests as a guide for development, beware of overdoing it. I am very concerned about long-term maintenance costs of such tests. What happens in this case is that people are effectively programming with tables – doing the same in code would allow you to benefit from IDE support for refactoring, file management and all sorts of other things that make IDEs useful. You lose all that by using scenario tables in order to make testers a bit more independent. I would rather suggest training the testers some basic coding skills so that they can write fixtures.

My experience with Fit and other agile acceptance testing tools is that they cost more than they’re worth. There’s a lot of value in getting concrete examples from real customers and business experts; not so much value in using “natural language” tools like Fit and similar.

The two failure patterns that Shore describes in his post are falling back on testers to write everything and merging acceptance and integration tests. I’ve experienced both of these myself, and it seems that they are common in general. We discussed both during the top 10 ways to fail with acceptance testing openspace session at CITCON Europe last year. However, there are good ways to solve both problems.

I never really expected customers to write anything themselves, but I was relatively successful in persuading them to participate in specification workshops that led to examples which were then converted to acceptance tests later. Another idea I discovered while doing the research for my new book is discussing the key examples with customers and then going off to write detailed test pages, which then get verified by the customers. The third good idea is doing ad-hoc specification writing sessions when a developer needs information, by involving a tester and a business analyst. This is a lot less formal than a specification workshop and gives you similar benefits if you have all the knowledge in the room (or readily available) most of the time.

Not preserving acceptance tests as a separate group and mixing quick and slow tests is something that most people, at least according to my ongoing research, get burned with at some point but again teams learn from that and improve.

One of the biggest benefits from acceptance testing for me was that the teams finally get a source of information on what goes on in the system as reliable as the code itself. Without acceptance tests, code is the only thing you can really trust and any other documentation gets outdated very quickly. And such tests are much easier to read and understand than the code because they are on a higher level and in a natural language. Having a live specification helps me quite a lot when change requests come in later. It also helps with handing over and taking over code. Acceptance tests stay relevant throughout the project because they are automated, and automated tests are kept up to date in order for them to pass. Automation, and consequently a tool, are necessary to get this benefit. With informal agreements and on-site reviews that Jim Shore describes, I guess something else needs to be in place to facilitate this.

I agree with Shore that it takes a while for the problems with tools such as FIT to surface, but I’m not sure whether that is tool related or not. Most people I spoke to so far said that it took them between six months and a year to discover that acceptance testing isn’t about the tools but about communication, and that the biggest benefit is in the examples as Shore wrote. A notable exception to six months to a year rule was Lisa Crispin‘s team who generally started out knowing what they need (but that’s because she had done it before). Clear examples and improved communication are the biggest benefits of the process, but using a tool brings some additional nice benefits as well. A tool gives us an impartial measure of progress. Ian Cooper said during the interview for my new book that “the tool keeps developers honest”, and I can certainly relate to that. With tests that are evaluated by an impartial tool, “done” is really “what everyone agreed on”, not “almost done with just a few things to fill in tomorrow”. I’m not sure whether an on-site review is enough to guard against this completely.

Concordion is really simple. Its instrumentation only allows programmers to set global test variables, execute fixture methods and compare actual results with expected values. Programmers can use special HTML element attributes to mark words or phrases that are used as test inputs or compared to test results. Web browsers will just ignore unknown element attributes, so Concordion test instrumentation is effectively invisible to people that are not interested in test automation. For example, here is a HTML document that I’ve used as a simple test:

<html xmlns:concordion="http://www.concordion.org/2007/concordion">
<body>
<h1>Free delivery</h1>
<ul>

 <li concordion:execute="#offer = checkFreeDelivery(#type, #books)">
 Free delivery <span concordion:assertEquals="#offer">is</span>
 offered to a <b concordion:set="#type">VIP</b>
 customer with <b concordion:set="#books">10</b> books in the cart.
 </li>

 <li concordion:execute="#offer = checkFreeDelivery(#type, #books)">
  It <b concordion:assertEquals="#offer">is not</b> offered to
  a <b concordion:set="#type">regular</b> customer
  with <b concordion:set="#books">10</b> books</li>

 <li concordion:execute="#offer = checkFreeDelivery(#type, #books)">
  It <b concordion:assertEquals="#offer">is not</b> offered to
  a <b concordion:set="#type">VIP</b> customer with
  only <b concordion:set="#books">9</b> books.</li>

</ul>
</body>
</html>

The concordion:execute command on the LI element specifies that the offer variable is set by executing the checkFreeDelivery method and passing type and books variables set in the element text. The concordion:set command is used to set a variable based on the inner text in the element. The concordion:assertEquals command inspects the inner text in the element and compares that with the result of a method or current variable value. For repetitive specifications and calculation rules, Concordion also supports attributes for tables similar to the FIT ColumnFixture.

At the moment, Concordion only supports Java fixtures — It actually works as a JUnit extension. This provides direct integration with JUnit test runners, making it much easier to execute Concordion tests from popular development environments and integrate into continuous build systems. It also shortens the learning curve required to start using Concordion. The fixture is simply a JUnit test class. It should be called the same as the HTML file (in this case, the file was ConcordionTest.html, so the class is ConcordionTest.java), and declare the methods that are used by concordion:execute.

package conctest;

import org.concordion.integration.junit4.ConcordionRunner;
import org.junit.runner.RunWith;

@RunWith(ConcordionRunner.class)

public class ConcordionTest {
	public boolean freeDelivery(String type, int books){
			if (books<9) return false;
			if (!"VIP".equals(type)) return false;
			return true;
	}
	public String checkFreeDelivery(String type, int books){
		return freeDelivery(type, books)?"is":"is not";
	}
}

My first impression of this is that the fixture model of Concordion does not depend on inheritance so it is a lot simpler to learn than FIT and somewhat more flexible to write Concordion than FIT fixtures. On the other hand, Concordion lacks the extensibility and powerful model of type adapters and cell handlers that enable FIT to bind domain objects and business services directly to the specification.

When the JUnit test is executed, Concordion runs through the files and executes commands, verifying expected outputs with actual results in assertEquals. JUnit test run will tell you whether all tests passed or were there failures, and Concordion also saves the results in the system temporary folder in HTML form, making it easier to see what actually went wrong. The example above intentionally has an error — here is the screenshot of the result:

Concordion does not have a test management tool and relies completely on the programmer’s IDE to manipulate and execute tests. I have mixed feelings about this. Although file management within an IDE makes it much easier to put tests in the same version control system as the domain code, I miss the ability to re-use parts of the test specification such as common set-ups or test components with macro variables that are available in FitNesse. It also puts tests completely under the control of programmers.

Concordion takes some ideas that have evolved as best practices to use FIT/FitNesse and makes them very explicit, actively discouraging other ways of working. For example, acceptance tests have to be stored in the same version control system as the code, there are no pre-built test building blocks that would encourage scripting. On the other hand, while FIT/FitNesse community is moving firmly towards using domain objects directly in acceptance tests and reducing the amount of glue code, to eliminate translation between layers and promote domain-driven design, Concordion model is stuck in requiring you to create an explicit test method for every verification in the fixture.

The thing that really worries me is the fact that HTML files are stored next to Java classes and that developers need to add non-standard HTML attributes to the test page. This smells to me of a hand-over of tests from business people to developers at some point, which is a practice that I don’t approve at all. In my opinion, tests should be shared between the whole team and not handed-over down the pipeline. I guess that this can be avoided with some discipline and research into tools that will surely not overwrite non-standard HTML attributes. But I would still like to see a proper test management tool for business people to use.

I’ve been hearing a lot about Concordion recently, mostly being mentioned by people as an interesting tool to look into. Safari has no books about it, all I could find on Google and Technorati blog searches are reports of people coming across the tool and doing some basic things with it, such as this post. I would be really interested in hearing stories from people that have real-world experiences with this tool to share. What happens six months later — are tests easier to manage than with FitNesse, has it turned out to be good enough tool to talk to business people? Is my fear of hand-over real, or is it not a problem at all?

The case against inheritance

FIT and its architecture started back in 2002 and heavily relies on inheritance as a way to extend the framework and integrate business domain code into the framework. That makes it hard to maintain the framework, which Mike pointed out, as lots of people depend on inner workings of the Fixture class. (I myself have complained a few times about his refactorings of the .NET test runner that broke my existing code). It also limits the options we have to integrate domain code with the framework. I often write a lot of boilerplate code to wrap domain objects into fixtures, since the integration layer has to extend from the Fixture class. Some options like target objects, system under test and domain adapters make it easier to connect domain objects directly to fixtures, but they require us to write lots of boilerplate code and we still have to use fixtures. It is hard to incorporate some generic test management functionality, such as starting and rolling back transactions or invoking the debugger, without changing the test runner or again extending fixtures.

Rich domain models and fixtures

Both FIT.NET and Java FitLibrary have started to move towards using a rich domain model (as in DDD) more effectively than with traditional mapping of each step into a fixture or a Dofixture method. Domain fixtures and domain adapters in FIT.NET and Java FitLibrary “explain” how to utilise rich domain objects. What we’ve noticed on the mini-session is that these domain adapters effectively do two things:

1. explain how to create and access domain objects (for setup and verification tasks)
2. explain how to find domain objects and execute methods on business services (action tasks between setup and verification)

In addition to that, it is interesting that FIT.NET and Java FitLibrary now have a mechanism to define meta-data about a whole test suite (suite configuration file in .NET and the suite fixture in FitLibrary).

Moving forward in that direction, we felt that we could lift the requirement to implement fixtures for a large majority of cases if the underlying business code is developed using a rich domain model and other DDD principles.

Integration with repositories

Most projects based on a rich domain model will now effectively use a repository to store and find objects, so the first benefit could be achieved by implementing support for a few popular repository method naming conventions instead of encapsulating it into a custom domain adapter/fixture. The suite meta-data could be used to select the appropriate naming convention and define the appropriate repository objects for domain objects. This could be used to prepare domain objects for the test or verify changes in domain objects later. For example, the traditional setup fixture becomes obsolete and could be replaced simply by a repository call. For example, this is a common set-up table:

If there is a repository defined for the Customer class, this table in the setup part of the test would be executed by instantiating a Customer object for every row, populating FirstName, LastName and Phone and calling CustomerRepository.Save(Customer c) method.

A very important new idea is to store the result of this operation directly into a symbol; we discussed the option to mark a column that contains the appropriate symbol key with a * or similar, but concluded that it would be better to just use the first column as the key name. So the previous table would create three Customer objects and store them automatically into symbols “Mike Smith”, “Mike Scott” and “Tom Cruise”. So this would effectively replace even Column Fixtures that are used to extract the result and store it into a symbol. The rationale behind storing objects automatically into symbols is that the objects prepared in the setup are most likely required later in the test (why else would they be created?).

If there is no repository, objects would just be created by instantiating the domain class and storing it into the symbol. If there is a repository, this would result in an additional call to the repository and the result of the repository call would be stored into the symbol. The reason behind that is to allow the repository to further amend the object if required by business rules.

If the first cell in the first row is not a class name, then it could map to a service method name or a repository method name. The corresponding method would get called by mapping the parameters directly and the result (if not void) would again be stored in the symbol. Repositories and services would be configured in the suite meta-data, so there would be no requirement to implement additional code.

In the verification part of the test, the same table (if the first cell maps to a domain class name) would act as a column fixture that retrieves the symbol value based on the first column, and then compares the other fields to actual domain object values retrieved from the symbol. (Possibly by going to the repository again to find an object by id if there is a repository, to support test-specific stuff). If the first cell in the first row maps to a method name, the method would be executed and the results would be compared to the table. A third option would work on repository finders and would also test for list length (similar to a list or set fixture). For example

would call CustomerRepository.FindAll() and compare the results with the table looking for missing or surplus customers as well.

would call CustomerRepository.FindActive() and so on.

Test-specific functionality

The acceptance sometimes do not list all the properties, to make tests more focused. If there is no fixture in between, the creation of objects in the repository might fail because of that. A solution for this case is to implement a decorator over the normal repository that adds the test-specific functionality (eg populates 10 remaining mandatory fields), and then using the test repository instead of the default repository in the suite meta-data configuration. This would also be the way to implement any other test-specific functionality. There is no way to avoid implementing this test-specific code, but this model would not require test-specifics to inherit from Fixture or require writing any other boilerplate code.

Talking to business services

One of the most important practices that DDD introduced is the ubiquitous language, a common jargon shared across all phases and participants of a project. One of the best practices for acceptance testing arising from that is using the same phrases for the same concepts in examples, acceptance tests and code. With this in mind, we really should not need any translation layers between the fixture tables and the domain code (both in objects and services). They should use the same jargon so we should be just able to glue them together by implementing a few naming conventions.

DoFixture and SequenceFixture support the concept of system under test, mapping calls directly to domain services. However, using SequenceFixture makes the test too technical and not really suitable for discussion with business people. Although this is relatively understandable:

It is much more natural to write this example as

DoFixture allows us to write scripts like this, but then requires the system under test to implement silly method name such as UserSendsInvitationToUserForRoom. No self-respecting programmer would create such a name in a business service.

Guessing the operation

Working on a few examples from the FIT book, we discussed how this link could be implemented better, for example creating something similar to DoFixture but that would map to more sensible method names, which would be written in normal domain objects and services. This would promote the idea of ubiquitous language even further, because it would suggest the correct service method or domain object method name. The final idea was to try out a few keyword combinations and find what the method name is, similar to service fixture but not requiring it to be the first keyword.

This could theoretically map to three methods: ChatService.Enter(User u, Room r), Room.Enter(User u) or User.Enter(Room r). If we have a ChatService in the suite meta-data configuration, then the algorithm could try to map keywords to either the service methods or class methods and look for the appropriate call. A possible pitfall is that this could lead to ambiguity. To promote the use of natural language, we would apply some basic transformations (such as stripping the final s from Enters), and ignoring some words (such as “a” or “on”). So we could use “Book a flight”, not forcing people to write “book flight”. Then:

would map to FlightService.BookFlight(Location from, Location to). The list of ignored words would probably also be configured on the suite level with some defaults.

The sentences could be even shorter, since the setup part would already have stored Ana, LOTR, LAX and JFK in symbols and we could extract the correct type.

We also discussed automatic object tree traversal, for example “Mike’s credit card number” would map to (Symbol(‘mike’)).CreditCard.Number.

Checking in natural language

Instead of

we would use the “is” keyword:

this would work very similar to “check” at the moment, meaning it will execute everything on the left how ever it is marked up, and compare the result to the cell on the right, but it reads much more naturally.

Tables or no tables

An interesting discussion followed after this on whether we need tables at all, or should we just try to recognise keywords. My personal opinion is that tables are good because they clearly identify what is a test script and what is just a description on the page. In any case, a bit of CSS tweaking can make the cells invisible.

Extension points

We also discussed the importance of providing a number of extension points or hooks for acceptance test execution, that would enable us to augment or modify the way tests are executed (eg run the test in a transaction and roll back, filter/debug on individual cells). The idea was to use something similar to filters (aspect-oriented) to attach to test runners and again encapsulate only test-specific code into that. The suite meta-data configuration file should allow us to specify filters that are applied to the whole test, to individual tables and individual cells (possibly with regex content matching to narrow it down).

Conclusion

With direct domain mapping to repositories and domain objects, setup and verification parts of most of the tests in a rich domain model project can easily be automated without writing fixtures if there is no test specific functionality. Smart keyword mapping will allow us to do the same for the middle parts of the test, where we mostly talk to domain services. Automatically storing objects from the setup part in symbols will make it easy to use those objects in the rest of the test. Using all that, we could effectively connect acceptance tests directly to rich domain models, without fixtures. Repositories and services would be configured in the test suite configuration file, so there would be no additional code required apart from genuinely test-specific code. Any genuinely test specific code could be encapsulated in test-specific repository decorators, and test-specific services, without the requirement to extend any class from the Fixture framework. Generic test-control functionality could be injected and reused across tests using configurable extension points, which should probably just implement a particular filter interface.

Challenges

Ambiguity seems to be the biggest challenge at the moment, and we need to work out strategies for avoiding ambiguity. Another challenge is to identify important naming conventions for repositories so that people can use this approach without changing the way they work at the moment.

An open question is how to divide the test clearly into three parts (setup, action, verification) while keeping them optional (eg support setup-verification or setup-action or action-verification). Rick’s domain fixture at the moment relies on a horisontal line (<HR/>) but this does not work with the standard FitServer. Possibly have some keywords (such as Given, When, Then from BDD) that stand as separate tables on the page.

Going forward

We experimented with relatively simplistic examples, and although this looks promising we need to try it out on some more complicated code. I promised to try to re-write some of my production tests using this model to identify potential pitfalls. If you did not give up reading this by now, then you are probably genuinely interested in the subject, so your feedback and ideas would also be greatly appreciated.

• a print-ready PDF
• an executable FitNesse wiki
• a live web site