Concordion: Agile Acceptance Testing with free-form text

By
gojko
– August 25, 2008Posted in: articles

I finally had some time to take a look at Concordion, an acceptance testing tool that I’ve heard about on several conferences. Concordion is an interesting alternative to FIT. It is developed by David Peterson and released under the Apache opensource license. Similar to FIT, Concordion uses HTML documents as an executable specification and requires some glue code (fixtures) to connect the executable elements of that specification to the domain code. Unlike FIT, Concordion does not require the specification to be in any particular format — you can write examples as normal sentences, without any restrictions.

Concordion is really simple. Its instrumentation only allows programmers to set global test variables, execute fixture methods and compare actual results with expected values. Programmers can use special HTML element attributes to mark words or phrases that are used as test inputs or compared to test results. Web browsers will just ignore unknown element attributes, so Concordion test instrumentation is effectively invisible to people that are not interested in test automation. For example, here is a HTML document that I’ve used as a simple test:

<html xmlns:concordion="http://www.concordion.org/2007/concordion">
<body>
<h1>Free delivery</h1>
<ul>

 <li concordion:execute="#offer = checkFreeDelivery(#type, #books)">
 Free delivery <span concordion:assertEquals="#offer">is</span> 
 offered to a <b concordion:set="#type">VIP</b>
 customer with <b concordion:set="#books">10</b> books in the cart. 
 </li>

 <li concordion:execute="#offer = checkFreeDelivery(#type, #books)">
  It <b concordion:assertEquals="#offer">is not</b> offered to 
  a <b concordion:set="#type">regular</b> customer 
  with <b concordion:set="#books">10</b> books</li>

 <li concordion:execute="#offer = checkFreeDelivery(#type, #books)">
  It <b concordion:assertEquals="#offer">is not</b> offered to 
  a <b concordion:set="#type">VIP</b> customer with 
  only <b concordion:set="#books">9</b> books.</li>

</ul>
</body>
</html>

The concordion:execute command on the LI element specifies that the offer variable is set by executing the checkFreeDelivery method and passing type and books variables set in the element text. The concordion:set command is used to set a variable based on the inner text in the element. The concordion:assertEquals command inspects the inner text in the element and compares that with the result of a method or current variable value. For repetitive specifications and calculation rules, Concordion also supports attributes for tables similar to the FIT ColumnFixture.

At the moment, Concordion only supports Java fixtures — It actually works as a JUnit extension. This provides direct integration with JUnit test runners, making it much easier to execute Concordion tests from popular development environments and integrate into continuous build systems. It also shortens the learning curve required to start using Concordion. The fixture is simply a JUnit test class. It should be called the same as the HTML file (in this case, the file was ConcordionTest.html, so the class is ConcordionTest.java), and declare the methods that are used by concordion:execute. 

package conctest;

import org.concordion.integration.junit4.ConcordionRunner;
import org.junit.runner.RunWith;

@RunWith(ConcordionRunner.class)

public class ConcordionTest {
	public boolean freeDelivery(String type, int books){
			if (books<9) return false;
			if (!"VIP".equals(type)) return false;
			return true;
	}
	public String checkFreeDelivery(String type, int books){
		return freeDelivery(type, books)?"is":"is not";
	}
}

My first impression of this is that the fixture model of Concordion does not depend on inheritance so it is a lot simpler to learn than FIT and somewhat more flexible to write Concordion than FIT fixtures. On the other hand, Concordion lacks the extensibility and powerful model of type adapters and cell handlers that enable FIT to bind domain objects and business services directly to the specification.

When the JUnit test is executed, Concordion runs through the files and executes commands, verifying expected outputs with actual results in assertEquals. JUnit test run will tell you whether all tests passed or were there failures, and Concordion also saves the results in the system temporary folder in HTML form, making it easier to see what actually went wrong. The example above intentionally has an error — here is the screenshot of the result:

Concordion does not have a test management tool and relies completely on the programmer’s IDE to manipulate and execute tests. I have mixed feelings about this. Although file management within an IDE makes it much easier to put tests in the same version control system as the domain code, I miss the ability to re-use parts of the test specification such as common set-ups or test components with macro variables that are available in FitNesse. It also puts tests completely under the control of programmers.

Concordion takes some ideas that have evolved as best practices to use FIT/FitNesse and makes them very explicit, actively discouraging other ways of working. For example, acceptance tests have to be stored in the same version control system as the code, there are no pre-built test building blocks that would encourage scripting. On the other hand, while FIT/FitNesse community is moving firmly towards using domain objects directly in acceptance tests and reducing the amount of glue code, to eliminate translation between layers and promote domain-driven design, Concordion model is stuck in requiring you to create an explicit test method for every verification in the fixture.

The thing that really worries me is the fact that HTML files are stored next to Java classes and that developers need to add non-standard HTML attributes to the test page. This smells to me of a hand-over of tests from business people to developers at some point, which is a practice that I don’t approve at all. In my opinion, tests should be shared between the whole team and not handed-over down the pipeline. I guess that this can be avoided with some discipline and research into tools that will surely not overwrite non-standard HTML attributes. But I would still like to see a proper test management tool for business people to use.

I’ve been hearing a lot about Concordion recently, mostly being mentioned by people as an interesting tool to look into. Safari has no books about it, all I could find on Google and Technorati blog searches are reports of people coming across the tool and doing some basic things with it, such as this post. I would be really interested in hearing stories from people that have real-world experiences with this tool to share. What happens six months later — are tests easier to manage than with FitNesse, has it turned out to be good enough tool to talk to business people? Is my fear of hand-over real, or is it not a problem at all?

Tags: , , , , , , ,