• L.G.Shattuck — Communicating Intent and Imparting Presence
• William Crain — The Mission: The Dilemma of Specified Task and Implied Commander’s Intent
• James Shore: How I use Fit
• FitNesse.Org — main FitNesse web site
• Fitnesse.info — Community site with lots of examples
• Fixture gallery — copy-paste examples for all important FIT fixture types

Here are the slides from the talk. The video should be online at skillsmatter.com soon as well (subscribe to the RSS feed to get notified about that).

These are the links that I mentioned in the talk:

• Spring/Hibernate rollback example
• DbFit project pages on FitNesse.info

You might also be interested in checking out Scott Ambler’s agiledata.org site.

I started the whole project wanting to follow up on my earlier free PDF document on how to use FitNesse in a .NET environment. Instead of a quick and dirty guide, I wanted to create something that would give readers a more complete overview, present both the tool and important underlying practices and offer some ideas how to effectively use it in real world projects. I thought about giving away this as a free PDF as well, but then decided that I want to create a proper book which involved paying someone else at least to copy-edit it. So I decided to try out a commercial print to cover the production costs.

Print-on-demand is a new generation of printing services, where books are not issued in batches of a few thousand, but printed a copy at a time. Several on-line stores offer this service now, and I chose to use Lulu.Com because it seemed to be the most popular at the time. In addition to printing, they offer to sell the book on their web site and push it through their distribution channels to major on-line stores as well. The concept seemed interesting, especially since I worked as a technical editor on more than 20 books for a traditional publisher between ’99 and ’05. Back then, it was not worth it to print anything less than 1000 copies at a time, and the idea of having a single copy of the book printed and bound for a few pounds was so crazy that I had to try it out.

Lesson #1: It does not take a lot of money to publish a book, but it takes a lot of effort. In total, I have spent a bit more than £1000 to get the book published, most of which was for copy-editing. I could have saved a few hundred pounds I was not looking for a copy editor with a very specific skill set: reading and editing XML files directly. Because the copy-editor changed the source XML files directly, I did not have to manually enter changes from a paper copy or from an exported Word document, so this saved a lot of time for me.

On the other hand, getting a book from concept to cash takes a huge amount of effort. In total, I spent about nine months working a few hours per day on the book. I did not fill in any timesheets for myself, so I do not know the exact figure, but I estimate that I invested between 500 and 700 hours into this project. Half of that was writing the book, the other half was preparing the book for print and solving publishing problems.

Lesson #2: The book is a great marketing tool. I never expected that my book will be the next Harry Potter, so the commercial side of it was not really that important to me as long as I covered the costs. So far, the book sales have covered the production costs and I’ve earned a bit of money as well, if I don’t count my own time spent on the effort. However, In terms of contacts and contracts that I got because of the book, the experiment has been quite fruitful. So, the money in/money out balance of the whole project is hard to calculate. I like to believe that it was worth it :).

An additional bonus is the fact that I learned quite a lot while researching for the book. I started the project convinced that I knew all there is to know about TDD and FitNesse, but I wanted to present a balanced view on best practices so I made myself read up a lot of blog posts and articles on agile acceptance testing and TDD. That allowed me to gain a much deeper understanding of the matter and fill in a lot of gaps that I did not even suspect I had. It was truly an eye-opening experience.

Lesson #3: DocBook rocks! Since I was financing the whole thing, I wanted to keep the production costs as low as possible but get a solid result. I looked for ways to automate and optimise everything I could. DocBook is the programmer’s publishing system, generating the book PDF from a simple XML markup system. It allowed me to write chapters as plain text files (XML), which makes versioning and collaboration incredibly easy. The three biggest problems I can remember from my involvement with a traditional book publisher are tracking changes, consistency of cross-references and making sure that the source code for examples included in the book can actually run. DocBook solves all problems fantastically easy. My copy-editor changed XML files directly and I could use diff to quickly identify the changes. I used subversion to version the files. DocBook supports cross-references that are resolved when the book is compiled into PDF, so it automatically inserts the title and real page number of the target section for me. Code files, images and things like that can be included externally, so the book was built using C# files that were compiled and unit tested as well. DRY to the max!

DocBook is not without its flaws — for example, it would be great if the standard supported embedding parts of code files (so that I could insert a snippet and not the whole source file). Some XML tags are too complex, like the one for linking to an external image. Since DocBook works on plain text files, all those issues can easily be resolved by external shell scripts.

Lesson #4: I need commercial tools for this. I tried to do everything with opensource tools, using Docbook on Linux, with Xalan to convert Docbook-XSL to XSL:FO and and FOP to transform that to PDF. The tools did the job, but with too much pain. Docbook is a great idea, but FOP (0.94) broke so often and with such unusable error messages that I wasted hours on fixing trivial issues. NullPointerExceptions get thrown when stuff cannot fit on the same page, but nothing tells you what stuff or which page. This is especially hard to fix when individual chapters can compile correctly, but the whole book fails (eg because resolved links add a line to the page, that moves the image to the next page and then we get a NullPointerException).There a few commercial XSL-FO tools, but my budget for this book did not allow me to buy one. Next time, I’ll definitely get a commercial tool to do the job.

Lesson #5: Don’t trust PDF files. Unfortunately, “It works on my machine” syndrome applies publishing as well. When my book was ready for the first printing, I ordered a test copy, spent twenty days waiting for it and then tried to chase it through the printer’s customer service. It turned out that the PDF was unprintable on their machines and they could not give me any explanation why. I had already printed an earlier version of the book through the same printer and I had changed the fonts meanwhile, so changing the fonts again was my first idea how to fix things. Since I did not know which font caused the problem, I replaced all the fonts and re-submitted the PDF. This time, the book came out in a week. I thought that PDF is platform-independent and that if I can view and print it, it should be the same with the printer’s machines. Unfortunately, that was a false promise. The consequence of all this: about two months of delay and my code font was not nearly as good as I wanted it to be. Trying out another font would delay the print even longer, so I decided to go with this one.

Lesson #6: Leave a big margin for mistakes. This is probably what hurt me the most. On-demand publishing process is not necessarily repeatable. They use local printers to satisfy requests around the globe, and the test print which I approved in UK is not necessarily the same as the one that US customers will get. When I finally got something that printed OK with all the font changes, the front and back covers were not ideal — it would have been better with a bit more space between the letters and the cover edges, but I was nervous to get the book done and I decided to go with the covers as they were. They seemed OK, so I thought that the covers don’t have to be perfect and that I’d rather get the book finally published. A few weeks later, I got complaints from some US customers that some letters were cut out on their covers. The text and images on the US print covers were, apparently, displaced half an inch compared to the UK print covers. I changed the covers and re-submitted the book, which gave me a chance to fix some other minor issues with the text but delayed the distribution for another month.

Lesson #7: Everything takes much longer than you expect. I suppose that I got used to the speed of change in software. Even with full pre-press on my machine, any slightest change to the book takes at least a week, because the printer has to print and ship it and it has to arrive in the post. In case of problems, it can take even longer to find that out. Customer support at Lulu is very accessible, meaning that they are available on instant messaging 24/7, but I did not find them particularly helpful. A few times when I had real problems, like that font issue, it took them at least two days to respond and they did not offer a resolution to the problem. If you want Lulu to dispatch your book to Amazon and Barns and Noble, it takes them about a month to review it. If they have any objections, even though it takes you ten minutes to fix it, it still takes them another month to re-review the book and approve it for distribution. From that point, it takes a few months for the book to actually appear on online stores.

Although digital pre-press, working with PDFs and automated typesetting and layout speeds up the work and makes it much more agile then with a traditional process I have seen before, the time to see the result is still not negligible. Granted, a lot of errors and problems can be caught with the PDF, but I would not believe the PDF for the final confirmation that everything is OK. Just for reference: I had the text with initial layout finished in early November last year, but the book was officially announced and started selling on Lulu.com in mid-January. It appeared on Amazon in late April this year.

Image credits: Joana Croft

The event will be organised by Skills Matter, and it is free but registration is required. For more info, see:

http://skillsmatter.com/podcast/open-source-dot-net/effective-test-driven-database-development

See you on Thursday!

I use WordPress for my blog, and so far I am relatively satisfied with it. As a very popular online software package, it does get attacked a lot and security updates are released every once in a while. My site was hacked last year and the bastards dumped a bunch of hidden porn site links in about twenty articles, which took me a few days to clear up. So I learned the hard way that when the admin console suggests an upgrade, I should take its advice. I also added a cron task to check for a few keywords in the database and alert me if someone starts advertising limb enlargement devices for free, and since then I had no real problems. That is, until my site became a hot spot for south-east Asian smut aficionados overnight.

My first guess was that someone was simply spamming me with fake referrer headers, since there was absolutely no reason why my web site would actually appear in Google’s search results for adult movies, Korean or with a different geographic origin. Web sites use request referrer headers to identify where the visitors are coming from. A web browser will send the address of the site where you click on a link to the linked web site, if you have not turned that off. It is not a 100% reliable mechanism to identify visitor sources, as some people turn that feature off and some browsers have bugs and send rubbish, but in general it works OK. With the recent surge in the number of blogs, a new kind of spamming started to take place online. Spammers send fake requests to web sites, putting the address of the web site they are advertising into the referrer header. The rationale behind it is, I guess, to make the site owners to click on the referrer link to see who is sending people to their web site.

But there were quite a few of those requests, much more than with typical spam. The visitors were led to the search web page, and did not look at any other page after that, which could be explained by the fact that they were probably disappointed to find only clips of an ugly bald guy talking about agile acceptance testing instead of their favourite underground adult stars. However, with the web page they downloaded images, css and javascript files, which spammers typically don’t do. I did not know which article actually brought the unexpected guests since only the search page was affected. The database lookup did not help either — luckily this time it seemed that the site was not hacked.

I tried out the query on Google, just for fun, to be absolutely amazed that my site was the third on the list. Sure enough, my search page was there. I simply had to click on that to see what happens, and a few seconds later I was looking at a spam web site. My web logs showed a hit from Google again, but I was not looking at my site. Clicking on the “cached” link on Google led to the same outcome. I grabbed the page using wget, which definitely would not jump out directly, and there I found the words “korean underground adult movies”, but only after the “There are no results for…” phrase. More interesting, after that, there was a HTML image tag with “-1.com” as the source, and an onError event redirecting people to the spam web site. When the page loaded, the browser could not find -1.com to load the picture, and fired the onError event, sending the visitors from my web site to some place they could probably watch something more to their liking. Not a bad trick at all!

God knows how they got Google to index my web page with both their keywords and the redirection tag as a search phrase, but they did. And it’s not only my blog, there’s a few thousand other sites with the same problem. Search on google for “onerror freeimagew” to see them. The results containing </title> in the site name will probably redirect you automatically to the spam site.

The problem was that my blog just dumped out whatever people put into the search form when it could not find any relevant posts. The input string was properly sanitised before it was sent to the database, and WordPress generally cleans up all user submitted comments from hostile content, but it looks like they did not think of someone using the search form to hack the web site. In any case, I just changed the theme search.php file to print “Sorry, no posts matched your criteria” when there are no results, and that fixed the problem. A proper solution would be to strip out HTML tags from the search but I was too lazy to look for all the places where the phrase could be set.

In any case, this is one more example how important it is to filter and sanitise everything put in by web site users, regardless of how safe it may seem, and never ever printing it back on the web site without checking for potential problems.

Image credits: Sonja Gjenero