• Disabling tests to make a build pass:

  If the build is failing because of a test, developers disable the test to make the build pass. This is a bad practice because tests become irrelevant or get lost — people don’t remember to fix and enable them later. If the test is deprecated, then delete it completely. If it is not deprecated, don’t disable it but make it pass.

• Continuous integration not failing when unit tests fail:

  The point of continuous integration is to automate problem checking and prevent a big-bang integration before a release. Broken unit tests should raise an alarm and get fixed before problems pile up. If the CI server does not break the build when a unit test fails, then CI configuration must be changed.

• Not monitoring customer tests:

  Ian put integration and acceptance tests under the “customer tests” group. These tests don’t break the build because they will not pass for most of the development, but they still might go down from 30% to 20%, for example. That is a sure sign that something bad happened, yet if nobody is monitoring the reports, this will again lead to a big bang integration on the end. In my eyes, integration tests and customer tests should be split into two parts: the first one (integration) should break the build, and the second one (acceptance) should not, but it should still be monitored.

• UI change causes tests to fail (interface sensitivity):

  If tests depend on the UI heavily, then they will be brittle and hard to maintain. I wrote about this earlier in Effective User Interface Testing.

• (3rd party) API changes cause tests to fail:

  If changes to 3rd party APIs propagate to tests, then tests are again hard and expensive to maintain. I guess that the bigger underlying problem here is that the business logic is not isolated properly from 3rd party libraries.

• Many tests fail when a single behaviour changes:

  This applies to unit tests, and signals that tests are not properly granulated and focused on code units, but try to test too much. Again, the issue arising from this is high cost of test maintenance.

• Data-sensitive tests:

  Tests that depend on some data pre-conditions (such as certain records existing in the database) are also brittle and will break when the data changes. Test harness should ideally set up all the pre-conditions for a test. A telling sign of this are tests that use hard-coded database IDs.

• (Shared) Context Sensitivity:

  If tests depend on other tests to set up the context, then the order of test execution becomes important and you can no longer run individual tests in isolation. This can lead to big problems, especially if the test runner does not guarantee the order of tests. Again, the test harness should ideally set up all the pre-conditions for a test and individual tests should be independent.

• Conditional test logic:

  The telling sign of this problem is that tests choose validations based on run-time context (if (…) test this… else test that…). This signals that the test is not clearly focused on a single thing, and that the author does not really understand what he is testing.

A quick summary

Tests that break without anyone reacting do not prevent problems from piling up. This defeats the point of tests being a traffic light that keeps the problems small and prevents a big-bang integration on the end. When tests fail, alarm bells should ring.

High maintenance cost of tests defeats the whole point of having them, as a way to guarantee that code changes are easy and cost-effective. Keep unit tests independent and focused on a single code unit. Then they will be easy to maintain and will support change rather then inhibit it.

Image credits:Lorenzo González

The importance of a common language is nothing new or specific to software projects. The western civilisation has been learning that complex projects have no chance of success without a common language at least for the last two thousand years, but it seems that we have still not taken that message seriously. In the software world, the language of projects has been getting much more attention recently, especially due to Eric Evans and his book Domain-Driven Design: Tackling Complexity in the Heart of Software. Evans coined the phrase Ubiquitous language to describe the best practice to fight different jargons on project teams. Using an ubiquitous language requires business analysts, customers and programmers to all have the same names for the same things. But getting everyone to use common phrases is just one part of the puzzle.

I have stumbled upon a few very interesting conclusions about the language and how important it is to choose good metaphors and names in Douglas Hofstadter’s book Gödel, Escher, Bach. That title has intrigued me for a few years, but as a quick look on Amazon told me that is has almost 800 pages, it had to wait for me to change my daily routine before I could even start reading it. The last few months have been much more quiet than usual, so I finally had some time to dive in. Although the book deals with something completely unrelated to software projects, it contains some very interesting ideas that we can apply to improve software development. I especially like the thoughts on how humans understand formal systems and how the meanings of abstractions form in our minds.

Before this starts to sound too philosophical, let me change the direction of this story. I was recently involved in a project where the common language was chosen and agreed upon, but caused a lot more confusion then I expected. Even with the definitions written down and published on a wiki site, a lot of time was wasted on misunderstandings of basic concepts and going back to explain the same things over and over again. The phrases of that common language were OK from the aspect that they made sense, but they were a compromise between business jargon and technical design and somehow fell short of pleasing anyone. It was a strange situation because all the people involved were techies, and I expected that people with technical education should generally be OK with abstracting things a bit, understanding and using formal definitions even if they do not necessarily bear the same meaning as in common everyday language. But that has not worked out as expected. I had a feeling of what went wrong, but I did not have strong arguments to support that opinion. Hofstadter, however, provides a very nice explanation for that on pages 90 and 91:

… Therefore, when someone gives a definition for a common word in the hopes that we will abide by that definition, it is a foregone conclusion that we will not do so, but will instead be guided, largely unconsciously, by what our minds find in their associative stories…

In short, redefining a common phrase is not going to work. Whether we like it or not, our understanding of a common concept gets in the way of applying it in a different context. “Gödel, Escher, Bach” offers some very interesting examples how attempting to impose definitions of common phrases and words was a very hard thing to do throughout history, starting with that Euclid’s Elements and his attempt to specify what “point”, “straight line” and “circle” mean.

On page 94, Hofstadter explains another interesting idea:

…The symbols automatically pick up passive meanings in accordance with the theorems they occur in…

So regardless of the actual name used for something, our mind gives it a passive meaning depending on the way in which it is used and cooperates with other concepts. If a name given to some concept does not match the way it is being used, we will effectively attach a different name and a meaning to it. The way things are used dictates how we perceive them, regardless of the names that were enforced for those things.

Armed with this new insight, here are some ideas that I will definitely try on my next project:

— To name a concept, start from the way it will be used and try to find a intuitive match for that.

— Avoid generic names such as “Controller”, “Manager” or “Content”. They already carry too much ambiguity.

— For all common phrases from the business or technical jargon that have been selected for the ubiquitous language, double-check that there are no other related meanings that could cause confusion.

Image credits: Dog Madic

I recently had a very unpleasant talk with one of our support developers. She was very annoyed, in a defensive mood, arguing that she is already too busy and cannot take on a bit more work. After twenty minutes of talking, she finally calmed down, and we came to the root of the problem. It’s not the programming part of her work that is killing her, but the fact that she has to go through several terminal servers to deploy changes to client environments, and that she has to merge all her changes into several revisions. And she does that for every single change, several times a day. So, in fact, she spent most of her time logging on to different computers and copying files between them. Although she understood the problem herself, the thought of automating those silly tasks never came to her mind.

It’s incredible how, for a profession based around telling machines how to perform dull and error-prone tasks instead of humans, so few programmers think about how to automate their own work. This is especially noticeable with the “Windows generation”, people who never effectively used the command line. GUI programs are, without a doubt, more user friendly and easier to use, but they are bloody hard to automate. When a task has to be done fifty times, or performed across the network, words “ease of use” take on a completely different meaning. That is the point when good programmers break from the usual practices and find better ways to do the job, and bad programmers stick to their old habits.

Breaking from the old habits

A few years ago, I did some consultancy work for a company whose application generated huge log files on clustered servers. To find the cause of any problem, programmers and support staff transferred gigabytes of data from client’s web servers to their computers in order to analyse them. Each transfer took at least 10 to 15 minutes. There was no special reason for pulling all the log files to local machines, people were just used to doing their job like that. I was even more amazed to see that even some “senior” programmers in that organization repeatedly (tried) to open monstrous log files in Notepad and than wasted the next 20 minutes waiting for their machine to unfreeze. All that just to find a few lines of text, which could have been done in a minute or two using findstr on the server.

When a solution clearly does not scale to the problem, we look for a better one, or a different way to implement it. The solution often requires using a different approach, or different tools. The same holds for non-programming tasks. We must look for better ways to perform daily tasks when the usual tricks don’t work. In this case, the proper solution was to use the command line. Programs like tail and grep were developed at a time when network connections were bad, and computing power was scarce, so they don’t require a lot of typing and clicking. Although available bandwidth and computing power have improved greatly over the last 20 years, such tools are still irreplaceable for processing large text files, especially on remote computers. Sadly, I meet more and more developers who never heard of those tools. If you are reading these lines and thinking to yourself “what the heck is grep“, here is just a brief introduction to some common tasks that these tools do really, really well:

grep

• get all lines containing a string (or a regular expression) from a large file – for example, get all log items for a particular session or user
• count the number of occurrences of a string (or regular expression) in a file – for example, find how many times someone has logged yesterday
• find all files that contain a string (or regular expression) – for example, list log files that contain a particular username

more

• view the start of a large file – for example, see the IIS log file format which prints column headers in the first line

tail

• quickly view the last few lines of a large file – for example, check out the last exception
• monitor how a log file is changing while other processes are writing into it – great for troubleshooting a live application, because it does not lock the file

awk

• extract lines from a large file and reformat them – for example build URLs by appending fields from a tab-separated log file to the site prefix, or create a block of SQL statements to populate tables from some data export file
• calculate statistics on file content – find top 20 client IPs by web bandwidth used and print the number of requests and total bandwidth used by each one
• split a file into several other files, based on line content – for example, extract multi-line exceptions from a log file
• quickly analyse a comma-separated, tab-separated or any other file format with multiple fields in a line

sed

• “replace all” for large files and sets of files

Working on Windows is often an excuse for not using command-line tools, but that is just an excuse. There are several free sets of tools which bring power to the Windows command line. For example, Cygwin package allows most popular Unix tools to run from Windows. Windows Resource Kit from Microsoft contains tail and qgrep (a fairly usable variant of grep). PsTools, another package now owned by Microsoft, contains gems like psexec that can run a program on a remote computer. Some tools, like GNU Awk have been directly ported to Windows by the GNU foundation.

On any variant of Unix, you can expect these tools to be already installed. Some trimmed-down replacements exist even for the standard Windows command line (like findstr which can be used instead of grep if you don’t look for regular expressions). However, most of these tools can just be copied to a remote machine and started (no need to “install” them), so you don’t require any special privileges to run them on client’s web servers, for example. You can just drop them on a remote computer and start using them. So don’t even try pulling the entire log file from the customers web site just to do a simple search: copy gawk there and process the log on the server.

In addition to being very efficient, these tools can easily be automated (scripted). It does not really matter whether shell scripts, batch files or some scripting language like vbscript, python or perl is used. Storing a grep or awk command line along with all the strange parameters will save you from the pain of writing them again tomorrow.

Three strikes, and you are out

“Don’t repeat yourself” is one of fundamental guidelines of Extreme Programming. Most good programmers today take care not to repeat the code, the ones that are even better find similarities in design and consolidate them, but very few apply the same rule to their habits or non-coding related work. I’m not just talking about half-hour tasks like sorting e-mail and analysing debug logs, but also small jobs like checking whether the client database is on-line or replacing seven occurrences of a field name in a script.

Apart from saving time, using replace all will also prevent errors, but you have to test it first before updating a 200 MB file on a client’s environment. The same tasks have to be applied for replacement in a 5 KB script file. There is always some overhead involved in automating the tasks, but when does it begin to pay off, and when is it unjustified? The rule I stick to is Roberts’ rule of Three (from Fowler’s Refactoring: Improving the Design of Existing Code), another version of DRY:

The Rule of Three by Don Roberts

The first time you do something you just do it. The second time you do something similar, you wince at the duplication, but you do the duplicate thing anyway. The third time you do something similar, you refactor.

This is also a good rule of thumb for tasks outside of coding. Read it like this: If you catch yourself doing something for the third time, chances are that you will have to do it again, so automate it. Use the replace all if you have to replace the same string more than two times. More importantly, stick to the rule when things get complicated! If you are not dealing with plain strings, but have to re-format the data a bit, don’t fall back to manual work: utilise regular expressions (if you don’t know how to use regular expressions, now is the time to learn). If this is the third time this week that you are doing the same replacement, don’t use the regular expressions manually, but write a script. When you catch yourself running that script every day (week, month) add it to the scheduled tasks list (or cron jobs).

Automating tasks is especially important when your client has to perform them. Having a nice automated installer instead of a two-page word document explaining all the steps required to set up the software goes a long way. It makes the task less error-prone so the customers will be happier, and it leaves us with more time for the fun stuff.

Image credits: D Lee/SXC