I recently had a very unpleasant talk with one of our support developers. She was very annoyed, in a defensive mood, arguing that she is already too busy and cannot take on a bit more work. After twenty minutes of talking, she finally calmed down, and we came to the root of the problem. It’s not the programming part of her work that is killing her, but the fact that she has to go through several terminal servers to deploy changes to client environments, and that she has to merge all her changes into several revisions. And she does that for every single change, several times a day. So, in fact, she spent most of her time logging on to different computers and copying files between them. Although she understood the problem herself, the thought of automating those silly tasks never came to her mind.

It’s incredible how, for a profession based around telling machines how to perform dull and error-prone tasks instead of humans, so few programmers think about how to automate their own work. This is especially noticeable with the “Windows generation”, people who never effectively used the command line. GUI programs are, without a doubt, more user friendly and easier to use, but they are bloody hard to automate. When a task has to be done fifty times, or performed across the network, words “ease of use” take on a completely different meaning. That is the point when good programmers break from the usual practices and find better ways to do the job, and bad programmers stick to their old habits.

Breaking from the old habits

A few years ago, I did some consultancy work for a company whose application generated huge log files on clustered servers. To find the cause of any problem, programmers and support staff transferred gigabytes of data from client’s web servers to their computers in order to analyse them. Each transfer took at least 10 to 15 minutes. There was no special reason for pulling all the log files to local machines, people were just used to doing their job like that. I was even more amazed to see that even some “senior” programmers in that organization repeatedly (tried) to open monstrous log files in Notepad and than wasted the next 20 minutes waiting for their machine to unfreeze. All that just to find a few lines of text, which could have been done in a minute or two using findstr on the server.

When a solution clearly does not scale to the problem, we look for a better one, or a different way to implement it. The solution often requires using a different approach, or different tools. The same holds for non-programming tasks. We must look for better ways to perform daily tasks when the usual tricks don’t work. In this case, the proper solution was to use the command line. Programs like tail and grep were developed at a time when network connections were bad, and computing power was scarce, so they don’t require a lot of typing and clicking. Although available bandwidth and computing power have improved greatly over the last 20 years, such tools are still irreplaceable for processing large text files, especially on remote computers. Sadly, I meet more and more developers who never heard of those tools. If you are reading these lines and thinking to yourself “what the heck is grep“, here is just a brief introduction to some common tasks that these tools do really, really well:

grep

• get all lines containing a string (or a regular expression) from a large file – for example, get all log items for a particular session or user
• count the number of occurrences of a string (or regular expression) in a file – for example, find how many times someone has logged yesterday
• find all files that contain a string (or regular expression) – for example, list log files that contain a particular username

more

• view the start of a large file – for example, see the IIS log file format which prints column headers in the first line

tail

• quickly view the last few lines of a large file – for example, check out the last exception
• monitor how a log file is changing while other processes are writing into it – great for troubleshooting a live application, because it does not lock the file

awk

• extract lines from a large file and reformat them – for example build URLs by appending fields from a tab-separated log file to the site prefix, or create a block of SQL statements to populate tables from some data export file
• calculate statistics on file content – find top 20 client IPs by web bandwidth used and print the number of requests and total bandwidth used by each one
• split a file into several other files, based on line content – for example, extract multi-line exceptions from a log file
• quickly analyse a comma-separated, tab-separated or any other file format with multiple fields in a line

sed

• “replace all” for large files and sets of files

Working on Windows is often an excuse for not using command-line tools, but that is just an excuse. There are several free sets of tools which bring power to the Windows command line. For example, Cygwin package allows most popular Unix tools to run from Windows. Windows Resource Kit from Microsoft contains tail and qgrep (a fairly usable variant of grep). PsTools, another package now owned by Microsoft, contains gems like psexec that can run a program on a remote computer. Some tools, like GNU Awk have been directly ported to Windows by the GNU foundation.

On any variant of Unix, you can expect these tools to be already installed. Some trimmed-down replacements exist even for the standard Windows command line (like findstr which can be used instead of grep if you don’t look for regular expressions). However, most of these tools can just be copied to a remote machine and started (no need to “install” them), so you don’t require any special privileges to run them on client’s web servers, for example. You can just drop them on a remote computer and start using them. So don’t even try pulling the entire log file from the customers web site just to do a simple search: copy gawk there and process the log on the server.

In addition to being very efficient, these tools can easily be automated (scripted). It does not really matter whether shell scripts, batch files or some scripting language like vbscript, python or perl is used. Storing a grep or awk command line along with all the strange parameters will save you from the pain of writing them again tomorrow.

Three strikes, and you are out

“Don’t repeat yourself” is one of fundamental guidelines of Extreme Programming. Most good programmers today take care not to repeat the code, the ones that are even better find similarities in design and consolidate them, but very few apply the same rule to their habits or non-coding related work. I’m not just talking about half-hour tasks like sorting e-mail and analysing debug logs, but also small jobs like checking whether the client database is on-line or replacing seven occurrences of a field name in a script.

Apart from saving time, using replace all will also prevent errors, but you have to test it first before updating a 200 MB file on a client’s environment. The same tasks have to be applied for replacement in a 5 KB script file. There is always some overhead involved in automating the tasks, but when does it begin to pay off, and when is it unjustified? The rule I stick to is Roberts’ rule of Three (from Fowler’s Refactoring: Improving the Design of Existing Code), another version of DRY:

The Rule of Three by Don Roberts

The first time you do something you just do it. The second time you do something similar, you wince at the duplication, but you do the duplicate thing anyway. The third time you do something similar, you refactor.

This is also a good rule of thumb for tasks outside of coding. Read it like this: If you catch yourself doing something for the third time, chances are that you will have to do it again, so automate it. Use the replace all if you have to replace the same string more than two times. More importantly, stick to the rule when things get complicated! If you are not dealing with plain strings, but have to re-format the data a bit, don’t fall back to manual work: utilise regular expressions (if you don’t know how to use regular expressions, now is the time to learn). If this is the third time this week that you are doing the same replacement, don’t use the regular expressions manually, but write a script. When you catch yourself running that script every day (week, month) add it to the scheduled tasks list (or cron jobs).

Automating tasks is especially important when your client has to perform them. Having a nice automated installer instead of a two-page word document explaining all the steps required to set up the software goes a long way. It makes the task less error-prone so the customers will be happier, and it leaves us with more time for the fun stuff.

Image credits: D Lee/SXC

Selenium is a free browser automation and testing library, written by folks at ThoughtWorks. It can simulate text input, mouse events and execute various tests on page content. It’s written in JavaScript, and is compatible with all major browsers and platforms.

FitNesse is an open-source test and collaboration server, based on the Framework for integrated tests (FIT), and supports testing Java, .Net, Python and even some other code. I think that it is a good choice for the second side of the web UI testing coin, because it enables tests to be written almost like in English language. As the UI is very close to clients’ eyes, tests can and should be written so that clients can verify them (and if you are really lucky, even help with writing and maintaining the tests).

UPDATE: I wrote a follow-up to this article in Feb. 2008, focusing on testing AJAX applications. Read it on http://gojko.net/2008/02/14/ajax-selenium-fitnesse

Installing Selenium

The core of Selenium is written in JavaScript, and is compatible with all major browsers. The Selenium Remote Console provides the glue between a test browser and .Net, Java or Python code, and enables us to write and run Selenium tests from almost all popular test frameworks.

Just download Selenium Remote Console from http://www.openqa.org/selenium-rc. Unpack the files somewhere on your disk, and then start the server from the server folder of the package, by executing java -jar selenium-server.jar. The server starts on port 4444 by default, if that port is taken you can change it by adding -port number to the command line.

Selenium works by embedding the test site, along with some control scripts, in a frameset web page. The control scripts are then used to automate the test page in the main frame, and execute various tests. That method of work will clash with browser security rules if both frames do not come from the same domain. Remote console can open Internet Explorer and Firefox with a security tweak to allow cross-domain scripting, but if you don’t want that, or want to use a different browser, then put Selenium JavaScript files into your web site. Download them from http://www.openqa.org/selenium-core/, and put the files from core folder in the archive into the selenium-server folder of your web site (so that RemoteRunner.html is available on /selenium-server/RemoteRunner.html).

A quick test

Let’s first do a quick test to make sure that Selenium RC is running correctly. We’ll start a browser window, go to Google, enter a search phrase and click on the Search button. I’ll do the examples in C#, but you can also use Java – interfaces are very similar. Create a new .Net project, add thoughtworks.selenium.core.dll as a reference (it’s in the dotnet folder of Remote Control installation), and then create this class:

using System;
using Selenium;
namespace SeleniumTest
{
    class Console
    {
        static void Main(string[] args)
        {
            ISelenium sel = new DefaultSelenium(
              "localhost", 4444, "*iehta", "http://www.google.com");
            sel.Start();
            sel.Open("http://www.google.com/");
            sel.Type("q", "FitNesse");
            sel.Click("btnG");
            sel.WaitForPageToLoad("3000");
        }
    }
}

Make sure that Selenium RC is running, close all open Internet explorer windows, and then execute the program. Selenium RC will open a new IE window (it may be minimised on start, but you will see a new button in the task bar), and then go to Google and execute a search.

click for full-size image

Selenium window has three frames. Top-left enables the user to display the log and debug the execution by examining the DOM tree. Top-right frame displays the last four executed Selenium commands, which comes useful when troubleshooting tests. The page under test is displayed in the central frame.

The first line initialises a selenium console instance, connecting it to a remote console on localhost port 4444.

ISelenium sel = new DefaultSelenium(
"localhost", 4444, "*iehta", "http://www.google.com");

The third argument is a browser string, and *iehta is the code for security-tweaked Internet Explorer which allows cross-domain scripting. Use *chrome for firefox with cross-domain scripting, or *iexplore, *firefox and *opera for Internet Explorer, Firefox and Opera without cross-domain security tweaks. You can also specify a full path to the executable instead of these special keywords.

The fourth argument is the URL to the selenium test site, and is really important only if you do not use cross-domain scripting, but want Selenium to run the scripts from the local domain. In this case, it will try to get the remote runner from http://www.google.com/selenium-server/RemoteRunner.html, but since Google is not hosting a Selenium installation, it will run the local scripts instead. In any case, this argument needs to be specified, so just set it to some live site even if you use cross-domain scripting (like in this case).

As you can guess from the names, method Type will simulate entering data from keyboard into a text field, and Click will simulate a mouse click. We accessed the text field and the button by their names, q and btnG. Selenium can also find elements by their ID (prefix it with “identifier=”), XPath expression (prefix it with “xpath=”), DOM path or a CSS property. See http://www.openqa.org/selenium-core/reference.html for more information on locators.

Connecting from FitNesse

FitNesse is really great for automating acceptance tests, and if you have not had a chance to play with it yet, I suggest reading through the online user guide for testing Java code, or my tutorial Getting fit with .Net for working in the .Net environment.

Instead of writing a new fixture type for every page or web site, we use a generic WebTest fixture. It extends DoFixture, so that other fixtures can be easily embedded into it (for checking the back-end data after a web script, or preparing the stage for the web test). Web test fixture allows the scripts to be described almost like English prose, like this:

User opens URL http://localhost:7711/login.aspx
User types testuser into username field
User types testpassword into password field
User clicks on Log In
Page reloads in less than 3 seconds
Page contains text You have logged in

To achieve the best effect, the script must relate to what users see on the screen. However, that is easier said than done. Text fields, check boxes and buttons are all instances of the “input” element, and although text fields are mostly distinguished by name, buttons names are typically not important, and users only see the “value” attribute of a button. WebTest fixture looks for various combinations of attributes: for example, when searching for a button, it first looks for an input element with type equal to submit or button, and the name attribute matching the query. If no such element is found, it looks for similar elements, with value attribute matching the query. On the end, it looks for a button with matching ID. All these combinations are listed in the buttonLocators array in the code:

 public static readonly string[] buttonLocators = new String[] {
	"xpath=//input[@type='submit' and @name='{0}']",
	"xpath=//input[@type='button' and @name='{0}']",
	"xpath=//input[@type='submit' and @value='{0}']",
	"xpath=//input[@type='button' and @value='{0}']",
	"xpath=//input[@type='submit' and @id='{0}']",
	"xpath=//input[@type='button' and @id='{0}']"};

Although looking for an element by ID or name would be quicker, we intentionally use Xpath and add extra information, to catch errors caused by wrong element types. GetLocator method checks a list of XPath expressions and returns the first matching element from the active document, so UserClicksOn method is very simple:

public void UserClicksOn(String buttonCaption){
            instance.Click(GetLocator(buttonCaption, buttonLocators));
}

We use the same technique to put text into text fields, areas and password fields. An array of Xpath expressions is used to check for various combinations of valid element types and attribute values, and the first matching element is then passed to Selenium Type method.

public void UserTypesIntoField(String what, String where){
      instance.Type(GetLocator(where.Replace(" ", ""), textFieldLocators), what);
}

Because text fields will mostly be referenced by a name or ID, but users will see the text before the field on the page (and probably try to reference it with that), we also strip the blanks from the name. So the test script line:

User types 10109 into security number field

will correctly map into an element named “securitynumber”.

With web tests, we typically want to check if the result was correct and if the site was responsive enough, and Selenium method WaitForPageToLoad can be used to check if a page loads in any given amount of time. The API is a bit weird, as it expects a string containing the number of milliseconds. We wrapped that into this WebTest fixture call:

public void PageReloadsInLessThanSeconds(String sec){
            instance.WaitForPageToLoad(sec + "000");
}

On the end, we need a method to verify that page contents are correct. Selenium method IsTextPresent can help with that:

public bool PageContainsText(String s){
            return instance.IsTextPresent(s);
}

We’ll also need to open a browser and set up the Selenium environment from the test fixture. To keep the test script format in english-like prose, we’ll use the following syntax:

Start Browser *iehta with selenium console on localhost at port 4444 and scripts at http://localhost:7711

In WebTest fixture, that is covered by the following method:

public void StartBrowserWithSeleniumConsoleOnAtPortAndScriptsAt
   (String browser, String rcServer, int rcPort, String seleniumURL){
            instance = new DefaultSelenium(
                 rcServer, rcPort, browser, seleniumURL);
            instance.Start();
}

On the end of each test, we need to shut down the browser, so that remote console does not run out of resources. For that, we’ll use the following method:

public void ShutdownBrowser(){
            instance.Stop();
}

With all that, the web test can be easily described with this FitSesse script:

!|webfixture.WebTest|

!|Start Browser|*iehta|With Selenium Console On| localhost| At Port |4444|And Scripts At|http://localhost:7711|

|User Opens URL|http://localhost:7711/login.aspx|
|User types|testuser|into|username|field|
|User types|testpassword|into|password|field|

|User clicks on|Log In|
|Page reloads in less than|3|seconds|
|Page contains text|You have logged in|

|Shutdown browser| 

Ajax testing

With web 2.0 pages, waiting for the page to reload before we continue testing is not the best choice. Selenium also supports
waiting for a JavaScript condition. Remember that Selenium script is being executed in a different frame, so referencing DOM elements and JS functions in your test site page will not work straight away. Prefix the references with
selenium.browserbot.getCurrentWindow() to get from the Selenium frame to your test page. Here’s an example:

instance.WaitForCondition(
  "(selenium.browserbot.getCurrentWindow().get_username()!=null)",
  timeout);

Remote execution

Selenium Remote Console will try to open a new browser session with different security settings, but under the profile of the current user (in fact, the user which started the remote console). This may cause problems when you already have an open browser on the same machine. The workaround is to use a different browser for testing. For example, I use Firefox for normal browsing, so RC can start IE and play with it. But, there is a much better solution.

As the name suggests, Remote Console can run on a dedicated test server, and be accessed remotely. Any browsers that RC opens will run on that machine. That enables developers to work with browsers on their machines and run the tests, without clashing with test execution. It also enables us to re-use the same FitNesse scripts to check how various browsers are behaving, even on different platforms. Just separate the initialisation and browser shut-down into SetUp and TearDown scripts, and then create a couple of different test suites with different SetUps which will use various browsers and remote console installations. Then, use symbolic links to import scripts from one test suite into all others. You will maintain scripts in one place, and they will test multiple browsers and platforms.

Extending WebTest

WebTest currently supports two more interesting methods: PageURLIs(String s) and UserSelectsFrom(String what, String where). The first allows you to verify the page URL (and test redirections). The second automates selecting an option from a drop-down menu (HTML select element), in the form of User Selects NY from Countries.

Feel free to modify and adjust WebTest to your needs. For start, see dotnet/doc/index.html in your remote control installation for more information on ISelenium interface. You can a add search by ID to the top of all locator arrays, so that tests run faster, but that will make them more error-prone. You will probably want to add a few different expressions to the textbox, button or select lists. If you add something which may be useful to others, please share it by writing a comment on the WebTest fixture page.

Photo credits:Sanja Gjenero/SXC