Chris McMahon and Marisa Seal are now the new mummy and daddy of the project. You can read what Chris had to say about it. Good luck!

Quick note before we continue: FIT fixtures described in this article can be downloaded from fitnesse.info/webtest. Both Java and .NET are supported and the fixtures are released under GPL. To learn how to set everything up, read the original article.

Effective AJAX testing with selenium.Wait

In the original article, I have suggested using WaitForCondition and selenium.browserbot.getCurrentWindow() to dynamically evaluate JavaScript expressions and block until they become fulfilled. Although this did the trick, the code was very ugly and error-prone, mixing JavaScript evaluations with Java/.NET expressions. It turned out that it was much better to poll browser a few times per second through Selenium RC API. This way, the code is much more consistent, and waiting conditions are much easier to write. Polling, in theory, causes unnecessary delay, but this case Selenium RC and UI testing itself introduce a lot of latency, so the polling does not really make things any worse. In fact, the Java version of Selenium Remote Control has API support for that: Wait class in com.thoughtworks.selenium package. The official JavaDoc for that class is a bit wrong and misleading, but with a bit of experimenting that class is very easy to use, and quite powerful. The official documentation suggests this usage pattern:

new Wait("Couldn't find close button!") {
    boolean until() {
        return selenium.isElementPresent("button_Close");
    }
};

Yet this code will not even compile with Selenium RC 9.2, because no such constructor exists. Instead, the error message should be passed while calling the blocking wait method later. Here is an example that blocks until some text appears on the page:

  // private class of WebTest fixture
  private class WaitForTextToAppear extends Wait{
    private String text;
    public  WaitForTextToAppear(String  text){
      this.text= text;
    }
    public boolean until(){
       return WebTest.this.pageContainsText(text);
    }
  }
 // method of WebTest Fixture
 public boolean waitSecondsForTextToAppear(int timeout, String text){
   Wait x=new WaitForTextToAppear(text);
   x.wait("Cannot find text " +text+ " after "+timeout+" seconds",timeout*1000);
   return true;
 }

Waiting In mini-language

In FitNesse, this would be used as:

|wait|5|seconds for text|Hello World|to appear|

This line would block the execution of the page up to five seconds, waiting for Hello World to appear anywhere in the page. If that text does not appear after five seconds, the test breaks. Using this pattern, it is very easy to implement any kind of asynchronous waiting without manually checking for JavaScript expressions and worrying about Selenium-to-application page DOM conversions. So far, I have implemented the following waiting methods for the testing mini-language:

|wait|5|seconds for element|username|to appear|

|wait|5|seconds for element|username|to disappear|

|wait|5|seconds for text|Hello World|to appear|

|wait|5|seconds for text|Hello World|to disappear|

|wait|5|seconds for field|username|to contain text|Hello World|

|wait|5|seconds for element|username_label|to contain text|Please enter username|

the difference between “wait for field to contain text” and “wait for elelement to contain text” is that the first method checks for the current value of an INPUT field (<input type=”text” name=”username” value=”Hello World”/>), and the second checks for text inside a dom element (<span id=”username_label”>Please enter username</span>). .NET Selenium RC API does not have this class, but it was not hard to implement, so the .NET fixture also supports this in the mini-language.

Locator lookup takes too long

Selenium uses “locators” to point to DOM elements that you want to automate. In the original article, I suggested a complex scheme of mapping user-friendly descriptions to locators so that we can use button captions, labels and similar visible text and labels to point to page elements. Under this scheme, the FitNesse-Selenium glue code tried out locators from an array until it found a match. This turned out to be such a performance penalty that I no longer suggest it. Each attempt to discover whether an element was present or not went through the full cycle of FitNesse, Selenium Remote Console, Browser, Selenium and back. In average, that increased the time for a UI test to execute by 500-600 percent. It turned out that, with a bit of care in naming, DOM element IDs can be used in all cases so that test pages are still descriptive enough, and indirect locator lookup can be avoided. So now I suggest using DOM element IDs directly, not using labels or captions.

Effective locators in mini-language

I have introduced a new parameter to control whether the lookup is performed or not. It is still on by default, but if you put this table into your page:

|set locator lookup|false|

tests will execute much much faster. This will require you, however, to use DOM element IDs exclusively to point to all page elements. Even with lookup turned on, tests should now run faster if you use DOM element IDs, because I’ve moved IDs to the top of lookup arrays for all element types.

Element type safety is not really important

In the original article, I suggested checking whether something is a button, input field or a link when doing methods like click, so that something like:

|user clicks on|save|button|

would fail if save was a link identifier. I thought that would introduce one more level of safety and avoid problems because of ambiguous definitions. Since trying out all those locators works really slow, I gave up on checking for the type of element, and use that only in special cases (such as radio buttons, which should be selectable by value as well as name). This did not, in practice, make tests any less effective. So I now suggest avoiding such complex checks.

Direct element access in mini-language

The WebFixture mini-language now supports a bunch of methods for generic elements, which are located using the DOM element ID or element name, without checking the exact type. So you can write something like:

|user clicks on|save|

and that is going to work for buttons, labels, text elements and anything you can click on.

Don’t pause, wait for specific events

In the original article, I introduced a few methods that blocked until the whole page loaded correctly. With AJAX testing, those methods are no longer applicable. To allow an asynchronous operation to complete before continuing with the test, lots of people used the ‘pause’ function. This caused more harm than good, so I’m now thinking of effectively removing that function or throwing an exception on the end of the pause to print a warning that the test should be rewritten. Pause is makes tests error prone and very brittle. If, for example, you pause for 5 seconds to wait for a form to load, then when you run the tests on different hardware or over network, the test itself might break because of longer latency (so the code implementation is still correct, but the test fails, which is a very dangerous anti-pattern). This gets solved by extending the pause to be long enough, so your tests are no longer running for 10 seconds but a minute… once the test suite starts running for longer than a minute people are no longer running tests before committing, and that makes the feedback loop even longer, reducing the effectiveness of tests.

It’s much better, if you can, to block and wait for a specific event. So instead of pausing, use one of the wait functions as described in the beginning of this article. Those functions still allow you to specify the ultimate timeout period, so that if the event does not happen after 10-15 seconds the test breaks. A very important difference is that, in normal case, the execution will continue straight after the event does happen. So you’ll wait exactly as long as you need to, and not a second longer.

DOM events don’t always fire automatically

When you type stuff in with Selenium, DOM elements such as onFocus, onBlur and onChange don’t fire automatically (not to mention onKeyUp and onKeyDown). Lots of AJAX code is built around those events, and tests fail if the code relies on events. Selenium allows you to triggers the events manually using fireEvent method. So, if your site behaves differently under test and when you play with it manually, missing events should be your first suspect.

Firing events manually in the mini-language

I have changed the UserTypesIntoField method to fire events as well, in the appropriate order. You can fire an event explicitly using the fire event for method:

|fire event|blur|for|username|


To fire events, don’t put the leading on — use click instead of onclick.

Some people prefer click-and-record

Although I’ve tried to introduce a customer-friendly mini-language for testing, some people liked the idea of using FitNesse to wrap Selenium tests more because it UI tests could then be integrated into automated integration testing and because they could prepare and verify their database during Selenium tests. But they still preferred using Selenium IDE to record tests, without having to manually write them. I have introduced a new fixture class, webfixture.PlainSeleniumTest, that more or less just forwards the commands to Selenium RC. So you can use a Selenium test table, paste it in FitNesse, change it just a bit, and run the test using Selenium Remote Control. The only changes required are deleting the test name (first row of the table) and optionally changing the open command to include a relative URL instead of an absolute one. That allows you to record tests using Selenium IDE, but still integrate them into Cruise Control and control the database or talk to your domain objects during tests.

Selenium tests can be stored as HTML files, and you can use HTML directly in FitNesse without converting it to the Wiki syntax. Just paste the table anywhere in the page, but put the HTML code between !- and -! to tell FitNesse to display the contents raw, without special formatting.

Originally, I thought that I could just get away by wrapping a Selenium RC instance into a Sequence Fixture as the system under test, and pass all commands to it. But Selenium RC API and browser-based Selenium scripts differ in some parts, especially for the waiting methods and verification methods. For example, verifyVisible in Selenium is called isVisible in Selenium RC. So I wrote a bunch of wrappers for verifyXXX and assertXXX methods to make tests run without changing.

The next problem was that the resulting Selenium test tables may have one additional empty cell if the method only has one argument, such as open url. Because of that additional argument, FitNesse will try to map the row to the method open(String, String) and fail because it could not find such a method. Because of that, I also wrote a bunch of wrappers for single-parameter methods that have a second string argument, which is just ignored. For example:

public void open (String url, String ignore){
  open(url);
}
public void click(String what, String ignore){
  instance.click(what);
}

With that, Selenium tests started working more or less out of the box. I occasionally have to add another method for one of the reasons mentioned above, but that is fairly quick and straightforward.

There’s one more trick related to this. Selenium scripts, especially those recorded, can be quite lengthy. FitNesse will flush output after each table, so if you leave the test as one big table, it will look like FitNesse is not doing anything for a while and then you’ll get all results at once. I’ve found it much better to just split the big table into several smaller tables, so that I can track progress while the test is running.

Use browser names instead of browser codes

Cory Foy suggested that we should not use browser codes like *iehta, but make tests even more user friendly by using descriptive browser names. I like that idea very much, so the start browser now supports both codes and names like IE and Firefox. It will map the name to the appropriate browser code for you.

Plenty of new methods

I have also added quite a few new methods to the mini-language for UI tests that check alerts, click on confirmations, inspect options available in SELECT elements, check whether elements are enabled or disabled and work with checkboxes and radio buttons. See the full list of supported methods in fitnesse.info/webtest.

Summary

On the end, here is a summary of lessons learned:

• Poll browser a few times per second through Selenium RC API using the Wait class to check for asynchronous events.
• Use DOM element IDs directly, don’t look for elements using xpath locators, labels or captions.
• Don’t use a generic pause. If you can, block and wait for a specific event.
• If your site behaves differently under test and when you play with it manually, missing events should be your first suspect. Fire events manually in that case.

FIT fixtures that implement the testing mini-language for .NET and Java can be downloaded from fitnesse.info/webtest. The code is released under GPL, so you can edit and experiment with it. If you add any new useful methods, please send them so that they can be included in the next version of this library…

Image credits: Timi az en vaok.

Selenium is a free browser automation and testing library, written by folks at ThoughtWorks. It can simulate text input, mouse events and execute various tests on page content. It’s written in JavaScript, and is compatible with all major browsers and platforms.

FitNesse is an open-source test and collaboration server, based on the Framework for integrated tests (FIT), and supports testing Java, .Net, Python and even some other code. I think that it is a good choice for the second side of the web UI testing coin, because it enables tests to be written almost like in English language. As the UI is very close to clients’ eyes, tests can and should be written so that clients can verify them (and if you are really lucky, even help with writing and maintaining the tests).

UPDATE: I wrote a follow-up to this article in Feb. 2008, focusing on testing AJAX applications. Read it on http://gojko.net/2008/02/14/ajax-selenium-fitnesse

Installing Selenium

The core of Selenium is written in JavaScript, and is compatible with all major browsers. The Selenium Remote Console provides the glue between a test browser and .Net, Java or Python code, and enables us to write and run Selenium tests from almost all popular test frameworks.

Just download Selenium Remote Console from http://www.openqa.org/selenium-rc. Unpack the files somewhere on your disk, and then start the server from the server folder of the package, by executing java -jar selenium-server.jar. The server starts on port 4444 by default, if that port is taken you can change it by adding -port number to the command line.

Selenium works by embedding the test site, along with some control scripts, in a frameset web page. The control scripts are then used to automate the test page in the main frame, and execute various tests. That method of work will clash with browser security rules if both frames do not come from the same domain. Remote console can open Internet Explorer and Firefox with a security tweak to allow cross-domain scripting, but if you don’t want that, or want to use a different browser, then put Selenium JavaScript files into your web site. Download them from http://www.openqa.org/selenium-core/, and put the files from core folder in the archive into the selenium-server folder of your web site (so that RemoteRunner.html is available on /selenium-server/RemoteRunner.html).

A quick test

Let’s first do a quick test to make sure that Selenium RC is running correctly. We’ll start a browser window, go to Google, enter a search phrase and click on the Search button. I’ll do the examples in C#, but you can also use Java – interfaces are very similar. Create a new .Net project, add thoughtworks.selenium.core.dll as a reference (it’s in the dotnet folder of Remote Control installation), and then create this class:

using System;
using Selenium;
namespace SeleniumTest
{
    class Console
    {
        static void Main(string[] args)
        {
            ISelenium sel = new DefaultSelenium(
              "localhost", 4444, "*iehta", "http://www.google.com");
            sel.Start();
            sel.Open("http://www.google.com/");
            sel.Type("q", "FitNesse");
            sel.Click("btnG");
            sel.WaitForPageToLoad("3000");
        }
    }
}

Make sure that Selenium RC is running, close all open Internet explorer windows, and then execute the program. Selenium RC will open a new IE window (it may be minimised on start, but you will see a new button in the task bar), and then go to Google and execute a search.

click for full-size image

Selenium window has three frames. Top-left enables the user to display the log and debug the execution by examining the DOM tree. Top-right frame displays the last four executed Selenium commands, which comes useful when troubleshooting tests. The page under test is displayed in the central frame.

The first line initialises a selenium console instance, connecting it to a remote console on localhost port 4444.

ISelenium sel = new DefaultSelenium(
"localhost", 4444, "*iehta", "http://www.google.com");

The third argument is a browser string, and *iehta is the code for security-tweaked Internet Explorer which allows cross-domain scripting. Use *chrome for firefox with cross-domain scripting, or *iexplore, *firefox and *opera for Internet Explorer, Firefox and Opera without cross-domain security tweaks. You can also specify a full path to the executable instead of these special keywords.

The fourth argument is the URL to the selenium test site, and is really important only if you do not use cross-domain scripting, but want Selenium to run the scripts from the local domain. In this case, it will try to get the remote runner from http://www.google.com/selenium-server/RemoteRunner.html, but since Google is not hosting a Selenium installation, it will run the local scripts instead. In any case, this argument needs to be specified, so just set it to some live site even if you use cross-domain scripting (like in this case).

As you can guess from the names, method Type will simulate entering data from keyboard into a text field, and Click will simulate a mouse click. We accessed the text field and the button by their names, q and btnG. Selenium can also find elements by their ID (prefix it with “identifier=”), XPath expression (prefix it with “xpath=”), DOM path or a CSS property. See http://www.openqa.org/selenium-core/reference.html for more information on locators.

Connecting from FitNesse

FitNesse is really great for automating acceptance tests, and if you have not had a chance to play with it yet, I suggest reading through the online user guide for testing Java code, or my tutorial Getting fit with .Net for working in the .Net environment.

Instead of writing a new fixture type for every page or web site, we use a generic WebTest fixture. It extends DoFixture, so that other fixtures can be easily embedded into it (for checking the back-end data after a web script, or preparing the stage for the web test). Web test fixture allows the scripts to be described almost like English prose, like this:

User opens URL http://localhost:7711/login.aspx
User types testuser into username field
User types testpassword into password field
User clicks on Log In
Page reloads in less than 3 seconds
Page contains text You have logged in

To achieve the best effect, the script must relate to what users see on the screen. However, that is easier said than done. Text fields, check boxes and buttons are all instances of the “input” element, and although text fields are mostly distinguished by name, buttons names are typically not important, and users only see the “value” attribute of a button. WebTest fixture looks for various combinations of attributes: for example, when searching for a button, it first looks for an input element with type equal to submit or button, and the name attribute matching the query. If no such element is found, it looks for similar elements, with value attribute matching the query. On the end, it looks for a button with matching ID. All these combinations are listed in the buttonLocators array in the code:

 public static readonly string[] buttonLocators = new String[] {
	"xpath=//input[@type='submit' and @name='{0}']",
	"xpath=//input[@type='button' and @name='{0}']",
	"xpath=//input[@type='submit' and @value='{0}']",
	"xpath=//input[@type='button' and @value='{0}']",
	"xpath=//input[@type='submit' and @id='{0}']",
	"xpath=//input[@type='button' and @id='{0}']"};

Although looking for an element by ID or name would be quicker, we intentionally use Xpath and add extra information, to catch errors caused by wrong element types. GetLocator method checks a list of XPath expressions and returns the first matching element from the active document, so UserClicksOn method is very simple:

public void UserClicksOn(String buttonCaption){
            instance.Click(GetLocator(buttonCaption, buttonLocators));
}

We use the same technique to put text into text fields, areas and password fields. An array of Xpath expressions is used to check for various combinations of valid element types and attribute values, and the first matching element is then passed to Selenium Type method.

public void UserTypesIntoField(String what, String where){
      instance.Type(GetLocator(where.Replace(" ", ""), textFieldLocators), what);
}

Because text fields will mostly be referenced by a name or ID, but users will see the text before the field on the page (and probably try to reference it with that), we also strip the blanks from the name. So the test script line:

User types 10109 into security number field

will correctly map into an element named “securitynumber”.

With web tests, we typically want to check if the result was correct and if the site was responsive enough, and Selenium method WaitForPageToLoad can be used to check if a page loads in any given amount of time. The API is a bit weird, as it expects a string containing the number of milliseconds. We wrapped that into this WebTest fixture call:

public void PageReloadsInLessThanSeconds(String sec){
            instance.WaitForPageToLoad(sec + "000");
}

On the end, we need a method to verify that page contents are correct. Selenium method IsTextPresent can help with that:

public bool PageContainsText(String s){
            return instance.IsTextPresent(s);
}

We’ll also need to open a browser and set up the Selenium environment from the test fixture. To keep the test script format in english-like prose, we’ll use the following syntax:

Start Browser *iehta with selenium console on localhost at port 4444 and scripts at http://localhost:7711

In WebTest fixture, that is covered by the following method:

public void StartBrowserWithSeleniumConsoleOnAtPortAndScriptsAt
   (String browser, String rcServer, int rcPort, String seleniumURL){
            instance = new DefaultSelenium(
                 rcServer, rcPort, browser, seleniumURL);
            instance.Start();
}

On the end of each test, we need to shut down the browser, so that remote console does not run out of resources. For that, we’ll use the following method:

public void ShutdownBrowser(){
            instance.Stop();
}

With all that, the web test can be easily described with this FitSesse script:

!|webfixture.WebTest|

!|Start Browser|*iehta|With Selenium Console On| localhost| At Port |4444|And Scripts At|http://localhost:7711|

|User Opens URL|http://localhost:7711/login.aspx|
|User types|testuser|into|username|field|
|User types|testpassword|into|password|field|

|User clicks on|Log In|
|Page reloads in less than|3|seconds|
|Page contains text|You have logged in|

|Shutdown browser| 

Ajax testing

With web 2.0 pages, waiting for the page to reload before we continue testing is not the best choice. Selenium also supports
waiting for a JavaScript condition. Remember that Selenium script is being executed in a different frame, so referencing DOM elements and JS functions in your test site page will not work straight away. Prefix the references with
selenium.browserbot.getCurrentWindow() to get from the Selenium frame to your test page. Here’s an example:

instance.WaitForCondition(
  "(selenium.browserbot.getCurrentWindow().get_username()!=null)",
  timeout);

Remote execution

Selenium Remote Console will try to open a new browser session with different security settings, but under the profile of the current user (in fact, the user which started the remote console). This may cause problems when you already have an open browser on the same machine. The workaround is to use a different browser for testing. For example, I use Firefox for normal browsing, so RC can start IE and play with it. But, there is a much better solution.

As the name suggests, Remote Console can run on a dedicated test server, and be accessed remotely. Any browsers that RC opens will run on that machine. That enables developers to work with browsers on their machines and run the tests, without clashing with test execution. It also enables us to re-use the same FitNesse scripts to check how various browsers are behaving, even on different platforms. Just separate the initialisation and browser shut-down into SetUp and TearDown scripts, and then create a couple of different test suites with different SetUps which will use various browsers and remote console installations. Then, use symbolic links to import scripts from one test suite into all others. You will maintain scripts in one place, and they will test multiple browsers and platforms.

Extending WebTest

WebTest currently supports two more interesting methods: PageURLIs(String s) and UserSelectsFrom(String what, String where). The first allows you to verify the page URL (and test redirections). The second automates selecting an option from a drop-down menu (HTML select element), in the form of User Selects NY from Countries.

Feel free to modify and adjust WebTest to your needs. For start, see dotnet/doc/index.html in your remote control installation for more information on ISelenium interface. You can a add search by ID to the top of all locator arrays, so that tests run faster, but that will make them more error-prone. You will probably want to add a few different expressions to the textbox, button or select lists. If you add something which may be useful to others, please share it by writing a comment on the WebTest fixture page.

Photo credits:Sanja Gjenero/SXC