How to test AJAX sites with FitNesse and Selenium RC

By
gojko
– February 14, 2008Posted in: fitnesse

UPDATE: My thinking on this has changed significantly in the years following this post, but according to Google it’s still quite popular. If you’re interested in combining FitNesse and Selenium, make sure to read this post as well: How to implement UI Testing without shooting yourself in the foot. It explains how to avoid some of the most common problems.

It’s been almost a year since I wrote Automating web tests with FitNesse and Selenium, offering an idea how FitNesse can be used to implement a nice customer-friendly mini-language for user interface testing. Since then, that article has been one of the most popular, if not the single most popular, article on this web site. I have helped several clients improve and integrate their UI testing based on the ideas in that article, expanding and improving the mini-language, especially for AJAX testing. Here is what I’ve learned about that in the last year.

Quick note before we continue: FIT fixtures described in this article can be downloaded from fitnesse.info/webtest. Both Java and .NET are supported and the fixtures are released under GPL. To learn how to set everything up, read the original article.

Effective AJAX testing with selenium.Wait

In the original article, I have suggested using WaitForCondition and selenium.browserbot.getCurrentWindow() to dynamically evaluate JavaScript expressions and block until they become fulfilled. Although this did the trick, the code was very ugly and error-prone, mixing JavaScript evaluations with Java/.NET expressions. It turned out that it was much better to poll browser a few times per second through Selenium RC API. This way, the code is much more consistent, and waiting conditions are much easier to write. Polling, in theory, causes unnecessary delay, but this case Selenium RC and UI testing itself introduce a lot of latency, so the polling does not really make things any worse. In fact, the Java version of Selenium Remote Control has API support for that: Wait class in com.thoughtworks.selenium package. The official JavaDoc for that class is a bit wrong and misleading, but with a bit of experimenting that class is very easy to use, and quite powerful. The official documentation suggests this usage pattern:

new Wait("Couldn't find close button!") {
    boolean until() {
        return selenium.isElementPresent("button_Close");
    }
};

Yet this code will not even compile with Selenium RC 9.2, because no such constructor exists. Instead, the error message should be passed while calling the blocking wait method later. Here is an example that blocks until some text appears on the page:

  // private class of WebTest fixture
  private class WaitForTextToAppear extends Wait{
    private String text;
    public  WaitForTextToAppear(String  text){
      this.text= text;
    }
    public boolean until(){
       return WebTest.this.pageContainsText(text);
    }
  }
 // method of WebTest Fixture
 public boolean waitSecondsForTextToAppear(int timeout, String text){
   Wait x=new WaitForTextToAppear(text);
   x.wait("Cannot find text " +text+ " after "+timeout+" seconds",timeout*1000);
   return true;
 }

Waiting In mini-language

In FitNesse, this would be used as:

|wait|5|seconds for text|Hello World|to appear|

This line would block the execution of the page up to five seconds, waiting for Hello World to appear anywhere in the page. If that text does not appear after five seconds, the test breaks. Using this pattern, it is very easy to implement any kind of asynchronous waiting without manually checking for JavaScript expressions and worrying about Selenium-to-application page DOM conversions. So far, I have implemented the following waiting methods for the testing mini-language:

|wait|5|seconds for element|username|to appear|

|wait|5|seconds for element|username|to disappear|

|wait|5|seconds for text|Hello World|to appear|

|wait|5|seconds for text|Hello World|to disappear|

|wait|5|seconds for field|username|to contain text|Hello World|

|wait|5|seconds for element|username_label|to contain text|Please enter username|

the difference between “wait for field to contain text” and “wait for elelement to contain text” is that the first method checks for the current value of an INPUT field (<input type=”text” name=”username” value=”Hello World”/>), and the second checks for text inside a dom element (<span id=”username_label”>Please enter username</span>). .NET Selenium RC API does not have this class, but it was not hard to implement, so the .NET fixture also supports this in the mini-language.

Locator lookup takes too long

Selenium uses “locators” to point to DOM elements that you want to automate. In the original article, I suggested a complex scheme of mapping user-friendly descriptions to locators so that we can use button captions, labels and similar visible text and labels to point to page elements. Under this scheme, the FitNesse-Selenium glue code tried out locators from an array until it found a match. This turned out to be such a performance penalty that I no longer suggest it. Each attempt to discover whether an element was present or not went through the full cycle of FitNesse, Selenium Remote Console, Browser, Selenium and back. In average, that increased the time for a UI test to execute by 500-600 percent. It turned out that, with a bit of care in naming, DOM element IDs can be used in all cases so that test pages are still descriptive enough, and indirect locator lookup can be avoided. So now I suggest using DOM element IDs directly, not using labels or captions.

Effective locators in mini-language

I have introduced a new parameter to control whether the lookup is performed or not. It is still on by default, but if you put this table into your page:

|set locator lookup|false|

tests will execute much much faster. This will require you, however, to use DOM element IDs exclusively to point to all page elements. Even with lookup turned on, tests should now run faster if you use DOM element IDs, because I’ve moved IDs to the top of lookup arrays for all element types.

Element type safety is not really important

In the original article, I suggested checking whether something is a button, input field or a link when doing methods like click, so that something like:

|user clicks on|save|button|

would fail if save was a link identifier. I thought that would introduce one more level of safety and avoid problems because of ambiguous definitions. Since trying out all those locators works really slow, I gave up on checking for the type of element, and use that only in special cases (such as radio buttons, which should be selectable by value as well as name). This did not, in practice, make tests any less effective. So I now suggest avoiding such complex checks.

Direct element access in mini-language

The WebFixture mini-language now supports a bunch of methods for generic elements, which are located using the DOM element ID or element name, without checking the exact type. So you can write something like:

|user clicks on|save|

and that is going to work for buttons, labels, text elements and anything you can click on.

Don’t pause, wait for specific events

In the original article, I introduced a few methods that blocked until the whole page loaded correctly. With AJAX testing, those methods are no longer applicable. To allow an asynchronous operation to complete before continuing with the test, lots of people used the ‘pause’ function. This caused more harm than good, so I’m now thinking of effectively removing that function or throwing an exception on the end of the pause to print a warning that the test should be rewritten. Pause is makes tests error prone and very brittle. If, for example, you pause for 5 seconds to wait for a form to load, then when you run the tests on different hardware or over network, the test itself might break because of longer latency (so the code implementation is still correct, but the test fails, which is a very dangerous anti-pattern). This gets solved by extending the pause to be long enough, so your tests are no longer running for 10 seconds but a minute… once the test suite starts running for longer than a minute people are no longer running tests before committing, and that makes the feedback loop even longer, reducing the effectiveness of tests.

It’s much better, if you can, to block and wait for a specific event. So instead of pausing, use one of the wait functions as described in the beginning of this article. Those functions still allow you to specify the ultimate timeout period, so that if the event does not happen after 10-15 seconds the test breaks. A very important difference is that, in normal case, the execution will continue straight after the event does happen. So you’ll wait exactly as long as you need to, and not a second longer.

DOM events don’t always fire automatically

When you type stuff in with Selenium, DOM elements such as onFocus, onBlur and onChange don’t fire automatically (not to mention onKeyUp and onKeyDown). Lots of AJAX code is built around those events, and tests fail if the code relies on events. Selenium allows you to triggers the events manually using fireEvent method. So, if your site behaves differently under test and when you play with it manually, missing events should be your first suspect.

Firing events manually in the mini-language

I have changed the UserTypesIntoField method to fire events as well, in the appropriate order. You can fire an event explicitly using the fire event for method:

|fire event|blur|for|username|

To fire events, don’t put the leading on — use click instead of onclick.

Some people prefer click-and-record

Although I’ve tried to introduce a customer-friendly mini-language for testing, some people liked the idea of using FitNesse to wrap Selenium tests more because it UI tests could then be integrated into automated integration testing and because they could prepare and verify their database during Selenium tests. But they still preferred using Selenium IDE to record tests, without having to manually write them. I have introduced a new fixture class, webfixture.PlainSeleniumTest, that more or less just forwards the commands to Selenium RC. So you can use a Selenium test table, paste it in FitNesse, change it just a bit, and run the test using Selenium Remote Control. The only changes required are deleting the test name (first row of the table) and optionally changing the open command to include a relative URL instead of an absolute one. That allows you to record tests using Selenium IDE, but still integrate them into Cruise Control and control the database or talk to your domain objects during tests.

Selenium tests can be stored as HTML files, and you can use HTML directly in FitNesse without converting it to the Wiki syntax. Just paste the table anywhere in the page, but put the HTML code between !- and -! to tell FitNesse to display the contents raw, without special formatting.

Originally, I thought that I could just get away by wrapping a Selenium RC instance into a Sequence Fixture as the system under test, and pass all commands to it. But Selenium RC API and browser-based Selenium scripts differ in some parts, especially for the waiting methods and verification methods. For example, verifyVisible in Selenium is called isVisible in Selenium RC. So I wrote a bunch of wrappers for verifyXXX and assertXXX methods to make tests run without changing.

The next problem was that the resulting Selenium test tables may have one additional empty cell if the method only has one argument, such as open url. Because of that additional argument, FitNesse will try to map the row to the method open(String, String) and fail because it could not find such a method. Because of that, I also wrote a bunch of wrappers for single-parameter methods that have a second string argument, which is just ignored. For example:

public void open (String url, String ignore){
  open(url);
}
public void click(String what, String ignore){
  instance.click(what);
}

With that, Selenium tests started working more or less out of the box. I occasionally have to add another method for one of the reasons mentioned above, but that is fairly quick and straightforward.

There’s one more trick related to this. Selenium scripts, especially those recorded, can be quite lengthy. FitNesse will flush output after each table, so if you leave the test as one big table, it will look like FitNesse is not doing anything for a while and then you’ll get all results at once. I’ve found it much better to just split the big table into several smaller tables, so that I can track progress while the test is running.

Use browser names instead of browser codes

Cory Foy suggested that we should not use browser codes like *iehta, but make tests even more user friendly by using descriptive browser names. I like that idea very much, so the start browser now supports both codes and names like IE and Firefox. It will map the name to the appropriate browser code for you.

Plenty of new methods

I have also added quite a few new methods to the mini-language for UI tests that check alerts, click on confirmations, inspect options available in SELECT elements, check whether elements are enabled or disabled and work with checkboxes and radio buttons. See the full list of supported methods in fitnesse.info/webtest.

Summary

On the end, here is a summary of lessons learned:

  • Poll browser a few times per second through Selenium RC API using the Wait class to check for asynchronous events.
  • Use DOM element IDs directly, don’t look for elements using xpath locators, labels or captions.
  • Don’t use a generic pause. If you can, block and wait for a specific event.
  • If your site behaves differently under test and when you play with it manually, missing events should be your first suspect. Fire events manually in that case.

FIT fixtures that implement the testing mini-language for .NET and Java can be downloaded from fitnesse.info/webtest. The code is released under GPL, so you can edit and experiment with it. If you add any new useful methods, please send them so that they can be included in the next version of this library…

Image credits: Timi az en vaok.

Tags: , , , , , ,