How to test AJAX sites with FitNesse and Selenium RC

UPDATE: My thinking on this has changed significantly in the years following this post, but according to Google it’s still quite popular. If you’re interested in combining FitNesse and Selenium, make sure to read this post as well: How to implement UI Testing without shooting yourself in the foot. It explains how to avoid some of the most common problems.

It’s been almost a year since I wrote Automating web tests with FitNesse and Selenium, offering an idea how FitNesse can be used to implement a nice customer-friendly mini-language for user interface testing. Since then, that article has been one of the most popular, if not the single most popular, article on this web site. I have helped several clients improve and integrate their UI testing based on the ideas in that article, expanding and improving the mini-language, especially for AJAX testing. Here is what I’ve learned about that in the last year.

Quick note before we continue: FIT fixtures described in this article can be downloaded from fitnesse.info/webtest. Both Java and .NET are supported and the fixtures are released under GPL. To learn how to set everything up, read the original article.

Effective AJAX testing with selenium.Wait

In the original article, I have suggested using WaitForCondition and selenium.browserbot.getCurrentWindow() to dynamically evaluate JavaScript expressions and block until they become fulfilled. Although this did the trick, the code was very ugly and error-prone, mixing JavaScript evaluations with Java/.NET expressions. It turned out that it was much better to poll browser a few times per second through Selenium RC API. This way, the code is much more consistent, and waiting conditions are much easier to write. Polling, in theory, causes unnecessary delay, but this case Selenium RC and UI testing itself introduce a lot of latency, so the polling does not really make things any worse. In fact, the Java version of Selenium Remote Control has API support for that: Wait class in com.thoughtworks.selenium package. The official JavaDoc for that class is a bit wrong and misleading, but with a bit of experimenting that class is very easy to use, and quite powerful. The official documentation suggests this usage pattern:

new Wait("Couldn't find close button!") {
    boolean until() {
        return selenium.isElementPresent("button_Close");
    }
};

Yet this code will not even compile with Selenium RC 9.2, because no such constructor exists. Instead, the error message should be passed while calling the blocking wait method later. Here is an example that blocks until some text appears on the page:

  // private class of WebTest fixture
  private class WaitForTextToAppear extends Wait{
    private String text;
    public  WaitForTextToAppear(String  text){
      this.text= text;  
    }
    public boolean until(){
       return WebTest.this.pageContainsText(text);
    }
  }
 // method of WebTest Fixture
 public boolean waitSecondsForTextToAppear(int timeout, String text){
   Wait x=new WaitForTextToAppear(text);
   x.wait("Cannot find text " +text+ " after "+timeout+" seconds",timeout*1000);
   return true;
 }

Waiting In mini-language

In FitNesse, this would be used as:

|wait|5|seconds for text|Hello World|to appear|

This line would block the execution of the page up to five seconds, waiting for Hello World to appear anywhere in the page. If that text does not appear after five seconds, the test breaks. Using this pattern, it is very easy to implement any kind of asynchronous waiting without manually checking for JavaScript expressions and worrying about Selenium-to-application page DOM conversions. So far, I have implemented the following waiting methods for the testing mini-language:

|wait|5|seconds for element|username|to appear|

|wait|5|seconds for element|username|to disappear|

|wait|5|seconds for text|Hello World|to appear|

|wait|5|seconds for text|Hello World|to disappear|

|wait|5|seconds for field|username|to contain text|Hello World|

|wait|5|seconds for element|username_label|to contain text|Please enter username|

the difference between “wait for field to contain text” and “wait for elelement to contain text” is that the first method checks for the current value of an INPUT field (<input type=”text” name=”username” value=”Hello World”/>), and the second checks for text inside a dom element (<span id=”username_label”>Please enter username</span>). .NET Selenium RC API does not have this class, but it was not hard to implement, so the .NET fixture also supports this in the mini-language.

Locator lookup takes too long

Selenium uses “locators” to point to DOM elements that you want to automate. In the original article, I suggested a complex scheme of mapping user-friendly descriptions to locators so that we can use button captions, labels and similar visible text and labels to point to page elements. Under this scheme, the FitNesse-Selenium glue code tried out locators from an array until it found a match. This turned out to be such a performance penalty that I no longer suggest it. Each attempt to discover whether an element was present or not went through the full cycle of FitNesse, Selenium Remote Console, Browser, Selenium and back. In average, that increased the time for a UI test to execute by 500-600 percent. It turned out that, with a bit of care in naming, DOM element IDs can be used in all cases so that test pages are still descriptive enough, and indirect locator lookup can be avoided. So now I suggest using DOM element IDs directly, not using labels or captions.

Effective locators in mini-language

I have introduced a new parameter to control whether the lookup is performed or not. It is still on by default, but if you put this table into your page:

|set locator lookup|false|

tests will execute much much faster. This will require you, however, to use DOM element IDs exclusively to point to all page elements. Even with lookup turned on, tests should now run faster if you use DOM element IDs, because I’ve moved IDs to the top of lookup arrays for all element types.

Element type safety is not really important

In the original article, I suggested checking whether something is a button, input field or a link when doing methods like click, so that something like:

|user clicks on|save|button|

would fail if save was a link identifier. I thought that would introduce one more level of safety and avoid problems because of ambiguous definitions. Since trying out all those locators works really slow, I gave up on checking for the type of element, and use that only in special cases (such as radio buttons, which should be selectable by value as well as name). This did not, in practice, make tests any less effective. So I now suggest avoiding such complex checks.

Direct element access in mini-language

The WebFixture mini-language now supports a bunch of methods for generic elements, which are located using the DOM element ID or element name, without checking the exact type. So you can write something like:

|user clicks on|save|

and that is going to work for buttons, labels, text elements and anything you can click on.

Don’t pause, wait for specific events

In the original article, I introduced a few methods that blocked until the whole page loaded correctly. With AJAX testing, those methods are no longer applicable. To allow an asynchronous operation to complete before continuing with the test, lots of people used the ‘pause’ function. This caused more harm than good, so I’m now thinking of effectively removing that function or throwing an exception on the end of the pause to print a warning that the test should be rewritten. Pause is makes tests error prone and very brittle. If, for example, you pause for 5 seconds to wait for a form to load, then when you run the tests on different hardware or over network, the test itself might break because of longer latency (so the code implementation is still correct, but the test fails, which is a very dangerous anti-pattern). This gets solved by extending the pause to be long enough, so your tests are no longer running for 10 seconds but a minute… once the test suite starts running for longer than a minute people are no longer running tests before committing, and that makes the feedback loop even longer, reducing the effectiveness of tests.

It’s much better, if you can, to block and wait for a specific event. So instead of pausing, use one of the wait functions as described in the beginning of this article. Those functions still allow you to specify the ultimate timeout period, so that if the event does not happen after 10-15 seconds the test breaks. A very important difference is that, in normal case, the execution will continue straight after the event does happen. So you’ll wait exactly as long as you need to, and not a second longer.

DOM events don’t always fire automatically

When you type stuff in with Selenium, DOM elements such as onFocus, onBlur and onChange don’t fire automatically (not to mention onKeyUp and onKeyDown). Lots of AJAX code is built around those events, and tests fail if the code relies on events. Selenium allows you to triggers the events manually using fireEvent method. So, if your site behaves differently under test and when you play with it manually, missing events should be your first suspect.

Firing events manually in the mini-language

I have changed the UserTypesIntoField method to fire events as well, in the appropriate order. You can fire an event explicitly using the fire event for method:

|fire event|blur|for|username|

To fire events, don’t put the leading on — use click instead of onclick.

Some people prefer click-and-record

Although I’ve tried to introduce a customer-friendly mini-language for testing, some people liked the idea of using FitNesse to wrap Selenium tests more because it UI tests could then be integrated into automated integration testing and because they could prepare and verify their database during Selenium tests. But they still preferred using Selenium IDE to record tests, without having to manually write them. I have introduced a new fixture class, webfixture.PlainSeleniumTest, that more or less just forwards the commands to Selenium RC. So you can use a Selenium test table, paste it in FitNesse, change it just a bit, and run the test using Selenium Remote Control. The only changes required are deleting the test name (first row of the table) and optionally changing the open command to include a relative URL instead of an absolute one. That allows you to record tests using Selenium IDE, but still integrate them into Cruise Control and control the database or talk to your domain objects during tests.

Selenium tests can be stored as HTML files, and you can use HTML directly in FitNesse without converting it to the Wiki syntax. Just paste the table anywhere in the page, but put the HTML code between !- and -! to tell FitNesse to display the contents raw, without special formatting.

Originally, I thought that I could just get away by wrapping a Selenium RC instance into a Sequence Fixture as the system under test, and pass all commands to it. But Selenium RC API and browser-based Selenium scripts differ in some parts, especially for the waiting methods and verification methods. For example, verifyVisible in Selenium is called isVisible in Selenium RC. So I wrote a bunch of wrappers for verifyXXX and assertXXX methods to make tests run without changing.

The next problem was that the resulting Selenium test tables may have one additional empty cell if the method only has one argument, such as open url. Because of that additional argument, FitNesse will try to map the row to the method open(String, String) and fail because it could not find such a method. Because of that, I also wrote a bunch of wrappers for single-parameter methods that have a second string argument, which is just ignored. For example:

public void open (String url, String ignore){
  open(url);
}
public void click(String what, String ignore){
  instance.click(what);
}

With that, Selenium tests started working more or less out of the box. I occasionally have to add another method for one of the reasons mentioned above, but that is fairly quick and straightforward.

There’s one more trick related to this. Selenium scripts, especially those recorded, can be quite lengthy. FitNesse will flush output after each table, so if you leave the test as one big table, it will look like FitNesse is not doing anything for a while and then you’ll get all results at once. I’ve found it much better to just split the big table into several smaller tables, so that I can track progress while the test is running.

Use browser names instead of browser codes

Cory Foy suggested that we should not use browser codes like *iehta, but make tests even more user friendly by using descriptive browser names. I like that idea very much, so the start browser now supports both codes and names like IE and Firefox. It will map the name to the appropriate browser code for you.

Plenty of new methods

I have also added quite a few new methods to the mini-language for UI tests that check alerts, click on confirmations, inspect options available in SELECT elements, check whether elements are enabled or disabled and work with checkboxes and radio buttons. See the full list of supported methods in fitnesse.info/webtest.

Summary

On the end, here is a summary of lessons learned:

  • Poll browser a few times per second through Selenium RC API using the Wait class to check for asynchronous events.
  • Use DOM element IDs directly, don’t look for elements using xpath locators, labels or captions.
  • Don’t use a generic pause. If you can, block and wait for a specific event.
  • If your site behaves differently under test and when you play with it manually, missing events should be your first suspect. Fire events manually in that case.

FIT fixtures that implement the testing mini-language for .NET and Java can be downloaded from fitnesse.info/webtest. The code is released under GPL, so you can edit and experiment with it. If you add any new useful methods, please send them so that they can be included in the next version of this library…

Image credits: Timi az en vaok.

I'm Gojko Adzic, author of Impact Mapping and Specification by Example. I'm currently working on 50 Quick Ideas to Improve Your User Stories. To learn about discounts on my books, conferences and workshops, sign up for Impact or follow me on Twitter. Join me at these conferences and workshops:

Specification by Example Workshops

Product Owner Survival Camp

Conference talks and workshops

36 thoughts on “How to test AJAX sites with FitNesse and Selenium RC

  1. Having developed a lot of tests with Selenium, I wanted to try FitNesse. However, it was not easily capable of testing certain parts of our web application. Simply put no test was simple enough that “english prose” could describe what to really test.

  2. Hi Eric,

    I’m not sure that I understand the problem. Is it in the lack of fitnesse-selenium rc commands or with expressing the test itself? If you can write a test in selenium language, you can plug it straight in using PlainSeleniumFixture (with only minor changes). If the WebTest “english prose” does not contain the commands that you would like to use, I’d be happy to add them. Could you give me more details on what you want to test?

  3. Hi Frederic,

    FIT fixtures are Java/C# classes, so you can use both WebTest and PlainSeleniumTest directly from Java or C#, although I don’t really see the point of that. One of central ideas with WebTest was that people with no programming knowledge (especially customers) could participate in writing and verifying automated web ui tests, so that programmers could spend their time doing more important stuff. If you want to write tests in Java/C#, you might as well use Selenium RC directly, that is what it is for :)

  4. Do you expect people with no programming knowledge to write this:

    |user clicks on|save|button|
    |set locator lookup|false|
    |wait|5|seconds for field|username|to contain text|Hello World|

  5. locator is part of a technical setup, i would not expect customers or QA to control that. The other commands are today being used in several teams by non-programmers very happily. As I have noted, some people like using Selenium IDE better to record the tests. In that case, PlainSeleniumFixture is used rather than WebTest to provide database or domain integration for pre-conditions and post-verifications.

  6. hi!,
    presently i am using selenium IDE for automating, planing to use C# with nunit …how i can use ? how i can use nunit……?
    what r the steps i have to follow to install n use nunit for testing ……could u plz help me …i started using selenium from past one month ……automated 20 scripts in IDE.
    hope u can help me
    waiting for ur reply
    regards

  7. Hi,

    I m currently working on a website developed in ASP.net. In a particular situation, where when tried clicking on a go buttton, which pull the ajax information and loads it on the same page, I could not record that button in selinium (firefox 2.0.0.8) and when tried typing the |click|setpostcode| in fitnesse directly and run the test, that particular function is performed.

    note : setpostcode is DOM element ID for the button.

    can you please help on this.

    rgds,
    Prem

  8. Sorry a mistake in my question, the particular function on DOM element ID for the button is not performed.

  9. Hi Prem,

    here are a few ideas:

    - there might be more than one setpostcode element on the page
    - it is possibly a div styled to look like a button
    - some “smart” javascript handling is used instead of a normal click

    can you send the piece of HTML code that contains that button

  10. Hi Gojko,

    not sure what happened there, i managed to submit your name while capitalising it. weird??

    Anyhoo, i have been trying your webtest fixture and have run into problems with the waitfor method you have supplied.

    In a simple test of

    1 openAndWait URL
    2 clickAndWait link
    3 pageURLIs URL
    4 element h3 contains text SOME_TEXT

    steps 3 and 4 will fail, because selenium races ahead. This often manifests itself as an “access denied” error.

    However if a place a

    pause 10 Seconds

    between the steps, the page has time to load, and the steps all pass.

    The trouble is that the page is made up of many components from third parties over which i have no control. I don’t want to stub those tests, i want to tests against the live services as its a very important test for us. Is readyState my only answer?

    Cheers,

    Stuart.

  11. Hi,

    with respect to my previous question on 9th july 2008 about the click button,
    this is the piece of code,

    <a href=”#” rel=”nofollow”>

    rgds,
    Prem

  12. Hi Stuart,

    If you use WebTest and not PlainSeleniumTest, see the command reference for that fixture, especially the bottom of the page. With WebTest, wait functions are called something like

    |wait|5|seconds|for …|

    OpenAndWait and ClickAndWait are not directly supported. I can implement them in the next version.

  13. Hi Prem,

    I’m having the same issue with links such as add stop

    When using the selenium.click method it doesn’t invoke the java script that is triggered on click and i can’t find it in the code… did you find a solution for this problem?

  14. Hi,

    Started Fitnesse tool. Recorded a simple google search using Selenium IDE, saved it as html file in myFixtures folder and copied it on the Frontpage of the Fitnesse tool. Added Test property and saved it. Executed startSeleniumRC.bat file. Clicked on Test in Fitnesse. Not getting the result.

    Pls clarify that the above said procedure is right/enough to collaborate fitnesse with selenium and execute a scenario.

    If you reply asap that will be verymuch useful for me..

    Thankyou..

  15. Hi,

    Fitnesse tool is in running state. Recorded a simple goole search using Selenium ide and saved it as a html file in myFixtures folder. Copied the script and pasted it on the Frontpage of the Fitnesse tool. Added Test property. Executed startSeleniumrs.bat file. Clicked on Test in Fitnesse. Not getting the result.

    Pls carify that the above said procedure is correct/enough to integrate Fitnesse with Selenium. If you clarify me asap that would be helpful.

    Thankyou.

  16. Hi Gojko,

    Thanks for your immediate reply.

    I went through STIQ. But my need is automating web applications using Fitnesse and Selenium. For that only i performed the steps mentioned in my earlier post. Pls guide me what should i do to automate web UI tests using Fitnesse and selenium.

    Also, can you tell what needs to be done to install Selenium RC? I executed “java -jar selenium-server.jar” in commad prompt. Is that enough to install selenium RC?

  17. Hi,

    I recorded login functionality using Selenium IDE. The recorded sript is

    package myFixtures;

    import com.thoughtworks.selenium.*;
    import java.util.regex.Pattern;

    public class thebridge extends SeleneseTestCase {
    public void setUp() throws Exception {
    setUp(“http://XXX.com/”, “*chrome”);
    }
    public void testNew() throws Exception {

    selenium.open(“http://XXX.com”);
    selenium.type(“loginid”, “AAAl”);
    selenium.select(“domain”, “BBB”);
    selenium.type(“password”, “CCC”);
    selenium.click(“login”);
    selenium.waitForPageToLoad(“30000″);
    }
    }

    Now i need to write a fitnesse fixture for execute the above functionality. Since i recordd the script using selenium ide, i have chosen PlainSeleniumTestusage template.
    The fitnesse test page code is
    !contents -R2 -g -p -f -h
    !path C:\Documents and Settings\user\Desktop\FITness Tool\fitnesse20080812\fitnesse\FitNesseRoot\files\examples\myFixtures

    !|myFixtures.thebridge|
    |start browser|firefox|localhost|4444|http://XXX.com|
    |open|http://XXX.com|
    |enter|username|aaal|
    |select|domain|bbb|
    |enter|password|ccc|
    |press|Login|
    |Page reloads in less than|3|seconds|
    |shutdown browser|

    but selenium throws exception as”could not find the fixture”.

    Pls tell me, do i need to do anything separately to use selenium scirpts from within fitnesse? Also, which fixture type should i use for my need? and if the above fixture table is not correct, pls correct me.

    thanku

  18. Hi,

    Now iam working on Fitnesse and selenium to automate our application. But i have a doubt that why Fitnesse and why selenium?. I tried to get answers by surfing in the net and iam not convinced with those details. Can you answer me in this regard?. Kindly mention if there are any related articles in net. Thank you.

    Diya

  19. Hi,

    Without integrating selenium with Fitnesse, can’t we directly verify any application by writing fixture code in java?

    I am not able to invoke commandline TestRunner using the command given in Fitnesse website. Will TestRunner be opened as a webpage to show the results?

  20. Hi,

    I’m using the PlainSeleniumTest web fixture. Occasionally, when I call waitForTextPresent I get an exception like this: “com.thoughtworks.selenium.SeleniumException: ERROR: Couldn’t access document.body. Is this HTML page fully loaded?” How do I get Selenium to wait for my page to fully load? I thought that was the whole point of using the wait methods. I can add a pause statement before waiting for the text to be present, but that’s just ugly.

    Any thoughts?

  21. Hi Andrew,

    *andClick methods from selenium are not directly exposed through seleniumRC. I implemented clickAndWait but for the rest you’ll just have to call waitForPageToLoad manually if you use PlainSeleniumTest.

  22. Hi All,

    I am testing our application using Fit/Fitnesse+Selenium tool, as per our requirement i want to export the execution results to HTML to maintain repository, can any one help me for exporting the result, my email id: ksksnath.reddy@yahoo.co.in
    appreciate your help :-)

  23. Hello Mr Gojko,

    I have been using selenium rc with java to create automated tests using testNG framework

    Now i want to get into creating acceptance testing using fitnesse

    Can you please provide me the guidelines or steps in how one could

    setup a simple acceptance testing using selenium rc-java with fitness from SCRATCH!

    Really would appreciate this and would be a great help for any newcomers to Fitnesse

    Particulary for function testers!!

    Many Thanks
    GK

  24. Hi Geekay,

    I don’t have any blog posts with the level of detail you need. I suggest googling for selenesse, this is a mashup of selenium and fitnesse.

  25. Hi Gojko,

    is there a way to feed fitnesse .html files instead of the wiki text? I want to create the HTML tables directly in some html editor and then supply a path to the file.

    Thank you very much

  26. Thank you. I read the articles and they make a lot of sense. I was just starting to get excited around Silenium so the Sine of death by UI testing was just on time. I guess it’s just that people forget the word Abstraction when they start moving toward testing “concrete” things and this plays them a trick.

    The !- -! escape is a nice work-around. I was hoping for something like being able to populate a folder with real .htm files and to just point fitnesse to that folder so that I don’t have to manually escape the html source. A colleague of mine wrote a utility tool for .net (http://fit-gui.sourceforge.net/) which I found very convenient but I want to use Java and couldn’t find something similar. May be I’ll try to rewrite his tool in Java.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>