Gojko Adzic

This is from my presentation on the Robot Framework at the Agile acceptance testing tools round-up event held on 27 May 2009 at Skills Matter

Robot Framework is a keyword-driven test automation framework which empowers testers to
automate and manage complex workflow scripts very efficiently. It is written in Python and supports scripting Python and Java applications (through Jython), but it also has flexible support for remote execution and different system-under-test libraries which allow it to be used for automating Selenium scripts and tools that are not written in Python.

Keywords

The concept of “keywords” is so central to the Robot framework that it is worth explaining before we actually get into the details of the framework. Keywords are basically actions that are executed during a test case – think of them as names for test steps. For example, login or open web page. They can have arguments (eg username to enter, or title of the page to verify).

These keywords can be implemented by the system under test, various extension libraries (eg Selenium) or with scripts combining other keywords. So, for example, you can use keywords that deal with a web interface such as “open page” or “click” to write a script for a “login” keyword and then use that higher level keyword in domain test scripts.

The idea is very similar to xUseCase recording principles, where low-level technical steps build up a higher level domain test step which is then used to efficiently specify and verify functionality. When the web site under test changes, we just need to modify scripts for individual keywords but tests that describe functionality from a domain perspective don’t need to change. This makes functional tests both more stable and resilient to change, and more understandable as you can look at the system from a higher level of abstraction.

So technically the Robot framework goes through the test script, executes keywords in the script either by calling your system under test, a keyword library or unwrapping a keyword script and executing that, runs the inputs and verifies the outputs.

Pekka Klärck, author of the Robot Framework, explained this idea more in an e-mail:

The most important reason RF is fully keyword-driven is that it makes the library (i.e. fixture) API very simple. In my opinion the biggest problem with FIT/FitNessese is that different kind of tests need different fixtures and you cannot thus easily reuse fixtures between tests.

Organising and editing tests

Robot framework reads test cases from disk files, and it supports three formats: HTML, TSV and RST. RST is a markup format similar to Wiki markup. In HTML and RST, tables are parsed for test keywords and everything outside of tables is just ignored, allowing you to put additional documentation next to the test cases. TSV allows you to use Excel or a similar spreadsheet for editing your test files. Test files in the same directory automatically become a test suite, and sub-folders are sub-suites, making it easy to organise tests into a hierarchy. In addition to Excel or HTML editors, you can also use the Robot Framework IDE (Python GUI application) to edit and manage the tests.

Robot framework files contain three sections: settings, variables and keywords. Settings define the global test environment – what is the system under test, how it is initialised, which keyword libraries are loaded automatically etc. Variables allow you to assign meaningful names to arguments that will be repeatedly used later in the test, such as a test username, balance, home page URL etc. This avoids the problem of magic strings and makes tests easier to modify and edit. The keyword section specifies test cases that are executed as part of the test. Ideally, the top-level keywords in the test case will represent entire scenarios that are then described using lower-level keywords or test libraries.

According to Pekka Klärck, this makes the Robot Framework easier to learn:

Fully keyword-driven approach also means that the test data is always in same format. This means, for example, that it is enough to learn just one test data syntax, and that it is relatively simple to to create general purpose test data editors. I see tool support as the biggest obstacle for ATTD and this is thus a very important benefit.

Filtering

Robot framework allows you to filter tests during execution based on several criteria, including tags and criticality.

Test tags are free-form textual labels which can be assigned to tests. Examples might be “fast-tests”, or “persistence” or “services”. You can specify a tag during execution to quickly run only test cases marked with a particular keyword. This enables you to establish free-form groups of tests separate from the test-suite hierarchy. So, for example, you can use the hierarchy to break tests down by functional blocks of the system and tags to break them down by iterations, and then quickly execute all tests for a particular iteration regardless of where they are in the hierarchy.

Robot framework also supports marking tests as critical or non-critical. Apart from filtering, similar to tags, criticality can decide whether the test run succeeded as a whole or not when a test breaks. Failing tests marked as “non critical” will not break the execution. This allows you to specify tests in development as non critical and regression tests as critical, so once a functionality is implemented it always has to pass, but while you are working on it the test is allowed to fail.

Compared to other tools

Compared to other tools that presented at the Acceptance testing tools round-up, Robot framework and its orientation to keywords is probably best suited for scripting tasks. It supports building scripts to produce higher-level keywords, for-loops, conditional and parallel execution and variables so it allows you to build up complex scripts without writing additional test fixture code. This empowers testers to do a lot more on their own, describing and verifying workflows in particular. On the other hand, tests feel very technical and less readable than with most other tools. So in a sense, Robot framework is the polar opposite of Concordion, which emphasises specification over scripting and readability over execution.

I would not say that inability to efficiently reuse parts of tests is a problem of Fit and FitNesse, but the attempt to do so is a misuse of FitNesse and one of the most common ways people shoot themselves into a foot with it. With acceptance testing focused on specifications, I always discourage people to reuse parts of acceptance tests in unrelated tests as that points out to the need to view at the system from a higher level of abstraction. However, for efficient scripting automation, this is crucial.

From my perspective, Robot framework is most useful for abstracting UI testing through Selenium and similar libraries for testers. As a further confirmation of that, Robot framework is the only tool from the set of tools we present today which supports saving screen-shots out of the box (through Swing/Jython integration). Its keywords allow you to fully utilise the ideas of domain-specific testing languages presented by Mickey Phoenix at Agile 2008 without writing code – so testers can do a lot more without developers helping them out. Most other systems use test fixtures (in code) to define the meaning of test steps and their execution, where with Robot framework you can do that by defining keywords with keywords and write scripts that have branching decisions, loops, parallel execution.

Links

If this sounds interesting, here are some links for you to continue the journey:

• Robot framework main web site
• Implementing executable requirements with RFDoc and Robot Framework
• Pekka Klärck’s web site

Get notified when I post something new - subscribe via RSS or Twitter!

Trackback URI | Comments RSS