Creating an Executable Specification from existing manual test scripts might seem as a logical thing to do when starting out with Specification by Example and Agile Acceptance Testing. Such scripts already describe what the system does, and the testers are running them anyway, so automation will surely help. Not really — this is in fact one of the most common failure patterns.
The problem is that manual and automated checks are affected by a completely different set of constraints. With manual testing, the time spent preparing the context is often a key bottleneck. With automated testing, people spend most time on understanding what is wrong when a test fails.
For example, to prepare for a manual test that checks user account management rules, you might have to log on to an administrative application, create a user, log on as that new user to the client application, and change the password after first use. To avoid doing this several times during the test, manual scripts often reuse the context. So you would create the user once, block that account and verify that the user cannot log on, reset the password to verify that it is re-enabled, then set some user preferences and verify they change the home page correctly. This approach helps a tester run through the script quicker.
With automated testing, time spent on setting up the user is no longer a problem. Automated tests generally go through many more cases than manual tests, and when they run correctly nobody is really looking at them. Once a test fails, someone has to go in and understand what went wrong. If a test is described as a sequence of interdependent steps, it will be very hard to understand what exactly caused the problem, because the context changes throughout the script. The fact that a single script is checking ten different things also makes it more probable that the test will fail because it is affected by lots of different areas of code. In the previous example with user account management, if the password reset function stops working, we won’t be able to set the user preferences correctly. If we had ten different, smaller, focused and independent tests instead of one big script, a bug in the password reset function won’t affect the test results for user preferences. That makes tests more resilient to change and reduces the cost of maintenance. It also helps us pin-point the problems quicker.
Instead of plainly automating manual test scripts, think about what the script is testing and describe that with a group of independent, focused tests. This will significantly reduce the automation overhead and maintenance costs.
|
|
Absolutely right – I’ve always thought that tests should be as small as they possibly can be to prove the specific condition / requirement under test, but no smaller
Arguably though, if the testers are going to the diligence of step level test scripting then they should be writing the steps to match the overall test condition / acceptance criteria and not documenting intra-test dependencies just because it saves time to execute them in that manner.
Experienced manual testers will know which shortcuts to make in execution, but scripting this way makes it harder for anyone else to pick up and interpret their tests (not to say it isn’t done mind!).
In any case they need to have documented the high level test description / condition / acceptance criteria that they wish to test with its expected result and the test automator needs to capture and automate to this in all tests. If they haven’t documented this then you’d probably be better off ignoring the scripts and assessing what coverage and set of scenarios they wanted before commencing an automation piece – from shit you can only make a shit pie I believe I recently read!
Before automating, it is worthwhile considering the whole set of tests to be automated and look for patterns and commonalities between tests as this allows you to build a structure from the outset that gives modules that can be re-used and called across tests and so you can create set-up and teardown for each test or suite of tests. Doing this also presents a good time to look for duplication, testing multiple things at once and dependencies within and between tests – breaking tests down into smaller, more focused tests as required.
One added step can be to create smoke tests for these core modules and execute them with each new build so if a failure occurs that is likely to have widescale downstream impact you can tell immediately which avoids wasting time investigating the same failure in many tests repeatedly.