At the StarEast conference in Orlando earlier this month, Kristan Vingrys, global test practice lead at Thoughtworks, presented an experience report on setting up a build pipeline in an environment with multiple development streams, 3rd party delivery teams and a mix of legacy mainframe and new code. Here are some of the interesting challenges they faced and solutions they implemented:
Slow feedback from monolithic tests
At some point, as they built up the functional test suite, the tests started taking a long time to execute. To prevent slow feedback, they divided a single build process into two: a development build that compiled software and ran unit tests, and a test build that executed longer functional tests. This separation quickly caused developers to start ignoring the test build, running only the unit tests before checking new code in and waiting only for the development build process to finish. As a result, the functional test build was frequently broken. To address that, they split a monolithic test pack into several packs targeting individual functional areas, so that developers could quickly execute the tests relevant for the work in progress. In addition, this allowed them to parallelise the execution of the functional test pack by running different functional areas on different build agents.
Upstream/downstream issues
There were several vendors working on the same project, and one of the upstream teams, writing the service layer, worked in waterfall with longer cycles and no regard for automated testing. As a result, their software was often broken which then caused problems for downstream teams building front end components, that wanted to run tests during development. To address this problem, the downstream front-end team built tests for the service layer, but the services team did not care too much about those tests. “Services team initially didn’t accept test failures as they were front-end tests,” said Vingrys, adding that “pain needs to be felt by the group that caused it.” To solve this problem, they got both teams to agree on the ‘contract tests’ and split the build process even further by introducing a pre-integration stage. In that step, the services team code was checked by contract tests before integration with the front-end code. This made the services team “feel the pain”.
In addition, the downstream team developed a ‘magic proxy’ that would record requests and responses of the services layer during testing. They could then use the proxy to replay correct responses while the services code was broken, to execute automated tests in isolation. This sometimes caused issues when services changed but tests were still running against old recorded responses. A key lesson they learned from this is that the proxy recordings need to be refreshed daily to ensure that the responses are synchronised with changes to service code.
Multiple vendors
Troubleshooting was a problem because of a complex environment and different vendors, some of which had fixed cost contracts and some were working on time and material costs. Vingys said that it resulted in “defect tennis”, responsibility being pushed from one vendor to another. To address this problem, they established a role of a central troubleshooter, who could understand all the systems. As a consequence, they ended up creating a central logging solution and making logging across multiple systems consistent, in particular around time and data. This enabled them to identify a transaction across different logs.
They also worked on creating artefacts that would promote joint ownership, for example contract tests, aiming to change the culture. “We celebrated success together and accepted failure together”, said Vingrys. They started to have regular meetings every morning across all teams, to go through all the issues. As a particularly useful idea, Vingrys pointed out that they were not discussing defects causes, but where is the best place to fix something and who would be able to do it. By jointly deciding where is the best place to fix something, they avoided the blame culture and defect tennis.
Supporting multiple environments
When the team started integrating with third party services on a separate environment, they realised that hard-coded values and expected data outputs were valid only on a single test environment. To address this problem, they separated configuration from tests, splitting tests into four components: test data, test intention, test implementation and test configuration.
As another key lesson to deal with multiple environments, Vingrys pointed out that it is important to deploy to all test environments regularly. They initially deployed to the integration environment only for exploratory testing and third party integrations, which wasn’t often. This caused huge issues every time they wanted to deploy. By deploying to all the environments and running tests frequently, they were able to isolate and solve small issues.
Regarding test data to support multiple environments, Vingrys pointed out that a particularly valuable lesson was not to insert data directly in the database or a legacy system, but through application or service code that would be normally responsible for managing it. “Sticking the data directly in the database is quick, but when the business rules changed tests broke because of invalid data and we spent a lot of time fixing that.”
Yvette Francino has a bit more on this in her interview with Kristan Vingrys.
|
|