G+D currency technology develops highly complex embedded software that operates in a challenging mechatronics environment. A major challenge has been the manual analysis of our automatic nightly test execution results - in particular with growing test sets and a complex test execution environment. Via an in-house developed tool that combines ideas of patterns and learning we have reduced our analysis time from hours to less than a minute. We believe this idea can drastically reduce time spent on analysis for any tester - even in manual system test - and will show how it can even help to automate processes in other contexts.

Target Audience: testers, developers, test managers, project managers
Prerequisites: Experience in test automation or analysis of test execution results
Level: Basic

Extended Abstract:
G+D currency technology has been developing its banknote processing system (BPS) products for many years now using an agile development approach which is supported by a sophisticated continuous integration (CI) pipeline. Every night thousands of unit tests as well as hundreds of functional tests are executed on our servers to ensure that the subsystem software operates correctly and that no regressions have been introduced during the day by software development. Our most complex BPS subsystem - the coordination subsystem - consists of multiple communicating processes which are developed based on a common platform by 6 independent teams in parallel sprints.Our test system used for functional tests is equally complex and includes simulations of other subsystems. Each functional test covers the entire cycle from subsystem software installation to taking the product through a complete workflow in a virtualized environment, and can take from 30 to 60+ min to execute. In each agile team, one automation tester is responsible for developing new and maintaining existing regression tests as well as monitoring each day the results of nightly automatic test execution. The effort spent on analysis is generally reasonable as long as most regression tests pass, but the required effort easily triples, e.g., when sporadic failures from our complex test environment interfere from time to time. We started realizing that the testers that work independently on similar but different products follow similar patterns in their (manual) analysis. A brilliant idea for a simple test result analysis tool enabled us to slash our time spent on analysis for each tester from hours to minutes.In this presentation, we want to present our pattern based approach and its integration into our CI pipeline, which we strongly believe can easily also be used by other testers to reduce their effort. We will talk about the ways in which we deployed the tool to even further automate our testing process at G+D, e.g., for smart automation of reruns for tests that have failed sporadically. Currently, we are even investigating to use this approach also to solve another big challenge we have been facing, which is to accurately and completely automatically track feature test coverage independent of the test implementation and the tests executed.