Test Automation: Is it Worth the Investment?

Continuous testing is nowadays the key to alleviating some of the most common problem symptoms in software development, such as poor quality and late discovery of project flaws. Tested software means high quality software that is free of bugs and problems that could later disrupt business operations. At the same time frequent software testing (i.e. “continuous testing”) ensures that the changes to the software and the refactoring of the code do not introduce problems. Furthermore, continuous testing facilitates early discovery of problems (e.g., bugs, flaws) at the design and development phases, prior to their propagation in deployment and production. This is directly associated with significant cost savings: A bug fix during the software design or implementation phase incurs a few hundreds of dollars cost (e.g., due to the need for allocating person hours for debugging and refactoring), while this very same bug can cause economic damage of thousands of dollars in case it propagates to production.

The merits of testing have given rise to software development best practices, which prioritize the writing and execution of tests as part of the software development lifecycle. For example, the “test-first” approach mandates that developers first write the test case for a unit or module, prior to writing the respective code. According to Test Driven Development, software is produced based on the following cycle:

• A test is written;
• Code with minimal functionality that passes the test is developed;
• The test is updated and the code is refactored accordingly;
• The cycle continues till all the features are implemented.

TDD is gradually become a cornerstone of agile software development, as it provides the foundation for ensuring the quality of development iterations. As we have explained in earlier post, agile software development is about iterative and evolutionary development cycles, which deliver functional versions of the product at the end of each iteration. Hence, the software delivered as part of each cycle is thoroughly tested. TDD and test-first programming not only ensure that all tests will be in place at the end of an iteration, but also that the release will be tested and functional.

Testing Automation: The Benefits

The need for continuous and thorough testing, gives rise to the automation of the testing procedure. Indeed, manual testing might be feasible in very simple projects, but it’s very expensive and virtually a mission impossible when it comes to continually running test cases for large scale projects. Testing automation has distinct advantages, when compared to manual testing including:

• Time and cost savings: Automated testing is by far faster and more cost-effective than manual testing. For example, a test suite with more than 10.000 tests could need 5-6 testing engineers working for two weeks in order to be completed manually. On the basis of an automated approach, the very same 10.000 tests could be executed by 1 person in 6 hours.
• Broader coverage: Automated testing covers use cases and functionalities (e.g., system level features), which can be hard to cover manually in a short span of time, hence being more effective, especially during regression testing.
• Reliability and consistency: Manual testing is more error prone and requires significant effort in order to consolidate the results in a structured and consistent way. On the contrary, automated tests are much more reliable and can be structured in order to provide consistency and fast consolidation of the results.
• Foundation for continuous testing and integration: The need for continuous testing and integration as part of agile programming practices leads to an explosion of the number of tests, which cannot be executed manually but can be tackled easily using automation testing.
• Shorter time to market: Automated testing accelerates the process of testing the functionalities of the product, which shortens time to market.
• Higher reusability: In several cases there is an opportunity for reusing test scripts (e.g., when they concern “similar” modules or use cases), which can further increase the ROI (Return-On-Investment) of the testing automation process.

Understanding the testing automation process

The testing automation process should typically involve the following steps, which will get you the most of automation:

• Testing Automation Scoping: As part of this step, teams have to specify the overall scope of the automated testing in terms of the business functionalities that should be tested and the tests that have to be written. Furthermore the feasibility of the overall testing task is assessed, including an estimation of the required resources.
• Testing Automation Design & Planning: This step includes the selection of automation frameworks and tools, as part of the creation of an automated testing environment that is compatible to the overall development environment. , identification of the resources (e.g., testing engineers) and the timing and frequency of the testing procedure.
• Testing Automation Development: This step involves the writing of the automation scripts that will support the testing. As already outlined, the production of these scripts is an integral element of agile software engineering processes, especially in cases where the test-first programming and TDD best practices are adopted.
• Execution of Automated Tests: This is the ultimate step where automation scripts are executed and test reports are produced. The execution of the tests is performed using the automation tools selected in the planning step.
• Automated Testing Maintenance and Evolution: This step involves updating the set of automation scripts that comprise the test suite. In the scope of agile software development such updates take place in every release cycle. They may involve revisions and improvements to existing tests, as well as the inclusion of new tests. The goal is to improve the automated testing process, much in the same way the quality and the scope of a product is also improved in every cycle.

Best Practices for Testing Automation

There are a number of best practices for planning and executing the automated testing process, which maximize productivity and the ROI of the testing automation process. In particular:

• Proper scoping of the process and focus on high-value test cases: It important to define the scope of the automation process and of the related testing suites. Test cases should focus on use cases with the highest business value, as well as on use cases that will be executed and tested very frequently. It is likely that 20% of the use cases are used 80% of the time, and these should be prioritized for testing automation. Moreover, test cases that cannot be performed manually must also be included in the automated testing plan.
• Set quality standards: As part of the design and planning process it’s important to set proper quality standards for the testing procedure. You must standardize comments, code indentation and effort logging in order to end-up with uniform high-quality scripts.
• Set, measure and track metrics and KPIs (Key Performance Indicators): It’s important to set proper measures for the success of the testing process. You should be able to track the number of defects, the amount of testing time per release cycle, the number of test cases executed, the level of human involvement (e.g., in person hours), as well as indicators of the productivity of process. Without proper metrics and their tracking, you won’t be able to ensure the effectiveness of the testing process.
• Proper selection of automation frameworks and tools: The selection of the proper automation tools is extremely important for the overall productivity and success of the testing process. Tools facilitate the development of testing scripts, along with required file conversions and the generation of relevant. There is a plethora of tools covering different environments and needs. A prominent example is LoadRunner, which provides testing engineers with an accurate end-to-end picture of the system under test, in order to help the identification and resolution of functional and performance issues. The choice of the automation tools should consider various criteria, including the development and deployment environment of the application under test, the nature of the application (e.g., web, mobile), compatibility with other frameworks used and more.
• Managerial Support: Beyond technical factors, the backing of the automation testing process by the management is always a critical success factor. The adoption of an automation testing culture is likely to put people outside their comfort zone, but also to create disputes within the team. Moreover, the effort to put in place an automated testing process can be significant. Management support is therefore essential in order to alleviate all these issues.

Overall, testing automation is an indispensible component of agile software engineering and a foundation for shortening release and innovation cycles. Therefore, putting these best practices to work, sooner than later, is important and will require serious commitment to get the most out of your automated testing process.