One of the top most priorities Businesses nowadays focus on, is to create innovative, business-differentiating technology products in the least possible time frame. They don't demand or rather need perfect software. Inorder to make this possible we need quick feedback on whether the latest innovations would work flawlessly or would it fail in production and also we need to make sure that the changes so made didn't break the existing functionality.
We understand the fact that every software needs to be thoroughly tested to ensure its quality before it goes into production. But in case of a continuous delivery pipeline where code changes and deployments are very frequent, it's virtually impossible to manually execute all the designed test cases for a small change in the codebase.
This is where continuous testing comes in. The process of executing automated tests as part of the software delivery pipeline in order to obtain feedback on the business risks associated with a software release as rapidly as possible is called Continuous testing. Or in other words continuous testing is a procedure of testing early, testing regular, test everywhere and automate. Continuous means undisrupted testing done on a continuous basis. A software change continuously moves from Development to Testing to Deployment in a Continuous DevOps process,. The Continuous Testing process helps us to find the risk, address them and improve the quality of the product.
So, from what I defined above you got a pretty decent idea that, well, it's all about test automation, right?. I hate to break it to you, but it's not that simple. Don't get me wrong test automation is necessary for continuous testing but its not merely enough. While test automation scripts are used to produce a set of pass/fail test cases in respect to the application requirements or user stories. Continuous testing, on the other hand focuses on the business risks and provides insights on whether the software can be released to production. Beyond test automation, continuous testing also focuses on aligning the tests with business risks and also performing exploratory testing to expose the critical issues in early iterations. It's not just a matter of more tools, or different tools. It requires a deeper transformation across people and processes as well as technologies.
Create the test suite. Here tester intuition is the key metric when creating the test cases. Then, execute them, either manually or by means of automation testing. Testers must ensure that the test data being used is apt. At this point any API testing if required is done by developers.
Perform a complete risk assessment and this forms the basis of test cases generation. Test automation now follows a model based approach, where in the run time behaviour of the software is dictated by a model already in place.
A session based exploratory testing is conducted to uncover the potential risk which goes undetected in model based approach. API testing is done by testers if functionality is exposed via API. Test automation is introduced into Continuous Integration through initial integrations with build and deployment tools.
Service virtualization is used to perform sophisticated testing of the API. Test data management ensures continuous stable availability of test data for continuous testing. At this point tests can be continuously executed as part of the delivery pipeline thus providing quick feedback and insights regarding business risks associated with the release build.
Comprehensive test automation has been established and is supported by sophisticated, stateful service virtualization and test data generation/provisioning. Metrics are in place to monitor and continuously improve the effectiveness of the software testing process. Continuous Testing is fully integrated into Continuous Integration and Continuous Delivery pipeline.
Experitest is the continuous testing platform for mobile and web apps, enabling you to run tests on over 2,000 real mobile devices and web browsers. It integrates seamlessly with your development, testing, and continuous integration ecosystem, and is fully compatible with Appium, Selenium, Jenkins, Travis CI and more.
Jenkins is a Continuous Integration tool which is written using Java language. This tool can be configured via GUI interface or console commands.
Download link: https://jenkins.io/
Travis is continuous testing tool hosted on GitHub. It offers hosted and on-premises variants. It provides a variety of different languages and a good documentation.
Download link: https://travis-ci.org/
SonarQube is an open-source platform developed by SonarSource for continuous inspection of code quality to perform automatic reviews with static analysis of code to detect bugs, code smells, and security vulnerabilities on 20+ programming languages.
Download Link: https://www.sonarqube.org/
Selenium is open-source software testing tool. It supports all the leading browsers like Firefox, Chrome, IE, and Safari. Selenium WebDriver is used to automate web application testing.
Download link: https://www.seleniumhq.org/
Appium is an open source project and has made design and tool decisions to encourage a vibrant contributing community.
Website Link: https://www.appium.io
Micro Focus Fortify Static Code Analyzer reduces software risk by identifying security vulnerabilities that pose the biggest threats to your organization. It pinpoints the root cause of the vulnerability, correlates and prioritizes results, and provides best practices so developers can develop code more securely.
One of the greatest advantages of continuous testing is the quality and speed of the release that goes into production.
Continuous testing follows a shift left approach where in testing is shifted left into the early stages of development as soon as a feature is implemented.
The main challenges that we would be facing are,
We should make sure that proper time and resources are allocated towards creating and maintaining the test automation scripts and to create a testing framework that supports reuse and data driven testing.
Project owners should make sure that the test engineers have the know how to automate complex test cases which usually pass through different technologies (SAP, API’S, Mobile devices, Main frames etc.)
One of the things testers should be wary of are false positives. In the initial stages, it's easy to manage the false positives, but as the test suite gets bigger and test frequency becomes higher managing false positives becomes a daunting task and in the end, teams tend to give up on test automation.
Continuous testing in Software Engineering, is a process of testing early, testing often, testing everywhere, and automate. The old way of testing was a handoff centric and the software is handed off from one team to another. Some of the popular Continuous Testing and Integration tools are Jenkins, Travis, and Selenium. Actionable feedback is delivered according to each stage of the delivery pipeline with Continuous Testing.