Continuous Testing In DevOps

Testing is very important in the release segment as it measures the level of product quality that delivers to customer/end users.

Testing is the concept that applied in any level of product manufacture/development from large-scale industries to small business. We have seen “Test Ok” as a symbol in all the products that delivered by big brands or small companies as assurance for the end user to use the product for the respective purpose. In fact, certain companies come with a big pamphlet about “How to use?” , “who to use?” , “what level of test performed for that particular product”, “Do the product has any hazardous limit” and final rating about product quality.

The final rating/QA result measures the product total quality that controls whether to deliver the product on the market or reject.


Testing in Software Development:

Quality analysis is not only specific to manufacturing industries or it is not that approach is applicable for software development company. Irrespective of the company, modules, product, practice or methodology, it is applicable to all the deliverables that deliver to customers/end users as it is standard (ISO) that everybody should follow being products development/manufacturers. Hence it is one of the phases in the SDLC also.

Usually, In the SDLC cycle, QA comes into the picture after the development completed. But there are many confusion or debate raised about “whether QA team needs to be part of project workgroup discussion or product design phase?”  It is still a debate in many companies, however, there are companies that take cross training decision and get the QA team also be part of project discussion.

In waterfall methodology, QA team usually prepare the test case either during the code development (if QA part of project discussion) or after the development. But it’s all about how effective the test case scenario prepared and how many days scheduled for the QA team to perform the testing for the bulk set of code that deployed in the testing environment. Due to the large code delivery, they are many challenges faced for getting the environment ready with test data,  system/functional integration issue while performing integration testing, incomplete regression test case scenario due to time constraints, etc but out of all these, least bother about final QA result.  

Types of Testing:

In software testing world, there are many test approaches followed to test the quality of code.

  • Unit Testing: Testing done by the developer to validate the changes based on the personal view.
  • Regression testing: Irrespective of any code changes, regression test will be executed on the whole application as kind of sanity check. Meaning, testing the changes on top of base code or base functionality to ensure no defects found on the base product. Regression testing applicable when there is upgrades & fixes introduced.
  • Integration testing: The code change will be combined with other software modules to perform the test as a group.
  • Functional Testing: The code change will not be considered, however, the testing will be happening to the functionality/Business/client requirements.
  • Load Testing: The code change will not be considered, rather pumping the system with more traffic/load to see the system performance. With this test result, TPS (transaction per second) is measured.
  • Exploratory testing: The test will happen against out of box test case scenarios.
  • Acceptance Testing: The test will happen to confirm whether the requirements are met. Mostly this kind of test performed by the client.

The testing approach in DevOps:

Though there are different types of testing exists before the DevOps process comes as industrial practice, the continuous testing is the approach that encompasses these testings in proper order to ensure that the test happened.


Continuous testing is one of approach to practice the DevOps process, as it is dictated to get fast feedback on the impact of changes. But it does not about automate the testing rather it emphasizes the concept of test early, test faster, test often and then automate the test.  

As I have discussed in the article, The combined team of Developer & tester who are called as software engineer in DevOps world, can parallelly prepare the test case scenarios as code for unit test, regression test, functional test, UI test & load test in order to achieve test early ,test often and automate it.

Automated Test:

The automated test is not new in the software industry, but “what is to be automated” is important, automation testing should be done at the right time in a right way to ensure a high-quality release.

For instance, There is password policy change requirements raised and the development also completed. As part of CI pipeline, the code has been build and next phase is to perform the test.

Continuous Tests in CI Pipeline:

Unit test: The automated unit test can be executed in an isolated form of passing different test case to the unit of modules where the changes were done.

For example, if “passowrd_policy.js” script file changed then that can be tested using simple script wrapping around this .js module or some test automation tools like selenium, testComplete, etc for passing different password as an input to validate the changes. There are tools like JUnit, nUnit, Pytest, etc as language specific module to automate this.

Test Driven Development: Since the Unit test is specific to the particular module. The developer should prepare the test case which can be written in code and that code can be executed on every stage during the development/code changes and this routine can be repeated until the unit test cases executed successfully.

Regression Test: If the unit test passed then the code build can move to regression test. The automated regression test should have always been revisited and updated with new test cases. The test cases can be applied using scripts & tools based on the complete base module. The test should cover almost all the necessary base functionality/code to ensure the changes not impacted the existing functionality/code.

In our example, the regression should be executed on the login page, password reset page & password encryption method, password force to reset & registration page, verify the error pattern in the application/system logs, etc.,

Functional Test: The next phase is to functional test, where the newly created test cases prepared against BRD / CR (business requirements/change request) to be executed.

In our example, the functional test case can be prepared to check whether the changes force all the existing customers to reset the password as per the new policy, What if existing customers that already have password as per new password policy, etc.,

Mostly the functional test can be executed on UI, so it can be automated using tool like selenium, testComplete, etc.

Integration Test: In the phase of integration test case scenario, the pages/modules/services that went through changes should be auto integrated into the co-services & modules without any impact.

In our example, if the new password policy allows special character, it should not impact the underlying system be malfunctioned. Nowadays most of the system integration happening through API. So in order to execute these test case scenarios, we can automate using tools like SoapUI, JMeter & RESTAssured, etc.

Load Test: The load test sometime will not be added to the CI pipeline due to resource limitation. However, the test should have happened at least once before code moves to production.

In our example, the test case can be “the password_policy.js cause any latency issue and makes the customer wait for the login page for a long time”. Performance test can be automated using Jmeter, etc..

Benefits of the Automated test process:

  • Having automated unit, regression, functional, integration & load test as part of CI pipeline, the timely & early feedback sent to the developer.
  • All these tests are part of CI if there is an issue identified. The corrective action will be done on the same day.
  • Reduce the manual effort.
  • Automated process ensures no manual error and ensure reliability.

Continuous Test in CD pipeline:

The continuous testing approach is not about automating the test case, rather it emphasis finding the problem as soon as possible in the meantime (MTTR) from code check-in to release in the deployment pipeline. Hence the automated test added as part of CI pipeline. However, there are scenarios where tester knows that automation cannot be done. So these test cases can be manually tested in continuous delivery (CD) pipeline.

For eg: In banking domain, the reconciliation process is very important to check ledger balance is equal to the sum of customer’s balance. If that does not happen, then manual cross verification should be done by verifying particular account activity to see what goes wrong. Sometimes, the test environment complete accounts details can be sent as a report to finance team who can do 4th eye verification.

Exploratory testing:

The scenario that explained above can be conducted as part of exploratory testing. In most product based companies, before the product deployed in production, the companies engage external tester/internal tester who will verify the application/product in the different dimension by critically thinking, carefully observe the application, creative test case scenarios, design the tests systematically with different diverse ideas. The exploratory testing can be executed manually to perform ad-hoc test quickly and capture the test cases while performing exploratory testing.

Learnings in CD pipeline: The issue/bugs that identified or test cases that captured during the exploratory testing can be added to automated CI test pipeline for continuous testing.

Acceptance Testing:

As part of the continuous delivery pipeline, the code changes can be deployed on UAT (User Acceptance Test) environment, where the client will have access to perform set of use case test by themselves. This test considers as sign-off to deploy the application into production.

In some companies, this test will happen with the concept of beta testing. Nevertheless, in service-based companies, UAT sign-off from the client might require and product based companies follow beta testing.

If you are the developer or tester, I would encourage you to attend this course to get into continuous testing practice

Leave a Reply