CI/CD testing: what is it and how does it work?

The Continuous Integration/Continuous Delivery (CI/CD) concept is widely used and discussed among teams working in software development.

This approach emerged from the need for software to receive updates more quickly while simultaneously having fewer bugs. Building a robust CI/CD pipeline is a huge challenge, and automating steps using these processes – such as unit, integration, functional, end-to-end (E2E), and performance tests – ensures greater reliability in code changes, making deliveries faster and more secure.

What is Continuous Integration?

Being more popular among projects with frequent updates, Continuous Integration (CI) plays a crucial role in ensuring that each component works perfectly, both individually and when integrated. It runs automations at a high frequency, whether once a day or hourly.

Practically speaking, developers build and test the code locally before pushing the changes to the repository. Once these changes reach the repository, a sequence of steps is automatically initiated.

The first, and most common, step is building the latest source code. If everything goes well, unit tests are initiated. If they pass, the build for testing environments is started. At the end of each step, a detailed report is generated with the process status, displaying the number of builds, any bugs that may have occurred, and the total number of tests conducted.

To better illustrate this flow, a CI pipeline performs the following tasks:

• Detect changes in the source code repository (e.g., new commits appear).
  
• Analyse the quality of the source code.
  
• Build a new version of the source code with each job.
  
• Execute unit tests.
  
• Execute integration tests.
  
• Build deployable artifacts.
  
• Generate a status report.

What is Continuous Delivery?

Continuous Delivery (CD) is a continuation of what happens in the CI stage. Both are interconnected because, while CI deals with building and testing, CD focuses on how to deploy all these packages.

Before deploying a new feature, CD includes a flow that involves automated tests, unit tests, API tests, load and integration tests. Each step is performed automatically, ensuring automated tests at the unit, integration, and system levels. As tests can fail at any level and environment, CI/CD must provide a way to quickly report failures to developers.

Depending on the policies and processes defined by teams, developers may follow these steps:

1. Before confirming changes, developers need to check if the current code build was successful. If not, they should fix errors before submitting new changes.
2. If the current build is successful, they should reset their workstation to the build configuration.
3. Create and test locally to ensure that the update doesn’t break any functionality. If successful, the team can confirm the new changes.
4. Ensure that CI is completed with new changes.
5. If the build fails, developers should stop and fix errors in the local environment before returning to step 3.
6. If the build is approved, the team continues working on other items to implement in the future.

What is CI/CD testing?

It’s the phase where quality in delivery is ensured with the least possible errors.

The goal is to push an update or release only when it passes functional requirements tests, remains stable, and is bug free. Typically, this is a fully automated process integrated into the pipeline.

How does CI/CD testing fit into the CI/CD process?

It is much easier to resolve and correct problems while they are still at the root. This type of action prevents code from proliferating on an unstable base, ultimately reducing time and effort in resolving potential errors.

Currently, automated testing tools seamlessly integrate with CI/CD tools, allowing test data to be fed directly into the pipeline. Depending on the tool used, we can decide whether to proceed to the next testing stage based on the results of previous tests, ensuring that at each stage, the team receives a report with results informing the next step.

To ensure greater efficiency in the pipeline, it is advisable to assemble the test suite with faster tests ahead of more complex ones – thus saving more time in deliveries.

When devising strategies for test creation, we must keep in mind the structure of the testing pyramid, balancing unit tests, integration tests, and end-to-end tests.

This well-structured approach to CI/CD testing ensures comprehensive coverage, maintaining efficiency and effectiveness throughout the entire software development lifecycle.

Tools that enable CI/CD testing

There are several tools that facilitate the use and practices of CI/CD testing, but the most popular ones are:

• Selenium
  It is the most popular platform for creating automated tests in web environments. Selenium supports various programming languages (such as Java, Python, and C#). Selenium integrates well with CI/CD pipelines and can be combined with other tools like Selenium Grid for parallel execution.
  
  
• Jenkins
  Primarily known as a CI/CD tool, Jenkins also supports test automation. It provides plugins for integration with various testing frameworks, including Selenium, JUnit, and TestNG. Jenkins can execute automated tests as part of the CI/CD pipeline and generate reports for test results.
  
  
• Cucumber
  It is a tool used for Behavior-Driven Development (BDD) and test automation. It facilitates collaboration between developers, testers, and business stakeholders through the use of executable specifications in plain text, called Gherkin. Cucumber supports integration with CI/CD tools like Jenkins, allowing automated tests to be executed as part of the pipeline.
  
   Discover more about Cucumber Testing in this article!
  

• Appium
  An open-source automation framework designed specifically for automating tests for mobile applications. It supports Android and iOS platforms, and allows writing tests using popular programming languages such as Java, Python, and Ruby. Appium can be integrated into CI/CD pipelines to automate mobile application tests across various devices and platforms.

Automated tests vs. manual tests

When it comes to CI/CD testing, both manual and automated tests are valid. Manual testing requires a QA (Quality Assurance) professional with a keen perception who can explore different ways to test an application, ensuring harmony across simple criteria such as functionality, usability, accessibility, and performance.

However, in situations where a large number of scenarios need testing, which increases the likelihood of human error due to lapses in attention during testing or in the control itself, this can become a factor that prolongs testing time. Hence the importance of automated tests, as they allow us to feed repetitive and complex test cases, saving the QA’s time.

Automated tests accelerate feedback cycles by continuously verifying code. It is crucial to automate the testing process as much as possible. This approach is suitable for executing large sets of repetitive tests and promptly detecting regressions. Types of tests that can be integrated into the CI/CD cycles include unit tests, integration tests, system tests, and regression tests.

Since not all test cases can be automated, it is always recommended to adopt an approach that combines both manual and automated tests to ensure continuous delivery.

CI/CD benefits

By implementing test pipelines in the CI/CD workflow, we ensure greater accuracy in testing, reduce regression times, enhance software quality, and save time in bug resolution.

Optimising these steps opens up more space for teams to explore creativity, bringing insights about the product and allowing individuals to contribute with more innovations.

Here are some of the key benefits of applying CI/CD concepts to software product development:

• Code quality and security.
  
• Increased customer satisfaction.
  
• Faster delivery speed, which comes from well-defined and structured pipelines.
  
• With well-defined automations, the team can redirect their focus to other important matters.
  
• A well-defined pipeline results in more robust and higher-quality products. This, in turn, attracts top talent and clients.

Conclusion

We have come to the conclusion that with well-defined test pipelines, the team can play a crucial role in ensuring efficient and quality delivery.

CI/CD are two recommended DevOps practices, as they address the misalignment between developers and the operational team. With the presence of automation, developers can release changes and new features more frequently, while operational teams experience better overall stability.