A Developer’s Guide to Continuous Performance Testing

TL;DR

Today’s fast paced development environments require new approaches to testing. Enter: continuous performance testing. Unlike traditional performance testing, continuous performance testing is part of every build, to monitor the application continuously under increased load. With continuous performance tests, organizations with lots of users can expect to improve the customer experience, prevent major outages, and find and resolve issues faster. To get started with continuous performance testing, companies should already have a CI pipeline in place. Tools like Speedscale, with its traffic replication capabilities, can help get continuous performance tests off the ground faster and with less effort.

Overview

One of the most important phrases of DevOps practices is “Test early, test often.” Most developers know to get early feedback from automated functional tests, like unit and integration tests, but it’s equally important to check non-functional requirements as well, including system indicators such as security, scalability, and performance. Performance requirements like response time, stability, and speed are especially important. That means you need automated performance testing as part of your test process.

As markets become more saturated, companies no longer have the luxury to postpone performance testing until the end of the development process when all features are developed. Continuous performance tests have been gaining popularity as more teams realize its impact.

This article will explain what continuous performance testing is, why it’s important, and share best practices and what you need to get started.

What is continuous performance testing?

Continuous performance testing is the continuous monitoring of the performance of an application under increased load. It’s typically done on every code commit, eliminating the need for manual performance tests that can be time-consuming and expensive. The key is to perform tests continuously, meaning you should add new test cases and update old ones as the application evolves so that your performance testing suite captures all relevant scenarios.

Are there different types of performance tests?

There are several types of performance testing, but one that is often used is load testing. Load tests are designed to simulate the activity of multiple users accessing software simultaneously. Similar to stress testing, the goal is to test the limits of how many requests the application can process without jeopardizing the system’s stability. Load tests are often automated with scripting languages for better execution control and accuracy.

Who can benefit from continuous performance testing?

This depends upon the company’s needs, but generally speaking, some companies will find continuous performance tests more valuable than others. These companies typically have:

• A large user base
• High volumes of regular interaction and/or seasonal spikes in traffic
• Invested a lot of time or money into a project that has a long lifespan
• A large amount of staff available to them

Each company’s needs will differ in regards to what they need from a continuous performance testing system.

Devops teams that work on small applications that don’t often see high spikes in traffic probably don’t need to invest in continuous performance testing. For example, if the application is a browser game where user actions are reasonably predictable and external inputs are moderate, there’s probably no reason to introduce continuous performance tests. Continuous performance testing for this type of application would likely reduce overall productivity more than improve anything. Instead, teams with these characteristics can focus more on doing periodic end-to-end tests, which gives them enough data to make meaningful decisions about architecture, trade-offs between the size of the application, and heavy workloads.

Benefits & challenges

With continuous performance testing companies can:

• Prevent major outages
• Improve the customer experience
• Ensure that performance service-level agreements (SLA) are met
• Ensure that performance doesn’t slip over time
• Find and resolve application performance problems faster

However, there are also challenges that can arise:

• It’s not always easy to automate
• It can be difficult to find test cases
• You can’t load test everything
• You need to keep your tests up-to-date
• It isn’t always possible to test on live systems

How is it different from traditional performance testing?

When you mention performance testing, most developers think of the following steps:

1. Identify all the important features that you want to test
2. Spend weeks working on performance test scripts
3. Perform the tests and analyze pages of performance test results

This approach may have worked well in the past, when most applications were developed using the waterfall approach. However, the times have changed. The waterfall approach is now mostly a thing of the past. Software development has become more agile, and as more teams use devops practices to develop and deploy their applications, the need for testing has changed—testing is now a part of the development cycle.

Traditional performance tests might be done after your release cycles or at certain milestones, rather than as part of the CI process. However, continuous performance tests can be done often and as part of every build.

Traditional load testing focuses on how a specific version of the software performs in peak load situations. Continuous load testing ensures that every new version of the application will perform well when in peak load conditions.

If a new version of the application doesn’t meet the necessary performance criteria, the whole build fails. After that, you need to go to the latest code changes and check and fix the performance issues. That means you can use continuous performance testing to validate code in CD pipelines, ensuring that there are no regressions in the performance of applications. Again, the goal is not just finding problems early, but being able to fix them quickly before they affect customers or users negatively.

While some companies find that continuous load testing is a great idea, some factors make using this method difficult, such as a shortage of production-like environments for different API versions and the lack of test data around permutations of traffic.

Why you need continuous performance testing

According to the Akamai Performance report, 49% of customers expect webpages to fully load within two seconds. 18% of customers expect instant page load. In this day and age, users believe that waiting more than two to three seconds for a page to load is unacceptable. When they encounter issues, they’ll either a) leave or b) be the first to let you know about any sluggishness or time-out issues. Laggy sites lead to poor experiences, which can lead to customer churn and revenue loss. So it’s important to measure performance now, not after a major outage.

49% of customers expect pages to fully load within 2 seconds.

With continuous testing, you continuously keep an eye on how well your application responds to load. As a result, you can catch performance-related problems early and fix them before they become significant issues.

To summarize the need for continuous performance testing:

• It ensures that your application is ready for production
• It allows you to identify performance bottlenecks
• It helps to detect bugs
• It helps to detect performance regressions
• It allows you to compare the performance of different releases

Performance testing should be continuous so that an issue does not go unnoticed for too long and hurt the user experience. Continuously testing will show you what your server load looks like at any given time, thereby giving insights into servers’ capacity limits and bottlenecks.

How to start continuous performance testing

To begin continuous performance testing, first make sure that you have a continuous integration pipeline, or CI pipeline, in place.

Step 1: Collect information from the business side

You need to consider what amount of requests you must be able to handle in order to maintain the current business SLAs. Also, you need to think about critical features of your application: is that the login, order processing, or checkout functionality?

Step 2: Start writing performance tests

Usually, the most straightforward approach to take is to start with testing the API layer. You can do this using tools like Speedscale, BlazeMeter, ReadyAPI from SmartBear, and Apache JMeter. There are plenty of tutorials on installing these tools, so this should not be overly complicated. Also, remember to store the tests under your main repository and treat them as first-class citizens. That means that you should pay attention to their quality.

Step 3: Select your use cases

The next step is to identify the scenarios you want to test. Here are some tips on how to write the best scenarios:

• Cover the most critical areas of the system first
• Base your test scenarios on the most realistic user usage
• Test for the end-to-end user experience
• Use concrete numbers instead of vague terms like “a heavy load”

Step 4: Collect results and fix issues

Finally, make sure to collect all your test results into a report that will be easy to read and understand. It’s vital to have meaningful reports so that you can plan your next steps well. The cycle of continuous performance testing doesn’t end once you collect the results though. You need to add all the performance issues to your product backlog with a plan to fix them accordingly. The results of the initial performance test represent the baseline for all future tests.

Ideally, you would perform the above process in a test environment that closely mimics the production environment, which is where tools that replicate production traffic, like Speedscale, come in.