In this article, you’ll be introduced to two tools: Speedscale and NeoLoad. Both of these tools offer you a way to load test your applications. This post will compare their ease of setup, development experience, fit within a modern infrastructure, and integration into CI/CD.
Load testing is not a new concept in any way: the term was common even before Google Trends started recording data in 2004. Because of this, engineers all over the world had a bunch of stable tools to use once they started switching their infrastructure to use Kubernetes instead of traditional deployment solutions like bare-metal or virtual machines.
Over time, it’s become clear that Kubernetes not only provides a lot of benefits for deploying applications, but also offers ways to improve their load testing capabilities.
Ease of Setup: Speedscale vs NeoLoad
When you want to get started with Speedscale, you first need to sign up. Once you’ve gotten an account and you’re logged into the dashboard, you will be greeted by a quick-start guide at the top of the screen, where it will stay until you finish the simple three-step guide. You will first be given instructions on how to install Speedscale into your Kubernetes cluster, which means you’ll be installing an operator in your cluster.
This can be done either by using Helm, or by installing Speedscale’s speedctl CLI tool. With the CLI tool, you simply run speedctl install, and the CLI tool will guide you through the installation process. No matter which approach you decide on, you’ll get detailed instructions, and have an easy time installing the tool.
To describe the setup experience of NeoLoad, it’s important to first understand NeoLoad’s high-level infrastructure. NeoLoad consists of two distinct parts, a controller and a load generator. As the name suggests, the controller takes care of controlling tests, like getting them started, gathering results, etc. The generator takes care of sending requests to your service. These two parts can be installed onto the same system without issue. However, it’s also possible to split them onto different systems.
Assuming you want to run NeoLoad on a single system, you install NeoLoad as a desktop GUI application. Before you can create and manage tests, however, there are a few more steps needed. NeoLoad works by recording user traffic, which is accomplished by sending traffic through a proxy that’s created when you download NeoLoad. So, in order to use NeoLoad, you have to configure your browser or your system to use a proxy, which means you will also have to install an SSL certificate (provided by the NeoLoad installation), in order to make things work. Once these steps are done, you can start creating tests.
There’s no doubt that for Kubernetes load testing purposes, Speedscale is the clear winner. Speedscale installs directly into your cluster with an operator, whereas NeoLoad has to run on a separate server. Of course, this can simply be a Pod that’s running in your cluster, but there is no official support for this use case. In any case, you have to make sure that NeoLoad can access your application via a URL, which is easy enough if you’re running your load generator inside of your cluster, but if you’re running it outside your cluster, you need to make your application publicly available. This could be important and so is something to keep in mind.
Developer Experience: Speedscale vs NeoLoad
The first time you log into Speedscale it can seem a bit confusing, as the user needs to understand some tool-specific terms and concepts, like what a test config is, as well as a report. It won’t take more than an hour or two at most, and you’ll have a good grasp of what these terms mean and how they’re used. With the quick-start guide to help you, it won’t take long for you to have your first test running. As long as you have prior experience with Kubernetes, you should be able to easily create a test.
Once you’ve instrumented your applications with the Speedscale sidecar, which takes place as part of the speedctl install process, Speedscale will use the sidecar proxy to record your traffic, saving the recorded data in the Speedscale dashboard. All the tests you create will be based on this recorded traffic, so you will be getting traffic that’s based on actual traffic that’s gone through your cluster.
To create a test in Speedscale, you add certain annotations to your Deployments in Kubernetes. This can either be a Deployment that’s already in your cluster, or it can be a new Deployment that you deploy with the annotations already in the manifest file. For a simple test, the annotations will look something like this:
annotations:
replay.speedscale.com/snapshot-id: "1e4c3935-9acb-1fa7-af1b-9c557e17ed55"
replay.speedscale.com/testconfig-id: "standard"
replay.speedscale.com/cleanup: "inventory"The snapshot-id refers to the snapshot you want to base your test on. As mentioned previously, Speedscale records all the traffic going in and out of your services. A snapshot refers to a given timeframe of recorded traffic. The testconfig-id refers to the test config you want to use; the standard test config is a one-to-one replica of the recorded traffic. There are also test configs that will replicate your traffic 100 times, and some that will introduce chaos. You’re also able to create custom test configs. Finally, cleanup defines what needs to happen after the test is done, with inventory being the option that returns your cluster to the state it was in before the test started.
NeoLoad is different from Speedscale, in that it uses a desktop GUI application to manage everything. With NeoLoad, you need to understand a lot of different parameters; fortunately the company provides a YouTube series that can help you get started.
As mentioned previously, you need to configure a browser to use the NeoLoad proxy before you can start creating tests, and you need to make sure that the system you’re using to create the tests (like your local PC), has access to your application. With those requirements met, you create a test by opening up a browser instance from a button click in the NeoLoad GUI. With the browser open, you simply click around your website like your user would. This part can be a bit tedious, as NeoLoad recommends naming each click, which it calls a transaction. So, the process for creating a test in NeoLoad is to name your transaction, click on the website, name your transaction, click on the website, etc.
Once you have created a test, you can run it by scheduling it on a system that has a load generator, which could be the same system that you’ve used to create the test. It’s also possible to hook your NeoLoad instance up to NeoLoad Web, and run a test there.
Which tool has the best developer experience? It really comes down to what you’re used to and what your goals are. While NeoLoad’s GUI allows for recording transactions via the browser, Tricentis makes their NeoLoad Web available as the main management interface. However for Kubernetes load testing, Speedscale is arguably the best choice, as it matches the principles of Kubernetes, like being headless and working through a web interface or a CLI tool.
Place in a Modern Infrastructure
In the last section, you saw that there’s a major difference between how Speedscale and NeoLoad go about managing each tool. Speedscale integrates directly into your Kubernetes cluster and is managed through a web interface or via a CLI tool, whereas NeoLoad relies on desktop installations. This raises the question, which tool is best in a modern infrastructure?
Well, what is a modern infrastructure? Ask ten engineers and you’ll get ten different answers. Still, there are some core principles that most developers can agree on. The core principle is that a modern infrastructure can be managed either as code, or at the very least through scripts. Any system that requires an engineer to manually click through installation options doesn’t scale, and should not be a part of a modern infrastructure.
Unfortunately, by default NeoLoad doesn’t fit into this category. There are options for running NeoLoad in a more modern way, like via Docker images, but it’s clear from the documentation and resources provided by NeoLoad that their focus is not on providing a solution that integrates into a modern Kubernetes infrastructure. Instead NeoLoad focuses on the age of monolithic solutions. For headless test creation as well as Postman imports, NeoLoad does offer NeoLoad compose, designed to be used in conjunction with NeoLoad’s command-line utility.
Integrating into CI/CD
Speedscale knows that integrating into CI/CD is an important feature for any infrastructure engineer. By running tests as a part of your CI/CD pipeline, you can shift left and become aware of any possible issues, even before you merge new code into your application’s codebase. In Speedscale, running a test will give you a report, which you can then use to check whether your test was a success or not. The report contains a status field, which you can programmatically check, and use as a gate in your CI/CD pipeline.
Whether a load test is successful or not is defined as part of a test config. For example, the standard test config specifies that the response from your service should match exactly what was recorded in the snapshot.
Because a test in Speedscale is triggered by adding annotations to a Deployment, or deploying a Deployment with the given annotations, it’s very easy to use in your CI/CD pipeline. You can use exactly the same deployment procedure that you use in development or production—you just need to make sure the correct annotations are added during the deployment. Speedscale provides official documentation on implementing CI/CD in a number of CI/CD tools, like GitLab CI/CD and Azure DevOps.
NeoLoad integrates into your CI/CD pipeline in much the same way as Speedscale, in that you can use the NeoLoad CLI tool to run your tests. The major difference is that with NeoLoad you have to manage your load generator yourself, whereas Speedscale makes it easy to just deploy new Pods in your container, that then run the full test.
Conclusion
Again, load testing is not a new concept, but there are certainly ways new ways to approach the process when using Kubernetes. Both the tools described in this post are reasonable options for load testing in general—which one is the right choice for you will depend on what your goals are.
If your goal is to be as modern as possible, and if you’re already using Kubernetes a lot in your infrastructure, then it’s likely that Speedscale will be the best option for you. On the other hand, if you’re working with enterprise and many different legacy systems, NeoLoad can also be a good choice.
Are you still unsure about the best tool for Kubernetes load testing? Check out our other comparison, between Speedscale and Gatling.