In this article, you’ll be introduced to two tools, Speedscale and JMeter. While they both provide the same functionality, namely the capability of load testing applications, they go about it in very different ways. This post will compare their ease of setup, the development experience, their integration into CI/CD, and how well their documentation is structured.
At a certain point of setting up your infrastructure and configuring deployment pipelines, you have to consider whether you should be implementing load testing as well. There are many ways to do load testing, and many different tools for this exact purpose have been around for years.
With load testing, you can validate that your application is able to withstand the load that you’re expecting it to, thereby decreasing the chance that any spikes in traffic will bring down your services once you get them deployed. If you’re running your applications in Kubernetes, there are now also tools that work specifically with Kubernetes load testing, which can help you validate that everything inside your Kubernetes cluster is interacting as you expect.
Ease of Setup
The first thing to do in order to get started with Speedscale is to sign up for the product. With this step completed you have access to the Speedscale dashboard, and you’re ready to get started. It’s assumed that you have knowledge of Kubernetes; you will be presented with an easily understandable quick-start guide that will get you through the installation process, as well as introducing the most important aspects of the tool.
The installation itself can be accomplished in one of two ways. You can use Helm, in which case the quick-start guide will show you what values need to be set, and define the values themselves. This will allow you to copy a simple helm command, so that you can get set up with Speedscale in just a few minutes.
The other approach is just as easy, which is to download the speedctl CLI tool. With the tool downloaded, you just need to run speedctl install, which will ask you a few questions, and again within a few minutes you will have the Speedscale operator installed.
As you can see, Speedscale integrates well into your Kubernetes installation, where JMeter has a somewhat different approach. JMeter is a relatively old application, with the first version being launched in 1998. While a great job has been done to maintain the tool and make sure it still works well on any system, it’s clear that not much has been done to match modern engineering standards.
To get JMeter set up, you first have to download the release file. Also note that to run JMeter, you need to have Java 8+ installed on your system. Once you’ve downloaded the release file and unpackaged it, you will find a jmeter.bat or jmeter.sh file in the /bin folder of the directory you’ve just downloaded.
Depending on the OS you’re running, clicking on either one of these files will open up the JMeter GUI. JMeter is not actually installed on your system: rather, it’s a script for you to use. To use JMeter from the terminal or from Powershell, you will have to manually add it to your PATH. All in all, getting JMeter set up is very easy and doesn’t require much prior experience with anything.
However, with JMeter you do need to be aware of how scalable the installation procedure is. As can be seen from the installation procedures, JMeter is positioned more as a tool to use locally, running load tests whenever you might need one. Speedscale, on the other hand, integrates directly into your Kubernetes environment with a Kubernetes operator, thereby allowing for deeper and more advanced use cases.
Developer Experience
As with any tool, Speedscale has some specific terms you will have to get familiar with. You will have to learn what a test config is and what it can do, as well as learning what a Speedscale report is. However, this shouldn’t take long, and in a matter of a few hours at the most you’ll be ready to start using Speedscale in your infrastructure.
As part of the speedctl install procedure, your applications are instrumented with the Speedscale sidecar. This sidecar will act as a proxy for your service, and record all the traffic going in and out. When you then want to create a test, you create a snapshot based on this recorded traffic. This means that you will be getting tests based on actual traffic that’s gone through your cluster, rather than manually creating your tests.
Creating a test in Speedscale is done by using the “Save Traffic” button in the web UI or adding annotations to your Deployment in Kubernetes. It’s up to you whether to use a Deployment that already exists in your Kubernetes cluster or a new Deployment you’ve made. Adding annotations to an existing Deployment makes sense if you want to perform a quick Kubernetes load test, whereas deploying a new Deployment with the given annotations is likely something you will do in a CI/CD pipeline.
Your developer experience with JMeter will heavily depend on your background and what kind of tool you’re looking to use. The tool itself is good and liked by many, for good reason. It performs a load test and has a lot of options you can configure, like the number of virtual users, parameters, and much more. However, if you are looking for a tool that’s modern and sleek, and matches the kind of modern principles that you find in Kubernetes, JMeter is unlikely to be your tool of choice.
There are two major reasons for this: JMeter is based on Java, and it relies heavily on a GUI. Both of these are characteristics that don’t fit into a modern DevOps-focused organization. It needs to be said that JMeter does have a CLI tool, but most of the documentation you’ll find focuses on creating and configuring tests with the GUI, and then using the CLI tool to run the tests.
With Speedscale, you can either use the Speedscale WebUI, or you can use the speedctl CLI tool. Either option is as good as the other, and the one to use will depend purely on your preference.
Integrating into CI/CD
More and more developers are wanting to shift left, and catch as many errors as possible during the development phase, rather than waiting until the application is in production. One of the best ways to do this is to integrate any possible test, including load tests, into your CI/CD pipeline. Both JMeter and Speedscale follow the same principles when it comes to CI/CD. To integrate with any CI/CD tool, you will have to run the CLI tool through a shell in your pipeline.
However, there is a major difference in how easily the two tools do this. With Speedscale, you will find clear instructions on implementing CI/CD for a number of providers. Even if your provider isn’t found in the documentation, it is fairly easy to determine which shell commands need to be run by looking at the documentation for other providers, and thereby getting started with your own.
With JMeter, you will struggle a bit more, partly because there’s no official documentation on how to integrate with any CI/CD provider. You will be able to find information on how to integrate with Jenkins, however when looking at the documentation you will see that it partly relies on a Jenkins plugin. Because of this, it’s not as easy to transfer the integration instructions to another system.
Documentation
The Speedscale documentation is not as polished as the documentation you typically find from other companies, like Google or Amazon. However, you can be sure to find answers to most of the questions you may have. You will also find that the documentation is clear and concise, and helps you get answers to your questions quickly.
You will quickly see the structure of the documentation, which makes everything easy to find. However, if you don’t want to click through menus to find relevant information, there’s a helpful search button.
The Jmeter documentation is aptly titled the "User’s Manual." This is very properly named, as the documentation isn’t presented as a Q&A as you’ll find on many other sites. Rather, the JMeter documentation is divided into what are essentially chapters, and you need to read through all the documentation like you would read through a book.
Neither of these approaches is better than the other, but Speedscale’s documentation definitely feels more modern, where JMeter gives the impression of a 20-year-old tool that focuses on being solid rather than modern.
Conclusion
If you’ve gotten to the point in setting up your infrastructure where you want to add load tests, and you’re trying to choose between Speedscale and JMeter, we hope this comparison has been helpful. Both tools are great options, but there’s no doubt that Speedscale is the optimal choice for Kubernetes load testing in particular. It’s a more modern tool that works along the same principles as the cloud and Kubernetes itself, and integrates more deeply into your infrastructure as a whole, rather than being a tool that works on the side.
If you’re still in doubt about the best tool for Kubernetes load testing, you can check out our other comparison, between Speedscale and Gatling.