GoReplay vs. Speedscale: Kubernetes Load Test Comparison

These days, there are many different types of test you can run to safeguard the performance of your services. Unit tests, end-to-end testing, and of course load tests. The type of test you choose to run will will be strongly linked to your desired outcome.

Historically, it’s been common to test your services by generating test traffic, meant to replicate how real users would use your product. However, quite often you’re not getting an exact replica of real user traffic, and you risk not testing all the important parts of your application.

To avoid this risk, the principle of shadowing has been invented. Essentially, a tool that implements shadowing will allow you to record real user traffic, and then later replay it against any service.

Both Speedscale and GoReplay boast this feature as one of the unique selling points of the product, and as such, this will be the main focus of this comparison.

Benefits of Shadowing

To properly understand how these two tools will be compared, it’s important to fully understand what it means to shadow traffic, and what advantages you want to get from a tool that implements shadowing. Some of the benefits of shadowing traffic are:

• Avoid having to write tests
• Test networking rules such as the API gateway or Ingress level
• Use real user traffic to generate tests
• Use traffic to test autoscaling rules
• Use recorded traffic to test circuit breaker patterns/failovers

It can already be said that both these tools support all these advantages. The major difference that will be explored throughout this article, is how easy the tools are to setup, what they’re like to work with on a daily basis, and whether they offer additional benefits on top of the ones just listed.

Ease of Setup

When you want to get started using Speedscale, there a few steps you need to go through. Luckily, Speedscale provides you with a great quick start guide to help you on your way. This guide is divided into three distinct sections:

• Install
• Observe
• Replay

Install is the relevant step here, and shows you how to install Speedscale. It’s possible to install Speedscale either by using Helm or by utilizing Speedscale’s own speedctl CLI tool. In both cases, the quick start guide will provide you with the exact commands you will need, which you can then paste into your terminal.

Given that Helm is one of a number of installation options, it should be clear that Speedscale does not work by installing onto your local system, but rather it installs directly into your Kubernetes cluster as an Operator. Once you’ve used either the Helm chart or the speedctl tool to instrument your services with Speedscale, you’re ready to get started.

GoReplay has a somewhat simpler installation process. To get started, you need to download the latest binary from the release page. With the binary downloaded, you then need to add it to your PATH. On Linux, this could mean moving it to /usr/local/bin. At this point, you’re ready to start using GoReplay.

There’s no doubt that GoReplay is the easiest tool to set up. However, part of the reason for this is that it comes with less functionality, and also doesn’t integrate directly with Kubernetes. Speedscale was developed to be a strong contender in the Kubernetes space specifically, whereas GoReplay is more focused on HTTP traffic in general.

This means that you don’t need to have any specific prior knowledge of GoReplay in order to use it, whereas Speedscale requires you to have at least a basic knowledge of how Kubernetes works. However, if you’re looking for a Kubernetes load testing tool, it’s fair to assume this won’t be an issue.

Shadowing

One of the most tedious parts of running load tests is creating the traffic needed. Because of this, the principle of "shadowing" was invented. A tool that implements shadowing allows you to use traffic from your production services, and replicate it in your testing environment. Shadowing is one of the features found in both Speedscale and GoReplay.

To shadow a service with Speedscale, you first need to instrument your service using the speedctl tool. Instrumenting a service with Speedscale means that a sidecar will be added to your service. This sidecar acts as a proxy and captures all traffic going in and out of your service.

This way, Speedscale will always capture all the traffic linked to your instrumented service. Because of this, you will always have usable traffic available to use and create a snapshot from.

With a snapshot of traffic saved in Speedscale, you can now replay traffic to any service in your Kubernetes cluster. This is done either by clicking "Replay" in the Speedscale WebUI, adding annotations to an existing deployment, or deploying a new deployment with given annotations.

In GoReplay, you are going to be spending a lot more time in the terminal, as there is no WebUI for GoReplay. To capture traffic with GoReplay, you need to run:

$ sudo gor --input-raw :8000 --output-file ./requests.gor

This will save all requests to a file. The --input-raw :8000 defines that GoReplay will record any traffic that occurs on port 8000.

This is an important thing to note about GoReplay: it needs to be run on the system where you want to record traffic. In a Kubernetes setup, this can be tricky. The best way to do it is to add a GoReplay container to your pod, use that container to record the traffic, and then save the recorded requests to some persistent storage.

It’s important to note that GoReplay is not a proxy. Instead, it creates a hook at the system level and records any traffic occurring at the HTTP socket, on a given port.

Once traffic has been recorded, you use almost the same command to replay the traffic:

$ sudo gor --input-file ./requests.gor --output-http http://staging.example.com

Where Speedscale is continuously recording traffic, GoReplay requires you to manually set up the tool and save the files somewhere. On top of this, GoReplay only collects and replays inbound traffic, whereas Speedscale allows you to mock the outgoing requests, making sure you only load test your own service.

While GoReplay is a great tool in and of itself, it’s clear that Speedscale wins when it comes to testing services inside a Kubernetes cluster. While this section has shown you how shadowing is implemented in each of these services, the principles of regular load testing are the same. However, Speedscale allows you to modify the captured traffic and generate more powerful load tests, whereas GoReplay only allows you to replay the captured traffic, unless you manually edit the file of the captured requests.

Developer Experience

When it comes to setting up the tool, Speedscale lags a bit behind, with more steps needed to get the tool implemented, whereas GoReplay is as simple as downloading a binary. However, when you then start using the tool, you’ll notice that GoReplay starts to lag behind, as Speedscale has a lot more features, and is more specifically developed for Kubernetes.

If you have a service that you’re running locally, like when you’re developing a new service or a new feature, GoReplay is likely to be the most optimal tool. It allows you to capture traffic and replay it locally, which can be great for a quick test.

When you need to implement shadowing and load testing as an integral part of your infrastructure, for example as part of your CI/CD pipeline, you likely want to choose Speedscale as your tool of choice. All in all, Speedscale offers a more comprehensive tool for not only load testing but also API mocking and traffic visualization.

CI/CD Integration

When most people get started working with load testing, they just want to do a one-off load test, ensuring that the application is capable of handling the load that’s expected of the development team.

However, as the organization and infrastructure grows, most teams realize that including load tests as part of the CI/CD pipeline can provide immense benefits:

• Continuous feedback on the resiliency of your application
• More trust in the CI/CD pipeline, with more parts of the application being tested
• You can get one step closer to continuous deployments, as more parts of your application is verified

But, how well do Speedscale and GoReplay respectively integrate into a CI/CD pipeline?

Integrating with Speedscale

Speedscale provides you with detailed instructions on how to integrate with a CI/CD provider. The specific steps vary a bit depending on what CI/CD provider you want to integrate with, however they all boil down to the following steps.

• Create a snapshot
• Deploy the test
• Fetch the report
• If the status says “PASSED”, the check will pass

Again, in the Speedscale documentation you can find detailed instructions for implementing this into specific CI/CD providers. However, they all utilize the speedctl CLI tool. So, if your CI/CD provider is not on the list, you should be able to implement a script yourself.

Integrating with GoReplay

The first thing to know about integrating GoReplay with a CI/CD provider, is that there are no official instructions on how to do this. So, you’ll have to create a custom script.

Before you get started on implementing this script, it’s assuming that you’ve used the gor CLI tool to create a file consisting requests. This file needs to be uploaded to, or be accessible by your CI/CD server.

With access to the replay file, you can now run your load test by running:

$ sudo gor --input-file ./requests.gor --output-http http://staging.example.com

This part is easy, but there are a few other parts that you need to take care of yourself, that Speedscale otherwise handles for you.

You need to spin up a new instance of your application that the load test should be performed on. Then you can run the test. After that, you need a third-party approach to verifying that the requests succeeded, as this is not supported by GoReplay. Then, you need to pass that verification off to the CI/CD provider.

Speedscale or GoReplay?

In the end, both tools let you record traffic and replay it, both accomplishing the basic goal of traffic shadowing. However, it can easily be argued that Speedscale is a better fit for modern architecture. Whereas GoReplay is a great tool for replaying traffic locally, using it efficiently in a Kubernetes setup is going to be a challenge.

From the beginning, Speedscale has had a focus on matching the modern nature of a Kubernetes setup. Not only does it integrate directly with your cluster via the sidecar mechanism, Speedscale also has a lot of extra intelligence that allows for more flexible shadowing.

For example, you might have a service where requests have to be within a certain amount of time in the past. I.e. the request has to have been made within the past 2 hours. When you record your traffic and replay it 3 hours later, this will be an issue, as the timestamp is now outdated.

Speedscale can automatically detect this and shift the timestamp, so it still matches the age criteria. This automatic detection is also the case with tokens and unique IDs. Even if Speedscale can’t detect and change the values automatically, you can manually configure Traffic Transforms.

Especially when it comes to shadowing in Kubernetes, you’re going to have a much better time with Speedscale, as it’s been developed to work directly with Kubernetes. If you’re still not sure what tool is best for you, check out one of our other comparisons, like the one between Jmeter and Speedscale