Performance testing is a crucial aspect of software development, serving a significantly different purpose than traditional methodologies like unit or end-to-end testing. Rather, testing performance is about response times, throughput, resource utilization, etc. Given the traditional complexity and cost of performance testing, it’s crucial to know how affordable performance testing can be achieved.
While unit testing provides confidence in an application’s ability to execute logic in a certain way—ensuring that X input provides Y output—performance testing provides confidence in an application’s ability to handle diverse situations and loads. In other words, unit testing verifies logic, while performance testing verifies behavior—unit testing and performance testing are generally described as functional and non-functional testing, respectively.
Functional testing is by far the most common form of testing, in large part because of the complexity and cost traditionally associated with performance testing, as it often requires a bigger time and monetary commitment than executing unit tests. Non-functional testing requires testing infrastructure that can replicate real-world behavior, which quickly becomes a complex and costly endeavor.
This is where traffic-generated tests comes into the picture, aiming to revolutionize performance testing by making it more affordable and accessible, specifically designed to address the concerns of engineers as well as executives. Performance testing helps businesses identify and address potential bottlenecks and performance issues before new code makes it into production, which is crucial when creating a good user experience. Conversely, tooling that’s complex for engineers to set up, use, and maintain will greatly reduce the usefulness of those tests.
This post covers how production traffic replication provides a new approach to performance testing, solving the challenges created by traditional tooling, and thus making it an attractive option for businesses looking to optimize their application performance, without breaking the bank.
Simplifying Performance Testing
A core principle of production traffic replication is to provide the powerful insights of traditional performance testing, while maintaining the simplicity and ease of unit testing. This unique approach to testing results in an intuitive and user-friendly experience suited to high-level executives who don’t necessarily have a deep technical understanding, without sacrificing the flexibility wanted by engineers.
This simplified approach can have a profound impact on key business metrics, as Vistaprint experienced when they introduced Speedscale into their testing process. Jeff Kwan—Principal Software Engineer—notes how Speedscale’s traffic-driven approach—as compared to K6’s scripting-driven approach—took a 4 week testing process down to three days.
The case study linked above can provide you with much greater insight into the positive impact this new approach has had on Vistaprint’s software development process, but these two examples alone should give you some idea of what you can expect.
This is just one example of how a traffic-based testing approach can have tangible effects on an engineering team, as well as reducing costs for business owners. The major reduction in time spent on testing also means, of course, a major reduction in engineering hours, which is—for many software companies—the biggest cost. Let’s dive deeper into the cost aspect of using traffic for performance testing.
Making Performance Testing Affordable
Although the previous section stated that a traffic-based testing approach reduces engineering hours, engineers aren’t working less—they’re spending fewer hours on each individual task. This allows engineers to utilize their working hours more efficiently, putting together more features or optimizations in the same time span, and subsequently increasing the ROI on salary cost.
But of course, time spent on tooling isn’t the only reason why traditional performance testing tools prove to be costly for businesses, although it is a major factor. Let’s explore some of the cost savings.
Reducing Infrastructure Management
Most tools, whether for testing or not, require teams to manage additional infrastructure such as servers or git repositories. While no tool can completely eliminate infrastructure management and still provide powerful insights, Speedscale gets you 95% of the way there. In non-technical terms, Speedscale requires a component known as an Operator being installed into your Kubernetes cluster, which is easily done by executing a single command.
From here, Speedscale takes care of managing everything, only requiring infrastructure management if new clusters need to be configured. This heavily reduces not only maintenance costs but also infrastructure costs, as Speedscale utilizes what you’re already paying for. Additionally, focusing solely on running in Kubernetes, the Speedscale team is able to continually optimize the Operator to run as efficiently as possible in this specific environment.
Comparing Different Approaches
With production traffic replication—the act of capturing real traffic for the purpose of replaying it later in a different environment—you can perform powerful comparisons between tools. One example of this is how Nylas used Speedscale to prove that Google’s Tau VMs provided them with over 40% price-performance.
Intuitive Interactions
As mentioned earlier, Speedscale has developed a user interface that’s intuitive for developers and executives alike. On the surface it may not be apparent how this relates to business impact, as it’s not common for non-technical employees like executives or product managers to use developer tools. This has traditionally been the case because of the necessary knowledge required in order to interact with these tools.
Given the intuitive nature of the Speedscale WebUI, non-technical personnel will only need a quick introduction from a developer before they’ll be able to start running performance tests themselves, changing configurations to determine behavior under various conditions like seeing what happens if the traffic load suddenly doubles.
Traditionally, product teams would have to create a set of requirements for an application’s performance, which the development team would then have to test for. This creates an environment where non-technical employees are reliant on developers in order to get any particular insight. Now, product managers can implement different test scenarios themselves, which provides two distinct advantages:
- It optimizes engineers’ hours even more, as developers won’t be doing what seems like tedious work, i.e., test creation;
- Getting non-technical people more involved in the testing process gives them more freedom, along with a better understanding of the application’s limits and behavior. This results in more fluid communication between technical and non-technical people.
Avoiding Sunk Cost With Powerful Integrations
A common issue when implementing new tooling in the software development process is the inability to leverage existing tools and knowledge, often requiring significant shifts in workflow or infrastructure. This is often a big driver of cost when implementing new forms of testing, in terms of both engineering hours and cost.
Speedscale believes in commoditizing traffic and allowing it to drive testing, rather than relying on scripting. This approach results in seamless integrations with various other tools like Postman and K6. This allows production traffic replication to drop into existing workflows, taking advantage of the time and labor savings without disrupting accepted processes.
For example, developers often appreciate using Postman to quickly fire off a single request—or multiple requests in succession—from their local PC. The challenge lies in the fact that these requests have to be created manually, which can lead to unrealistic results during local testing. To solve this challenge and accommodate those wishing to use Postman, Speedscale makes it easy to export recorded traffic to Postman collections. The same is true in terms of k6, a popular load testing tool.
While Speedscale does provide powerful ways to perform load testing, some developers may find situations where they either find k6 to be more suitable, or simply prefer it because of their existing knowledge. In any case, Speedscale can export recorded traffic to k6 scripts, allowing developers to utilize what they know while at the same time reaping the benefits of using production traffic.
To summarize, no tool is ever likely to be the single best or most preferred in all circumstances. Speedscale’s big advantage lies in its ability to integrate well with common tools and development paradigms like CI/CD.
The Importance of Simple and Affordable Testing
Up until now, you’ve mostly read about the enhancements and benefits provided by Speedscale, but it’s important to understand that it’s not just a matter of improving something that already—in some capacity—works. Implementing complex and costly testing introduces a variety of risks that can end up affecting important business metrics.
Introducing complex tooling can easily lead to a frustrating workflow for developers, with the increased frustration leading to lower productivity and other adverse effects. Although some tools may provide great insights, introducing them is unlikely to be worth it if maintenance costs are increased fivefold, which some companies have experienced.
However, cost is far from the only business metric possibly affected by a sub-optimal development experience. In today’s competitive landscape, it’s crucial to move fast. There’s an increasing trend of “building in public,” which has led companies to push out new features more rapidly, leaving others with three choices: (1) release new features at a slower pace but with lengthier testing processes, (2) release new features riddled with bugs due to poor testing, or (3) ensure that the testing process is as quick and efficient as possible.
Implementing tools that are complex and costly has never been desired, but in the context of modern cloud development, adopting tools that are affordable and simple to use is more important than ever.
This is all before considering the growing influence of AI on software development, which adds another dimension to this discussion. Although AI is a diverse topic, it’s hard to ignore its impact. Numerous companies based on AI technologies have swiftly emerged, and while time will tell whether these companies become successful, it’s undoubtedly led to a rise in companies being able to start up and push out new features rapidly. While AI is not related to anything else talked about in this post, it’s here to point out that the modern software development industry is changing, and whether that’s because of AI or not, the inability to move fast and efficiently is very likely to cause challenges for some organizations.
Automation at the Core
While traffic replay is the key feature enabling simple and cost-effective performance testing, the core principle behind Speedscale’s ease of use is automation. This truly shines through when looking at key features like automatic mock creation, continuous traffic recording, automatic infrastructure integration, automatic test execution, and deep integration into CI/CD pipelines. Automating these menial tasks helps developers reduce cognitive load, allowing those developers to redirect their focus towards innovative projects that reduce churn, enhance brand reputation, and ultimately increase revenue.
In addition to this, having automation at the core of everything ensures the ability to run tests continuously and consistently, which reduces the likelihood of bugs making it into production, thereby resulting in a better user experience.
Taking the Next Step with Speedscale
To reiterate, Speedscale enables developers to utilize their time more efficiently, allowing them to focus more on creating innovative features and optimizations. By streamlining developers’ workflows, you’ll see a reduction in not only maintenance cost but also opportunity cost as well.
Outside of business metrics, you’ll likely see that this improved efficiency fosters a better work environment, happier teams, and ultimately, a superior product. But of course, key metrics such as reduced churn and increased brand reputation are to be expected as well.
As an executive or decision-maker, it’s crucial to explore how Speedscale’s unique approach to performance testing can impact on the different parts of your organization. To gain a better understanding of how this tool can fit into your development process, talk over the potential benefits with engineering managers or engineers themselves.
If you’re reading this as a business leader, you may benefit from having an engineer try to follow the traffic replay tutorial, which guides them through setting up and using Speedscale. The entire process shouldn’t take more than an hour or two at most, and you’ll have the opportunity to get a technical person’s take on how well the tool performs—something that, as already mentioned, is crucial when choosing a new tool.