Integrating Speedscale with Jenkins

Minimizing and automating the path from development and production is necessary in order to stay competitive and keep customers happy. As engineering teams strive to solve this by quickly and efficiently rolling out new features, updates, and bug fixes, continuous integration and deployment (CI/CD) has come to be regarded as an industry best practice. One of the most popular CI/CD solutions is Jenkins, an open-source job execution system. Jenkins provides engineering teams the tools necessary to automate the build, test, and deployment process of their software.

Jenkins is capable of integrating with a wide variety of tools and platforms, including Speedscale. Speedscale allows engineering teams to continuously stress test their APIs using real world traffic without the need for complicated scripting. This makes it a perfect candidate for Jenkins integration for pre-deployment validations. Even before this blog was available, companies like Grofers have integrated Speedscale traffic replays as part of their CI process allowing for more robust checks and high levels of automation.

In this article, you’ll learn how you can include Speedscale as a part of your Jenkins powered CI/CD platform.

Prerequisites for integrating Speedscale with Jenkins

The details of setting up a fully operational Jenkins environment are outside the scope of this article, so we will assume that you have access to one already. If you do not, there are pre-built options available, including a Docker image.

Whether your Jenkins jobs run within docker containers or directly on Jenkins executor nodes, these environments will need to have the following installed:

• speedctl
• kubectl
• jq

You will also need to create a Speedscale traffic snapshot of your API. To learn more about how to do this, please refer to the Speedscale documentation.

Integration

CI/CD Stages

We’ll discuss the Jenkins integrations as a series of discrete stages. There are many different ways to accomplish this, all of which depend on the specific needs of your team, application, or build system. For example, a very simple workflow might look like:

Jenkinsfile

A Jenkinsfile is groovy script, checked into source control at the root of your project that defines the stages of a Jenkins Pipeline. Based on our workflow shown above, let’s define a simple pipeline:

#!groovy

pipeline {
  agent any

  stages {
    stage('validate') {
      steps {
        sh 'kubectl apply -f qa-deployment.yaml -n qa'
        sh 'sh speedscale.sh'
      }
    }
    stage('deploy') {
      steps {
        sh 'kubectl apply -f deployment.yaml -n prod'
      }
    }
  }
  post{
    success{
      echo "======== pipeline executed successfully ========"
    }
    failure{
      echo "======== pipeline execution failed ========"
    }
  }
}

In this simple pipeline, there are two discrete steps:

2. validate: a test Kubernetes deployment is made to a qa namespace, followed by a speedscale.sh script that will start the scenario and wait for the results.
3. deploy: your application is deployed into a prod namespace if the previous step succeeds

Note that we’re assuming that your repository is already configured for Jenkins integration, so we won’t discuss how to configure a new Jenkins job. If you don’t, work with your Jenkins administrator to define a new job that uses SCM to watch your repository for changes and execute the Jenkinsfile in your project.

Running Speedscale

In the validate stage above, you’ll notice that there is an initial kubectl apply step to apply a specific yaml file from your repository into a qa namespace. This step could be anything, but the intent here is to deploy your application into a non-production environment; the qa-deployment.yaml could contain the Kubernetes resource definitions necessary to facilitate this.

The second step is the bulk of this stage. In this step, we’ll execute a speedscale.sh script that performs the remaining work necessary to initiate our Speedscale test and check the results. This script can either live in your repository or exist as a utility available in your Jenkins environment. For example, this script might look something like:

#!/bin/bash

# ensure that you have speedctl available on your path
export PATH=$PATH:/var/lib/jenkins/.speedscale

# create a unique report tag that identifies this run
REPORT_TAG=$(git rev-list --abbrev-commit --max-count=1 HEAD)-$(date +%s)

# modify a patch template file and apply it to your kubernetes test deployment
sed "s/REPORT_TAG/${REPORT_TAG}/g" scenario.yaml.tpl > scenario.yaml
kubectl patch deployment my-deployment --patch "$(cat scenario.yaml)" -n qa

echo "Waiting for scenario report to be available"

for i in {1..20}; do
  echo "Checking for available report (attempt ${i})"

  # make sure the report exists
  report_id=$(speedctl report list --tag "${REPORT_TAG}" | jq -r .Id || true)
  if [[ "$report_id" == "" ]]; then
    echo "Report not found, sleeping"
    sleep 30
    continue
  fi
  
  # get and check the status for the report ID we found
  status=$(speedctl report get ${report_id} | jq -rc '.status' | tr '[[:upper:]]' '[[:lower:]]' || true)
  case ${status} in
    "" | "initializing" | "in progress" | "running")
      echo "Report not ready, sleeping (status ${status})"
      sleep 30
      ;;

    "complete" | "missed goals" | "stopped" | "passed")
      echo "Report complete (status ${status})"
      events=$(speedctl report get ${REPORT_TAG} | jq -rc '.events' )
      if [[ "${events}" == "" ]]; then
        echo "Passed!"
        exit 0
      else
        echo "Failed!"
        exit 1
      fi
      ;;
  esac
done

In this script, we are first generating a unique tag that can be used to identify the report for this Speedscale run. We then need to patch the test deployment we created in the previous step in our Jenkins pipeline stage to start the test. Because kubectl does not support templated yaml files, we are performing an intermediate step with a file called scenario.yaml.tpl in which we replate a template variable with the unique tag we generated. Patching your test deployment with the resulting file will initiate the test, so let’s take a look at what this template file might look like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  annotations:
    test.speedscale.com/scenarioid: 954957b1-8aa5-453d-bace-a766f1992a2f
    test.speedscale.com/testconfigid: standard
    test.speedscale.com/cleanup: "true"
    test.speedscale.com/tag: REPORT_TAG
    sidecar.speedscale.com/inject: "true"

Once patched, speedctl can be used to check the status of the run:

status=$(speedctl report get ${SCENARIO_TAG} | jq -rc '.status' | tr '[[:upper:]]' '[[:lower:]]' || true)

The status returned from this command will be one of complete, missed goals, stopped, passed, or may be an empty string (which means the test is starting). When the test is complete and the report is available, checking the events field will give us an indication of the test outcome, no events meaning that the test succeeded.

Deploy!

As mentioned before, the previous stage is the bulk of this simple example. If this stage fails to complete successfully, Jenkins will fail the pipeline at that stage. Using Speedscale as a pipeline gating mechanism means you can do lots of things such as promoting your application to a staging environment, executing additional downstream jobs, sending notifications, etc. In our simple case we have chosen to perform a production deployment.

Conclusion

Continuous integration and deployment enables software development teams to automate the build and deployment of their applications and APIs, of which Speedscale can become an integral part. You’ve now seen how Speedscale can become a part of a Jenkins-powered CI/CD pipeline to ensure APIs are continuously validated and pass stress tests before they go into production.