Building a Kubernetes friendly CLI

If you’re working with Kubernetes (k8s) on a daily basis you probably spend a lot of time with command line interfaces (CLIs) like kubectl and istioctl. In my opinion, the overall ergonomics of these tools are excellent and worthy of emulation. They’re easy to configure, relatively friendly to SREs and easy to automate.

The first step in designing Speedscale’s customer-facing tools was to deconstruct what makes these tools great so we could make our own CLI feel natural to our target user. Come along and learn a few tricks for building K8s-focused CLIs from our own experience building ssctl for our customers.

0. The cool k8s kids are using Golang

If you get deep into building Kubernetes focused apps, you should probably learn Golang. I know, I know… Over the years Python has rightfully earned its place on the front of the SRE’s toolbelt and nodejs is useful up and down the stack, but being lazy is a key part of being a great engineer so hear me out. The smart people at Google, Aspen Mesh, Linkerd, etc are building a huge library of OSS code that interacts with K8s. If you want to do something clever with K8s, start in the same ballpark as the rest of the ecosystem.

1. Cobra is incredible

The basic scaffolding of any CLI is its ability to parse command line arguments, manage configuration files, and communicate with the environment. In these dimensions, Cobra, and its hatch-mate Viper, are incredible and they are quickly becoming foundational for any Golang CLI.

spf13/cobra

spf13/viper

ssctl handles common operations for deploying and maintaining the Speedscale
    sidecar proxies and the isolation test environment. Most commands are run in conjunction with
    kubectl

Usage:
  ssctl [command]

Available Commands:
  check        Prints config to help make sure that the config file syntax is correct
  controlplane Deploys Speedscale control plane services
  creds        Deploys ONLY credentials
  deploytest   Deploys the complete test environment (generator and responder)
  generator    Deploys generator ONLY with dependent services
  get          Retrieve resources from Speedscale database
  help         Help about any command
  inject       Inject Speedscale sidecar to Kubernetes pod resources
  responder    Deploys responder ONLY with dependent services
  runtest      Restarts the test generator
  uninject     Uninject Speedscale sidecar from Kubernetes pod resources
  version      Prints current version

Flags:
      --config string    config file (default is $HOME/.speedscale.yaml)
  -c, --context string   Uses a specific context from those listed in ${HOME}/.speedscale/config
  -h, --help             help for ssctl

Use "ssctl [command] --help" for more information about a command.

(mostly auto generated by cobra)

2. Structure your config file with contexts

We’ve interviewed countless k8s admins and few best practices exist for cluster and namespace management. Give the SRE managing these diverse environments a break by adding contexts to your configuration. This will let them quickly switch between production, staging, etc.

current-context: my-context
contexts:
  - name: my-context
    tenant: dev
    kubernetes: sstest
    loglevel: debug
    containertype: dev

3. Invest in a comprehensive “check” command

Kubernetes is full of sharp edges to get stuck on. One of the fantastic things about linkerd is how the check command works. It not only gives a simple thumbs up/down but it analyzes the cluster to ensure compatibility before deployment. Few best practices exist for Kubernetes and every deployment is a unique so this makes things a ton easier.

$ ssctl check
Config Filename: /Users/matthewleray/.speedscale/config.yaml
Current Context: my-context
Namespace: sstest
Tenant Key: dev
Subtenant Key: default
Container Type: :dev
Log Level: debug
Kubernetes Key: sstest
SubTenant (full dump): {ID:default Name:default Stream:sstenant-dev RootTenant:{ID:dev Name:dev Cloud: Region:us-east-1 Bucket:sstenant-dev}}
DLP Token Config ID: none
Container Registry: gcr.io/speedscale

√ basic configuration check successful

4. Produce declarative yamls imperatively

You access the front door of K8s declaratively using yaml or json. Type up your yaml, feed it into kubectl apply and you’re good to go. But that isn’t enough for us because we have some unusually complicated interactions taking place between our cloud service and the kubernetes cluster. Specifically, we need to configure k8s network routing, secrets, configmaps, etc based on scenarios compiled by our cloud service. To state it directly, our yaml mutates based on external factors.

We found a useful pattern in istioctl’s experimental kube-inject command to handle this type of situation.

Installing the Sidecar

This particular command takes an existing yaml, parses it, and inserts Istio specific goodies before redeploying. Our use case has a lot more steps but it works based on the same principle.

5. CLIs and Operators go hand in hand

My last bit of advice is to view a CLI and Operator as two pieces of a puzzle that work in lock step. A CLI is more like an API call whereas an Operator is a “SysAdmin in a box.” If you find yourself repeatedly executing step by step instructions or needing to manage state in your K8s cluster then you may need an Operator. In our case, the CLI does most of the heavy lifting of initial configuration but we intend to utilize our Operator for most of the actual test execution. Here’s a quick tutorial that is helpful to get started:

Writing Your First Kubernetes Operator

————–

Many businesses struggle to discover problems with their cloud services before they impact customers. For developers, writing tests is manual and time-intensive. Speedscale allows you to stress test your cloud services with real-world scenarios. Get confidence in your releases without testing slowing you down. If you would like more information, schedule a demo today!