REST API Observability for Python

Introduction

In this blog post we’ll help answer the age old question, “What does this service talk to and what does it say?” We’ll see how to inspect inbound and outbound REST API calls to see what calls are being made and what incoming traffic causes a reaction. This can be pretty handy when you’re taking over maintenance of an existing service, or if your code just isn’t behaving the way you expect.

For this exercise let’s observe the world’s simplest and least reliable Python application using Speedscale’s desktop observability tool. This pre-made demo app accepts inbound HTTP GET requests and returns the current Hackernews top 10 new articles. Note that speedscale-cli can inspect API calls for services running in docker and on your local desktop. But to keep things simple, we’ll run the app locally and route traffic using a socks5 proxy.

The cli can do all kinds of fancy things like transparently unwrapping TLS and tracking multiple traffic snapshots. You can find out more about it in the docs.

1. Install speedscale-cli

Visit the github repository for speedscale-cli:

https://github.com/speedscale/speedscale-cli

Alternatively, you can skip ahead and install using curl:

				
					curl -sL https://downloads.speedscale.com/speedscale-cli/install | sh

speedscale init
				
			

2. Clone the demo app

https://github.com/speedscale/demo-hackernews

3. Start the proxy

Start capturing by specifying your application’s HTTP port. By default our demo app listens on port 8080. 

				
					speedscale start capture -p 8080
				
			
 
Now speedscale will forward all requests made to proxy port 4143 to your application on port 8080.
 

4. Add TLS Cert

You may be prompted to trust a newly generated certificate to support TLS. This is required so the proxy can capture TLS traffic made to external systems. The Hackernews API does not allow unsecured connections so this is required.
For Python urllib you’ll need to take one more step. Copy and paste a line from the speedscale start capture output for SSL_CERT_FILE and export it as an environment variable. It should look something like this.
 
				
					export SSL_CERT_FILE=${HOME}/.speedscale/certs/tls.crt
				
			
 

5. Configure SOCKS Proxy (for Outbound Calls)

The real fun of observing an app is getting the complete call history of both inbound and outbound requests. By setting some environment variables we can redirect outbound requests through the proxy as well to enable outbound visibility.  Set the HTTP proxy environment variables and start the demo application

				
					export http_proxy=socks5://127.0.0.1:4140
export https_proxy=socks5://127.0.0.1:4140
./hn.py
				
			
 
Python3 and urllib automatically support recognize and these proxy environment variables. Many other languages and libraries will as well.

6. Capture

Let’s find out what’s going on in the world by running a GET against our new endpoint.

				
					curl http://localhost:4143/test
{
"0": "Ask HN: Best PC Laptop less than $1000 for young gamer?",
"1": "BioNTech CEO says vaccine likely to protect against severe Covid from Omicron",
"2": "Ask HN: What devices do you use as daily drivers?",
"3": "The problem with Python's map and filter \u2013 Abhinav Omprakash",
"4": "How to write a great Stack Overflow question (6 steps)",
"5": "Meta / Facebook\u2019s Head of Crypto is leaving",
"6": "Retroactive: Run Aperture, iPhoto, and iTunes on macOS Big Sur",
"7": "Ask HN: Why did plain text never catch on?",
"8": "A Thousand Brains: A new theory of intelligence",
"9": "The one-more-re-nightmare compiler"
}
				
			

 

7. Stop the Proxy

Stop recording and finalize the analysis.

				
					speedscale stop capture
				
			

 

8. Inspect

Now inspect the snapshot you just created.
				
					speedscale inspect
				
			
Check out the single inbound curl you ran and all the outbound transactions that resulted. You can do this with most desktop applications and quickly see what API calls they depend on.
 

About The Author