Using gRPC with Golang

In this tutorial, you will learn how to work with the gRPC Golang library for microservice communication by creating a simple note-taking application.

APIs and service-to-service communication are what make modern microservice architecture possible. REST is generally the preferred implementation pattern, but if you only use REST, you could miss out on the significant performance gains that gRPC can offer. gRPC can provide better speed and efficiency than REST APIs.

What Is gRPC?

gRPC is a Remote Procedure Call (RPC) framework. RPC is an action-based paradigm, similar to remotely calling a function from another microservice. This makes gRPC a type of inter-process communication (IPC) protocol built around Protobufs to handle messaging between the client and the server. gRPC is perfect for intensive and efficient communication, because it supports client and server streaming.

In contrast, REST is a resource-based protocol, which means the client tells the server what resource needs to be created, read, updated, or deleted based on the body of the request.

This difference means you can’t just translate your REST API into gRPC. Instead, you need to adapt your design to the other protocol:

Why Is gRPC Faster than REST?

gRPC utilizes the HTTP/2 protocol, which offers multiple ways to improve performance:

• Header compression and reuse to reduce the message size
• Multiplexing to send multiple requests and receive multiple responses simultaneously over a single TCP connection
• Persistent TCP connections for multiple sequential requests and responses on a single TCP connection
• Binary format support such as protocol buffers

Protocol buffers are a key element in gRPC. They provide more efficient binary data representation than other text-based formats such as JSON or XML. Because of their binary format, message sizes are significantly smaller. The combination of smaller messages and faster communication offers stronger performance than REST-based APIs.

Now that you have a basic understanding of gRPC, you’re going to implement a simple client-server application in Go.

Implementing gRPC

To see gRPC in action, you’re first going to build a small note-taking application. Then you’ll use the client to send notes to the server that can be archived and retrieved based on keywords.

The tutorial uses Go 1.17 and the gRPC-Go library 1.45.0.

You can find the entire code in this GitHub repo.

Creating the Go Project

Start building your application by initializing a new Go project. Be sure to replace the name in the command line with the name of your repository:

go mod init github.com/xNok/go-grpc-demo

Install the gRPC-Go library:

go mod edit -require=google.golang.org/grpc@v1.45.0
go mod download

Install the protocol buffer compiler (protoc) using the instructions that match your operating system, according to the installation docs. For Linux, this tutorial uses the following command:

sudo apt install protobuf-compiler

Next, install Golang code generators for the protocol buffer:

go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.26
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.1

The binaries for these two tools need to be in your path. For Linux, use:

export PATH="$PATH:$(go env GOPATH)/bin"

Implementing the Application Logic

It is considered good practice to separate the business logic from the server code. To do this, you’ll create a notes folder for your application’s business logic.

Protocol buffers provide more than a fast and efficient data format; they come with a rich ecosystem of tools to make development easier. A declarative .proto file describes your data structures (messages) and application interfaces (services). This file is simple to write and gives a high-level definition of what your application will be doing. It’s also language-agnostic and can generate the code required to manipulate these data structures.

Begin by creating a notes.proto file in the notes folder. A .proto file starts by indicating which version of the protocol buffer you’re using, the name of the package to prevent collision, and a series of options to help the code generator. In your case, you only need to specify the name of the Go package where the generated files belong:

syntax = "proto3";
package notes;

option go_package = "github.com/xNok/go-grpc-demo;notes";

In option go_package, github.com/xNok/go-grpc-demo is the name of the module matching the name of the GitHub repo, and notes defines the package for the generated code.

Next, define the data structure:

// The notes service definition.
service Notes {
  // Saving a note
  rpc Save (Note) returns (NoteSaveReply) {}
  // Retrieving a note
  rpc Load (NoteSearch) returns (Note) {}
}

// The request message containing the note title
message Note {
  string title = 1;
  bytes body  = 2;
}

// The response message confirming if the note is saved
message NoteSaveReply {
  bool saved = 1;
}

// The request message containing the note title
message NoteSearch {
  string keyword = 1;
}

You’ve created a service called Notes that can perform two actions: Save a note and Load a note. You also specified that you want to use rpc for these functions as the communication protocol. The Save function takes a Note (composed of a title and a body) as an input and returns NoteSaveReply, which contains a Boolean to confirm if the note was saved. The Load function takes the NoteSearch message as input that provides your desired keyword and returns a Note.

The syntax differs from Go struct and interface but should be easy to learn. Note that the types aren’t the same as in Go. Check the language guide to start on the right foundation.

Finally, note the attributes. Attributes are always assigned an ID; for instance, in the Note message, the title has ID 1 and the body has ID 2.

The data-structure definition is now complete. Next, you’ll generate the related code using protoc, the compiler for protocol buffer definitions files you installed earlier:

protoc --go_out=. --go_opt=paths=source_relative 
    --go-grpc_out=. --go-grpc_opt=paths=source_relative 
    notes/notes.proto

The command generates two new files: notes.pb.go, containing the code to handle the data serialization/deserialization for the protocol buffer; and notes_grpc.pb.go to handle the gRPC communication protocol. These files represent several hundreds of lines of code you don’t have to write.

You now have the structure of your application but still need some business logic. Create a file called note.go in the notes folder. You’ll code the logic here:

package notes

import (
        "errors"
        "io/ioutil"
        "log"
        "os"
        "path/filepath"
        "strings"
)

// Save a Note to the disk with the title as filename
func SaveToDisk(n *Note, folder string) error {
        filename := filepath.Join(folder, n.Title) //title should be sanitized
        return os.WriteFile(filename, n.Body, 0600)
}

// Scan files in a folder to find first occurrence of a keyword
func LoadFromDisk(keyword string, folder string) (*Note, error) {
        filename, err := searchKeywordInFilename(folder, keyword)
        if err != nil {
                return nil, err
        }
        body, err := os.ReadFile(filepath.Join(folder, filename))
        if err != nil {
                return nil, err
        }
        return &Note{Title: filename, Body: body}, nil
}

// Scan a directory and if a file name contains a substring, return the first one
func searchKeywordInFilename(folder string, keyword string) (string, error) {
        items, _ := ioutil.ReadDir(folder)
        for _, item := range items {

                // Read the whole file at once
                // this is the most inefficient search engine in the world
                // good enough for an example
                b, err := ioutil.ReadFile(filepath.Join(folder, item.Name()))
                if err != nil {
                        // This is not normal but we can safely ignore it
                        log.Printf("Could not read %v", item.Name())
                }
                s := string(b)

                if strings.Contains(s, keyword) {
                        return item.Name(), nil
                }
        }
        return "", errors.New("no file contains this keyword")
}

You’ll notice in the above code that you reuse the Note generated for you by protoc in the notes.pb.go file.

Implementing the gRPC Server

Create a folder called notes_server and inside this folder create a main.go file. Start by implementing the minimal server configuration:

package main

import (
        "context"
"flag"
        "fmt"
        "log"
        "net"

        "github.com/xNok/go-grpc-demo/notes"
        "google.golang.org/grpc"
)

var (
        port = flag.Int("port", 50051, "The server port")
)

func main() {
        // parse arguments from the command line
        // this lets us define the port for the server
        flag.Parse()
        lis, err := net.Listen("tcp", fmt.Sprintf(":%d", *port))
        // Check for errors
        if err != nil {
                log.Fatalf("failed to listen: %v", err)
        }
        // Instantiate the server
        s := grpc.NewServer()

        // Register server method (actions the server will do)
  // TODO

        log.Printf("server listening at %v", lis.Addr())
        if err := s.Serve(lis); err != nil {
                log.Fatalf("failed to serve: %v", err)
        }
}

With this code, you have a working server but you haven’t registered any action that this server can perform. All the structure for handling gRPC communication is generated in notes/notes_grpc.pb.go. The only thing left is to implement the notes.NotesServer interface defined in that generated file.

Since you already have your business logic in the notes package, implement the notes.NotesServer interface by creating the required methods (Save and Load):

// Implement the notes service (notes.NotesServer interface)
type notesServer struct {
        notes.UnimplementedNotesServer
}

// Implement the notes.NotesServer interface
func (s *notesServer) Save(ctx context.Context, n *notes.Note) (*notes.NoteSaveReply, error) {
        log.Printf("Received a note to save: %v", n.Title)
        err := notes.SaveToDisk(n, "testdata")

        if err != nil {
                return &notes.NoteSaveReply{Saved: false}, err
        }

        return &notes.NoteSaveReply{Saved: true}, nil
}

// Implement the notes.NotesServer interface
func (s *notesServer) Load(ctx context.Context, search *notes.NoteSearch) (*notes.Note, error) {
        log.Printf("Received a note to load: %v", search.Keyword)
        n, err := notes.LoadFromDisk(search.Keyword, "testdata")

        if err != nil {
                return &notes.Note{}, err
        }

        return n, nil
}

Finally, register the notesServer you created. You only need to add a single line of code where the TODO comment is:

// Register server method (actions the server will do)
notes.RegisterNotesServer(s, &notesServer{})

Start the server:

go run ./notes_server/main.go

Implementing the gRPC Client

Create a folder called notes_client. Inside this folder, create a main.go file. Once again, you’ll implement the minimal client configuration, then add the gRPC elements. You want to create a command line that can either save a note or load a note. Here is the basic structure for this:

package main

import (
        "context"
        "flag"
        "fmt"
        "log"
        "os"
        "time"

        "github.com/xNok/go-grpc-demo/notes"
        "google.golang.org/grpc"
        "google.golang.org/grpc/credentials/insecure"
)

var (
        addr = flag.String("addr", "localhost:50051", "the address to connect to")
)

func main() {
        flag.Parse()
        // Set up a connection to the server.
        // TODO

        // Define the context
        ctx, cancel := context.WithTimeout(context.Background(), time.Second)
        defer cancel()

        // define expected flag for save
        saveCmd := flag.NewFlagSet("save", flag.ExitOnError)
        saveTitle := saveCmd.String("title", "", "Give a title to your note")
        saveBody := saveCmd.String("content", "", "Type what you like to remember")

        //define expected flags for load
        loadCmd := flag.NewFlagSet("load", flag.ExitOnError)
        loadKeyword := loadCmd.String("keyword", "", "A keyword you'd like to find in your notes")

        if len(os.Args) < 2 {
                fmt.Println("expected 'save' or 'load' subcommands")
                os.Exit(1)
        }

        switch os.Args[1] {
        case "save":
                saveCmd.Parse(os.Args[2:])
                // Call the server
                // TODO

        case "load":
                loadCmd.Parse(os.Args[2:])
                // Call the server
                // TODO

        default:
                fmt.Println("Expected 'save' or 'load' subcommands")
                os.Exit(1)
        }
}

Now, add the element required to communicate with the server. The first TODO is to establish the connection. Use the code generated by protoc and the client constructor NewNotesClient:

// Set up a connection to the server.
        conn, err := grpc.Dial(*addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
        if err != nil {
                log.Fatalf("did not connect: %v", err)
        }
        defer conn.Close()
        c := notes.NewNotesClient(conn)

For the next two TODOs, use the client you instantiated to remotely call the Save and Load functions from the server. Here is the code for the save command:

case "save":
                saveCmd.Parse(os.Args[2:])    
                _, err := c.Save(ctx, &notes.Note{
                        Title: *saveTitle,
                        Body:  []byte(*saveBody),
                })

                if err != nil {
                        log.Fatalf("The note could not be saved: %v", err)
                }

                fmt.Printf("Your note was saved: %vn", *saveTitle)

Here is the code for the load command:

case "load":
                loadCmd.Parse(os.Args[2:])
                note, err := c.Load(ctx, &notes.NoteSearch{
                        Keyword: *loadKeyword,
                })

                if err != nil {
                        log.Fatalf("The note could not be loaded: %v", err)
                }

                fmt.Printf("%vn", note)

Your client is now ready to use. Create a testdata folder to hold your notes (the name of the output folder is hard coded in the code). Next, try to take some notes:

go run notes_client/main.go save -title test -content "Lorem ipsum dolor sit amet, consectetur "

Can you retrieve the note you just saved?

go run notes_client/main.go load -keyword Lorem

Going Further with gRPC

So far you’ve only been using simple unary gRPC communication, in which a single request is sent to the server and you get a single response back. gRPC also supports three types of streaming protocol: server streaming, client streaming, and bidirectional streaming. Each works well for a large amount of asynchronous data.

You’re going to implement a SaveLargeNote function that utilizes streaming. First, update your service definition in notes.proto to include:

        // Save a note via Streaming
  rpc SaveLargeNote (stream Note) returns (NoteSaveReply) {}

Notice that you use the stream keyword to indicate the direction of the stream. The stream keyword on the function argument is on the caller side for client-server streaming.

Generate the protocol buffer code using protoc. The command is the same as before:

protoc --go_out=. --go_opt=paths=source_relative 
    --go-grpc_out=. --go-grpc_opt=paths=source_relative 
    notes/notes.proto

Implement the new SaveLargeNote function in the server code:

func (s *notesServer) SaveLargeNote(stream notes.Notes_SaveLargeNoteServer) error {
        var finalBody []byte
        var finalTitle string
        for {
                // Get a packet
                note, err := stream.Recv()
                if err == io.EOF {
                        log.Printf("Received a note to save: %v", finalTitle)
                        err := notes.SaveToDisk(&notes.Note{
                                Title: finalTitle,
                                Body:  finalBody,
                        }, "testdata")

                        if err != nil {
                                stream.SendAndClose(&notes.NoteSaveReply{Saved: false})
                                return err
                        }

                        stream.SendAndClose(&notes.NoteSaveReply{Saved: true})
                        return nil
                }
                if err != nil {
                        return err
                }
                log.Printf("Received a chunk of the note to save: %v", note.Body)
                // Concat packet to create final note
                finalBody = append(finalBody, note.Body...)
                finalTitle = note.Title
        }
}

Note these elements in the above code:

• stream.Recv() lets you consume a packet (the order of packets in a stream is preserved)
• io.EOF error ends a stream
• stream.SendAndClose is used to close the stream and send the response to the client

Now you need to update the client. You’ll add a new option -l that will instruct the client to use the new SaveLargeNote:

saveLargeBody := saveCmd.Bool("l", false, "flag to upload a note broken as a stream")

Create the stream by splitting the Body into chunks. The following code needs to be placed inside the save case. When using the save comment, if saveLargeBody is provided, then use the gRPC stream.

if *saveLargeBody {
                        stream, err := c.SaveLargeNote(ctx)
                        if err != nil {
                                log.Fatalf("Fail to create stream: %v", err)
                        }
                        chunks := split([]byte(*saveBody), 10)
                        for _, chunk := range chunks {
                                note := &notes.Note{
                                        Title: *saveTitle,
                                        Body:  chunk,
                                }
                                if err := stream.Send(note); err != nil {
                                        log.Fatalf("%v.Send(%v) = %v", stream, note, err)
                                }
                        }
                        _, err = stream.CloseAndRecv()
                        if err != nil {
                                log.Fatalf("%v.CloseAndRecv() got error %v, want %v", stream, err, nil)
                        }
                } else {

Note these elements in the above code:

• stream.Send sends a packet to the stream
• stream.CloseAndRecv closes the stream and waits for the server response

Here is the split function used to create a chunk of the Body:

// create bytes chunks for streaming
func split(buf []byte, lim int) [][]byte {
        var chunk []byte
        chunks := make([][]byte, 0, len(buf)/lim+1)
        for len(buf) >= lim {
                chunk, buf = buf[:lim], buf[lim:]
                chunks = append(chunks, chunk)
        }
        if len(buf) > 0 {
                chunks = append(chunks, buf[:len(buf)])
        }
        return chunks
}

You can learn how to implement the remaining two communication options with the gRPC basic tutorial.

Conclusion

gRPC is a faster, smarter option for service-to-service communication and client-server mobile applications. As you saw in this tutorial, gRPC and Golang make a good combination for microservice communication. What you need to do is change your perspective from creating resource-based APIs to creating action-based APIs.

If you’re creating gRPC applications and need to test app performance, Speedscale can help. The traffic replay framework conducts API testing in Kubernetes, offering automatically generated tests based on real-time data that’s stored in Speedscale’s data warehouse for analysis. Speedscale supports gRPC, REST, and GraphQL, among other protocols and environments. To see how Speedscale can help you, sign up for a free trial.

To check your work on this tutorial, you can find the code on GitHub.