gRPC is a great technology with tight interface constraints and high performance, and is used in k8s and many microservices frameworks.

as a programmer, you learned it right.

Having written some gRPC services in Python before, I’m ready to use Go to get a feel for the original gRPC program development.

This article features speaking directly in code, with out-of-the-box complete code to introduce various ways to use gRPC.

The code has been uploaded to GitHub, and the following is officially started.

introduce

gRPC is a cross-language, open source RPC framework developed by Google Based on Protobuf. Designed based on the HTTP/2 protocol, gRPC can provide multiple services based on a single HTTP/2 link, making it more mobile-friendly.

entry

The first step to a simple gRPC service is to define the proto file, because gRPC is also a C/S architecture, which is equivalent to clarifying the interface specification.

therefore

syntax = "proto3";
package proto;
// The greeting service definition.service Greeter {    // Sends a greeting    rpc SayHello (HelloRequest) returns (HelloReply) {}}
// The request message containing the user's name.message HelloRequest {    string name = 1;}
// The response message containing the greetingsmessage HelloReply {    string message = 1;}

copy the code

Generate gRPC code using the gRPC plugin built into protoc-gen-go:

protoc --go_out=plugins=grpc:. helloworld.proto

copy the code

after executing this command, a helloworld.pb.go file is generated in the current directory, which defines the interfaces of the server and the client, respectively:

// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://godoc.org/google.golang.org/grpc#ClientConn.NewStream.type GreeterClient interface {  // Sends a greeting  SayHello(ctx context.Context, in *HelloRequest, opts ...grpc.CallOption) (*HelloReply, error)}
// GreeterServer is the server API for Greeter service.type GreeterServer interface {  // Sends a greeting  SayHello(context.Context, *HelloRequest) (*HelloReply, error)}

copy the code

the next step is to write the code of the server and the client, and implement the corresponding interfaces respectively.

server

package main
import (  "context"  "fmt"  "grpc-server/proto"  "log"  "net"
  "google.golang.org/grpc"  "google.golang.org/grpc/reflection")
type greeter struct {}
func (*greeter) SayHello(ctx context.Context, req *proto.HelloRequest) (*proto.HelloReply, error) {  fmt.Println(req)  reply := &proto.HelloReply{Message: "hello"}  return reply, nil}
func main() {  lis, err := net.Listen("tcp", ":50051")  if err != nil {    log.Fatalf("failed to listen: %v", err)  }
  server := grpc.NewServer()  // 注册 grpcurl 所需的 reflection 服务  reflection.Register(server)  // 注册业务服务  proto.RegisterGreeterServer(server, &greeter{})
  fmt.Println("grpc server start ...")  if err := server.Serve(lis); err != nil {    log.Fatalf("failed to serve: %v", err)  }}

copy the code

client

package main
import (  "context"  "fmt"  "grpc-client/proto"  "log"
  "google.golang.org/grpc")
func main() {  conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure())  if err != nil {    log.Fatal(err)  }  defer conn.Close()
  client := proto.NewGreeterClient(conn)  reply, err := client.SayHello(context.Background(), &proto.HelloRequest{Name: "zhangsan"})  if err != nil {    log.Fatal(err)  }  fmt.Println(reply.Message)}

copy the code

This completes the development of the most basic gRPC service, and then we continue to enrich this “basic template” and learn more features.

stream mode

next, look at how streams are flowing, and as the name suggests, data can be sent and received in a steady stream.

the flow is divided into one-way flow and two-way flow, here we directly through the two-way flow to give an example.

therefore

service Greeter {    // Sends a greeting    rpc SayHello (HelloRequest) returns (HelloReply) {}    // Sends stream message    rpc SayHelloStream (stream HelloRequest) returns (stream HelloReply) {}}

copy the code

add a stream function to specify the stream attributes by keyword.SayHelloStreamstream

the helloworld.pb.go file needs to be regenerated, and i won’t go into more detail here.

server

func (*greeter) SayHelloStream(stream proto.Greeter_SayHelloStreamServer) error {  for {    args, err := stream.Recv()    if err != nil {      if err == io.EOF {        return nil      }      return err    }
    fmt.Println("Recv: " + args.Name)    reply := &proto.HelloReply{Message: "hi " + args.Name}
    err = stream.Send(reply)    if err != nil {      return err    }  }}

copy the code

add functions to the “base template”, and nothing else needs to be changed.SayHelloStream

client

client := proto.NewGreeterClient(conn)
// 流处理stream, err := client.SayHelloStream(context.Background())if err != nil {  log.Fatal(err)}
// 发送消息go func() {  for {    if err := stream.Send(&proto.HelloRequest{Name: "zhangsan"}); err != nil {      log.Fatal(err)    }    time.Sleep(time.Second)  }}()
// 接收消息for {  reply, err := stream.Recv()  if err != nil {    if err == io.EOF {      break    }    log.Fatal(err)  }  fmt.Println(reply.Message)}

copy the code

sending a message through a goroutine, the loop of the main program receives the message.for

the executor will find that both the server and the client have printouts.

validator

next is the validator, which is a natural requirement that comes to mind, because when it comes to requests between interfaces, it is necessary to perform proper validation of the parameters.

here we use protoc-gen-govalidators and go-grpc-middleware to implement this.

install first:

go get github.com/mwitkow/go-proto-validators/protoc-gen-govalidators
go get github.com/grpc-ecosystem/go-grpc-middleware

copy the code

next modify the proto file:

therefore

import "github.com/mwitkow/go-proto-validators@v0.3.2/validator.proto";
message HelloRequest {    string name = 1 [        (validator.field) = {regex: "^[z]{2,5}$"}    ];}

copy the code

the parameters are checked here, and the requirements of the regular rule need to be met before they can be requested normally.name

there are other validation rules, such as verifying the size of the number, which are not covered here.

next, generate the *.pb.go file:

protoc  \    --proto_path=${GOPATH}/pkg/mod \    --proto_path=${GOPATH}/pkg/mod/github.com/gogo/protobuf@v1.3.2 \    --proto_path=. \    --govalidators_out=. --go_out=plugins=grpc:.\    *.proto

copy the code

after successful execution, there will be one more helloworld.validator.pb.go file in the directory.

it is important to note here that it is not possible to use the previous simple command, and you need to use multiple parameters to specify the directory where to import the proto file.proto_path

the official gives two dependency cases, one is google protobuf and the other is gogo protobuf. i’m using the second one here.

even with the above command, you may encounter this error:

Import "github.com/mwitkow/go-proto-validators/validator.proto" was not found or had errors

copy the code

but don’t panic, the probability is the problem of referencing the path, be sure to look at your own installation version, as well as the specific path in.GOPATH

finally, there is the server-side code transformation:

ingest package:

grpc_middleware "github.com/grpc-ecosystem/go-grpc-middleware"grpc_validator "github.com/grpc-ecosystem/go-grpc-middleware/validator"

copy the code

then add the authenticator function during initialization:

server := grpc.NewServer(  grpc.UnaryInterceptor(    grpc_middleware.ChainUnaryServer(      grpc_validator.UnaryServerInterceptor(),    ),  ),  grpc.StreamInterceptor(    grpc_middleware.ChainStreamServer(      grpc_validator.StreamServerInterceptor(),    ),  ),)

copy the code

after starting the program, we use the previous client code to request, and we will receive an error:

2021/10/11 18:32:59 rpc error: code = InvalidArgument desc = invalid field Name: value 'zhangsan' must be a string conforming to regex "^[z]{2,5}$"exit status 1

copy the code

because it does not meet the requirements of the service-side regularity, but if the parameter is passed, it can be returned normally.name: zhangsanname: zzz

Token authentication

Finally to the authentication link, first look at the Token authentication method, and then introduce the certificate certification.

first transform the server, with the experience of the above validators, then you can use the same way, write an interceptor, and then inject it when initializing the server.

authentication function:

func Auth(ctx context.Context) error {  md, ok := metadata.FromIncomingContext(ctx)  if !ok {    return fmt.Errorf("missing credentials")  }
  var user string  var password string
  if val, ok := md["user"]; ok {    user = val[0]  }  if val, ok := md["password"]; ok {    password = val[0]  }
  if user != "admin" || password != "admin" {    return grpc.Errorf(codes.Unauthenticated, "invalid token")  }
  return nil}

copy the code

metadata.FromIncomingContext the user name and password are read from the context, and then compared with the actual data to determine whether it is authenticated.

interceptors:

var authInterceptor grpc.UnaryServerInterceptorauthInterceptor = func(  ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler,) (resp interface{}, err error) {  //拦截普通方法请求，验证 Token  err = Auth(ctx)  if err != nil {    return  }  // 继续处理请求  return handler(ctx, req)}

copy the code

initialize:

server := grpc.NewServer(  grpc.UnaryInterceptor(    grpc_middleware.ChainUnaryServer(      authInterceptor,      grpc_validator.UnaryServerInterceptor(),    ),  ),  grpc.StreamInterceptor(    grpc_middleware.ChainStreamServer(      grpc_validator.StreamServerInterceptor(),    ),  ),)

copy the code

In addition to the validators above, there are more Token authentication interceptors.authInterceptor

finally, there is the client transformation, which needs to implement the interface.PerRPCCredentials

type PerRPCCredentials interface {    // GetRequestMetadata gets the current request metadata, refreshing    // tokens if required. This should be called by the transport layer on    // each request, and the data should be populated in headers or other    // context. If a status code is returned, it will be used as the status    // for the RPC. uri is the URI of the entry point for the request.    // When supported by the underlying implementation, ctx can be used for    // timeout and cancellation.    // TODO(zhaoq): Define the set of the qualified keys instead of leaving    // it as an arbitrary string.    GetRequestMetadata(ctx context.Context, uri ...string) (        map[string]string,    error,    )    // RequireTransportSecurity indicates whether the credentials requires    // transport security.    RequireTransportSecurity() bool}

copy the code

GetRequestMetadata the method returns the necessary information required for authentication, the method indicates whether to enable safe link, which is generally enabled in a production environment, but for the convenience of testing, it is not enabled here for the time being.RequireTransportSecurity

implement the interface:

type Authentication struct {  User     string  Password string}
func (a *Authentication) GetRequestMetadata(context.Context, ...string) (  map[string]string, error,) {  return map[string]string{"user": a.User, "password": a.Password}, nil}
func (a *Authentication) RequireTransportSecurity() bool {  return false}

copy the code

connect:

conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure(), grpc.WithPerRPCCredentials(&auth))

copy the code

Okay, now our service has token authentication. If the username or password is incorrect, the client receives:

2021/10/11 20:39:35 rpc error: code = Unauthenticated desc = invalid tokenexit status 1

copy the code

if the user name and password are correct, you can return normally.

one-way certificate authentication

there are two ways to authenticate a certificate:

1. one-way authentication
2. two-way authentication

first, let’s take a look at the one-way authentication method:

generate a certificate

Start by generating a self-signed SSL certificate through the openssl tool.

1. generate a private key:

openssl genrsa -des3 -out server.pass.key 2048

copy the code

2. remove the password from the private key:

openssl rsa -in server.pass.key -out server.key

copy the code

3. generate a csr file:

openssl req -new -key server.key -out server.csr -subj "/C=CN/ST=beijing/L=beijing/O=grpcdev/OU=grpcdev/CN=example.grpcdev.cn"

copy the code

4. generate certificate:

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

copy the code

To say a little more, let’s look at the three files that an X.509 certificate contains: key, csr, and crt.

• key: a private key file on the server that is used to encrypt data sent to the client and decrypt data received from the client.
• csr: A certificate signing request file that is submitted to a certificate authority (CA) for signing the certificate.
• crt: A certificate signed by a certificate authority (CA), or a developer-signed certificate that contains information about the certificate holder, the holder’s public key, and the signer’s signature.

gRPC code

once the certificate is available, all that’s left is to transform the program, starting with the server-side code.

// 证书认证-单向认证creds, err := credentials.NewServerTLSFromFile("keys/server.crt", "keys/server.key")if err != nil {  log.Fatal(err)  return}
server := grpc.NewServer(grpc.Creds(creds))

copy the code

there are only a few lines of code to modify, and it’s simple, followed by the client.

since it is a one-way authentication, you do not need to generate a separate certificate for the client, you only need to copy the crt file on the server side to the corresponding directory of the client.

// 证书认证-单向认证creds, err := credentials.NewClientTLSFromFile("keys/server.crt", "example.grpcdev.cn")if err != nil {  log.Fatal(err)  return}conn, err := grpc.Dial("localhost:50051", grpc.WithTransportCredentials(creds))

copy the code

well, now our service supports one-way certificate authentication.

but before it’s over, there may be a problem here:

2021/10/11 21:32:37 rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0"exit status 1

copy the code

The reason is that Go 1.15 began to deprecate CommonName, and SAN certificates are recommended. If you want to be compatible with the previous way, you can support it by setting the environment variables, as follows:

export GODEBUG="x509ignoreCN=0"

copy the code

Note, however, that as of Go 1.17, environment variables no longer take effect and must be san-styled. Therefore, in order to upgrade the subsequent Go version, it is better to support it as soon as possible.

two-way certificate authentication

finally, let’s look at two-way certificate authentication.

GENERATE A CERTIFICATE WITH A SAN

IT’S STILL A CERTIFICATE, BUT THIS TIME IT’S A LITTLE DIFFERENT, WE NEED TO GENERATE A CERTIFICATE WITH SAN EXTENSIONS.

WHAT IS A SAN?

A SAN (Subject Alternative Name) is an extension defined in the SSL standard x509. An SSL certificate that uses a SAN field extends the domain names supported by this certificate so that a single certificate can support the resolution of multiple different domain names.

Copies the default OpenSSL configuration file to the current directory.

Linux systems in:

/etc/pki/tls/openssl.cnf

copy the code

Mac systems in:

/System/Library/OpenSSL/openssl.cnf

copy the code

modify the temporary configuration file, locate the paragraph, and then remove the comments for the following statement.[ req ]

req_extensions = v3_req # The extensions to add to a certificate request

copy the code

then add the following configuration:

[ v3_req ]# Extensions to add to a certificate request
basicConstraints = CA:FALSEkeyUsage = nonRepudiation, digitalSignature, keyEnciphermentsubjectAltName = @alt_names
[ alt_names ]DNS.1 = www.example.grpcdev.cn

copy the code

[ alt_names ] the location can be configured with multiple domain names, such as:

[ alt_names ]DNS.1 = www.example.grpcdev.cnDNS.2 = www.test.grpcdev.cn

copy the code

for the convenience of testing, only one domain name is configured here.

1. generate a ca certificate:

openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -subj "/CN=example.grpcdev.com" -days 5000 -out ca.pem

复制代码

2. generate server-side certificate:

# 生成证书openssl req -new -nodes \    -subj "/C=CN/ST=Beijing/L=Beijing/O=grpcdev/OU=grpcdev/CN=www.example.grpcdev.cn" \    -config <(cat openssl.cnf \        <(printf "[SAN]\nsubjectAltName=DNS:www.example.grpcdev.cn")) \    -keyout server.key \    -out server.csr    # 签名证书openssl x509 -req -days 365000 \    -in server.csr -CA ca.pem -CAkey ca.key -CAcreateserial \    -extfile <(printf "subjectAltName=DNS:www.example.grpcdev.cn") \    -out server.pem

copy the code

3. generate a client certificate:

# 生成证书openssl req -new -nodes \    -subj "/C=CN/ST=Beijing/L=Beijing/O=grpcdev/OU=grpcdev/CN=www.example.grpcdev.cn" \    -config <(cat openssl.cnf \        <(printf "[SAN]\nsubjectAltName=DNS:www.example.grpcdev.cn")) \    -keyout client.key \    -out client.csr
# 签名证书openssl x509 -req -days 365000 \    -in client.csr -CA ca.pem -CAkey ca.key -CAcreateserial \    -extfile <(printf "subjectAltName=DNS:www.example.grpcdev.cn") \    -out client.pem

copy the code

gRPC code

next, start modifying the code, first looking at the server:

// 证书认证-双向认证// 从证书相关文件中读取和解析信息，得到证书公钥、密钥对cert, _ := tls.LoadX509KeyPair("cert/server.pem", "cert/server.key")// 创建一个新的、空的 CertPoolcertPool := x509.NewCertPool()ca, _ := ioutil.ReadFile("cert/ca.pem")// 尝试解析所传入的 PEM 编码的证书。如果解析成功会将其加到 CertPool 中，便于后面的使用certPool.AppendCertsFromPEM(ca)// 构建基于 TLS 的 TransportCredentials 选项creds := credentials.NewTLS(&tls.Config{  // 设置证书链，允许包含一个或多个  Certificates: []tls.Certificate{cert},  // 要求必须校验客户端的证书。可以根据实际情况选用以下参数  ClientAuth: tls.RequireAndVerifyClientCert,  // 设置根证书的集合，校验方式使用 ClientAuth 中设定的模式  ClientCAs: certPool,})

copy the code

look at the client:

// 证书认证-双向认证// 从证书相关文件中读取和解析信息，得到证书公钥、密钥对cert, _ := tls.LoadX509KeyPair("cert/client.pem", "cert/client.key")// 创建一个新的、空的 CertPoolcertPool := x509.NewCertPool()ca, _ := ioutil.ReadFile("cert/ca.pem")// 尝试解析所传入的 PEM 编码的证书。如果解析成功会将其加到 CertPool 中，便于后面的使用certPool.AppendCertsFromPEM(ca)// 构建基于 TLS 的 TransportCredentials 选项creds := credentials.NewTLS(&tls.Config{  // 设置证书链，允许包含一个或多个  Certificates: []tls.Certificate{cert},  // 要求必须校验客户端的证书。可以根据实际情况选用以下参数  ServerName: "www.example.grpcdev.cn",  RootCAs:    certPool,})

copy the code

home and dry.

Python client

As mentioned earlier, gRPC is cross-language, so at the end of this article we will write a client in Python to request the Go server.

use the simplest way to achieve:

the proto file uses the proto file of the original “base template”:

syntax = "proto3";
package proto;
// The greeting service definition.service Greeter {    // Sends a greeting    rpc SayHello (HelloRequest) returns (HelloReply) {}    // Sends stream message    rpc SayHelloStream (stream HelloRequest) returns (stream HelloReply) {}}
// The request message containing the user's name. message HelloRequest {    string name = 1;}
// The response message containing the greetingsmessage HelloReply {    string message = 1;}

copy the code

similarly, the pb.py file needs to be generated from the command line:

python3 -m grpc_tools.protoc -I . --python_out=. --grpc_python_out=. ./*.proto

copy the code

after successful execution, two files, helloworld_pb2.py and helloworld_pb2_grpc.py, are generated in the directory.

this process may also report errors:

ModuleNotFoundError: No module named 'grpc_tools'

copy the code

don’t panic, it is missing packages, install it:

pip3 install grpciopip3 install grpcio-tools

copy the code

最后看一下 Python 客户端代码：

import grpc
import helloworld_pb2import helloworld_pb2_grpc

def main():    channel = grpc.insecure_channel("127.0.0.1:50051")    stub = helloworld_pb2_grpc.GreeterStub(channel)    response = stub.SayHello(helloworld_pb2.HelloRequest(name="zhangsan"))    print(response.message)

if __name__ == '__main__':    main()

复制代码

In this way, you can request Go-enabled server-side services through the Python client.

summary

This article uses the actual combat perspective and directly speaks in code to illustrate some applications of gRPC.

Content includes simple gRPC services, stream processing patterns, validators, token authentication, and certificate authentication.

IN ADDITION TO THIS, THERE ARE OTHER THINGS WORTH INVESTIGATING, SUCH AS TIMEOUT CONTROL, REST INTERFACES, AND LOAD BALANCING. IN THE FUTURE, WE WILL FIND TIME TO CONTINUE TO IMPROVE THE REST OF THE CONTENT.

The code in this article has been tested and verified, can be executed directly, and has been uploaded to GitHub, and small partners can read the source code again and against the content of the article to learn.