Load Balancing Secure gRPC traffic using Envoy proxy and Kubernetes
Nowadays, when we’re building, developing a project (whether it’s a web application, mobile application, etc) from scratch we try to break it into small pieces called microservices. We also try to add as much abstraction as we can between this different pieces to make the application scalable, high available and secure. In this article, I won’t describe the history of microservices and the different tools we can use nor explain how kubernetes (k8s) works but instead, I will focus on explaining how to deploy a gRPC application with different level of abstraction.
Getting started with the environment
First of all, I need to define some terms and tools that will be used in this article:
- Kubernetes: A container orchestration system (more information on https://kubernetes.io/docs/concepts/)
- Envoy proxy: Created by Lift, Envoy is a modern, high performance, edge and service proxy (https://www.envoyproxy.io/)
- gRPC: a Remote Procedure Call system developed by Google (https://grpc.io/)
- Load Balance: distributing a set of tasks to a set of resources to make the process more efficient (https://en.wikipedia.org/wiki/Load_balancing_(computing))
Alright, now that we defined the basics, let’s set up our environment. To follow this tutorial, you will need to install minikube on your machine (you can easily follow the official documentation to install it: https://minikube.sigs.k8s.io/docs/start/) and kubectl (https://kubernetes.io/docs/tasks/tools/install-kubectl/) and docker (https://docs.docker.com/get-docker/). I won’t show how to create an application that uses gRPC, instead, I’m gonna use the google bookstore application available on github (https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/endpoints/bookstore-grpc) and deploy it, so feel free to pull the repository.
A schema worth way more that words so here is the infrastructure of the project:
Basically, the client is going to send a request to get the different books store in the bookshelf of the application, provide a JSON Web Token (JWT), to authorize the connection, and a TLS certificate to authenticate. Then envoy is going to check both the JWT by checking a local JWKS (or a external one, e.g. a JWKS in a AWS S3 bucket), that contains all the public keys used to verify the JWT, and if the TLS certificate is legit.
Set-up the workspace
Now that we know what the infrastructure looks like, we can deploy everything in our minikube cluster.
First you need to create the minikube cluster:
minikube start --memory=4096You can also see kubernetes dashboard if you’d like with the minikube dashboard command.
Alright, now that the cluster is ready, we first need to build the image of the bookstore application in the minikube environment, to do so you need to run the following commands in the ./endpoints/bookstore-grpc/ folder:
eval $(minikube docker-env)
docker build -t bookstore .You can check that the images has been build in the minikube environment with docker images.
Before that we can deploy the infrastructure, we need to create our own SSL Certificate Authority to enable the TLS connection and our own token (JWT and JWKS)
Create our own SSL Certificate Authority
In the section, I won’t explain how SSL/TLS works, I’m just going to show you how to create the certificate.
So first, we need to become a certificate authority:
openssl genrsa -out rootCA.key 2048 # Create key for CA
openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1825 -out rootCA.pem # Root certificateYou need to install this root certificate to your client. In our case, you need to copy the certificate in the file called “roots.pem” which is the file used by the bookstore client when the use of TLS is enabled.
We can now sign certificates for our server (this is how envoy is going to check if the connection is secured or not):
# Create CA-signed certificates for servers
# The key
openssl genrsa -out dev.bookstore.com.key 2048
# CSR (Certificate Signing Request) file
openssl req -new -key dev.bookstore.com.key -out dev.bookstore.com.csrWe also need a file to generate the certificate because of the SAN:
dev.bookstore.com.ext:
echo "authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names[alt_names]
DNS.1 = <minikube IP> # <-- Replace by you own minikube IP
And now, we can create the certificate:
openssl x509 -req -in dev.bookstore.com.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out dev.bookstore.com.crt -days 825 -sha256 -extfile dev.bookstore.com.extA script is available on the github of the project (https://github.com/Sindvero/distributed-grpc-app/blob/master/create_ssl_cert.sh)
Create the Json Web Token and the Json Web Key Set
The last step in our set-up is to create, first a JWT. To do so, we need a private/public key pair:
ssh-keygenYou can enter any file name you like (mine is jwt_key). Now that we have a key pair, we can create our JWKS that envoy is going to use to verify the token we’ll create later. I’m using the following website to create my JWKS: https://russelldavies.github.io/jwk-creator/
You’ll need to specify that the “Public Key Use” is “Signing”, the algorithm is “RS256” and paste the public key that you just generate and the website open because we’ll need the JWK later for our yaml file.
Finally, we need to generate our token, to do so, you can use my python script to generate you token and you can modify it as you’d like: https://github.com/Sindvero/distributed-grpc-app/blob/master/gen_jwt.py. Keep this token somewhere as we’ll need it later to test our deployment.
If you want to learn more about JWT, please visit the official website (https://jwt.io/)
Deploying the Infrastructure
Now, that we have everything, we can finally create the yaml files and deploy the infrastructure (all the yaml file are available in my github: https://github.com/Sindvero/distributed-grpc-app)
First of all we need to create all the yaml files needed for the application:
Let’s first create the yaml file to deploy the bookstore application: deployment-bookstore.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: bookstore-deployment
spec:
replicas: 3
selector:
matchLabels:
app: bookstore-app
template:
metadata:
labels:
app: bookstore-app
spec:
containers:
- name: bookstore-container
image: bookstore:latest
imagePullPolicy: Never
ports:
- containerPort: 8080Feel free to change anything you’d like but keep the containerPort at 8080 because it’s on this port that the boostore application is listening.
Now, we need to make our application accessible with a kubernetes service: boostore-svc.yaml:
apiVersion: v1
kind: Service
metadata:
name: bookstore-service
spec:
selector:
app: bookstore-app
type: ClusterIP
clusterIP: None
ports:
- port: 8080I chose ClusterIP as type without an IP instead of NodePort to avoid too many hops when envoy is going to redirect the traffic.
Let’s create the configmap to push our jwt-key to the minikube cluster so that envoy could use it:
apiVersion: v1
kind: ConfigMap
metadata:
name: jwt-key
data:
jwt_key.json: |
{
"keys": [
{
"kty": "RSA",
"n": [Your own n],
"e": "AQAB",
"alg": "RS256",
"use": "sig"
}
]
}Remember the JWK you created, this is where you need to paste it, in the data part, just the jwt_key.json of your jwt-configmap.yaml file.
let’s also create a secret to push our certificate and key to the cluster so that envy could use it:
kubectl create secret tls envoy-certs --key dev.bookstore.com.key --cert dev.bookstore.com.crt --dry-run -o yaml | kubectl apply -f -Replace the key and cert with your own, you can also choose to create a secret yaml file for your secret.
Now let’s create the yaml file for envoy:
- The envoy configmap for the configuration file needed by envoy:
apiVersion: v1
kind: ConfigMap
metadata:
name: envoy-conf
data:
envoy.yaml: |
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address:
protocol: TCP
address: 127.0.0.1
port_value: 8090
static_resources:
listeners:
- name: bookstore-listener
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 8000
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_https
route_config:
name: local_route
virtual_hosts:
- name: https
domains: ["*"]
routes:
- match:
prefix: "/"
route:
cluster: bookstore-service
http_filters:
- name: envoy.filters.http.jwt_authn
config:
providers:
oidc_provider:
issuer: test@gmail.com
# audiences:
# - bookstore
local_jwks:
filename: /etc/jwt/jwt_key.json
rules:
- match: { prefix: "/" }
requires:
provider_name: oidc_provider
- name: envoy.filters.http.grpc_web
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.grpc_web.v3.GrpcWeb
- name: envoy.filters.http.cors
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
alpn_protocols: "h2,http/1.1"
tls_certificates:
- certificate_chain:
filename: "/etc/ssl/envoy/tls.crt"
private_key:
filename: "/etc/ssl/envoy/tls.key"
clusters:
- name: bookstore-service
connect_timeout: 0.5s
type: STRICT_DNS
dns_lookup_family: V4_ONLY
http2_protocol_options: {}
lb_policy: ROUND_ROBIN
# tls_context: {}
load_assignment:
cluster_name: bookstore-service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: bookstore-service
port_value: 8080if you have the same names as mine, you can copy paste this file without changing anything. If not, you’ll need to change the address in the clusters -> endpoint -> socket address with yous. It needs to be the name of the service of your application (Kubernetes uses DNS resolution so no need to worry about that). If you change the issuers of the JWT in the python script, you also need to change it in the yaml file.
- The envoy deployment to deploy envoy:
apiVersion: apps/v1
kind: Deployment
metadata:
name: envoy
spec:
replicas: 1
selector:
matchLabels:
app: envoy
template:
metadata:
labels:
app: envoy
spec:
containers:
- name: envoy
image: envoyproxy/envoy-alpine:v1.16.0
command: ["envoy"]
args: ["-l", "trace", "-c", "/etc/envoy/envoy.yaml"]
ports:
- name: https
containerPort: 8000
volumeMounts:
- name: config
mountPath: /etc/envoy/
- name: certs
mountPath: /etc/ssl/envoy/
- name: jwt
mountPath: /etc/jwt/
volumes:
- name: config
configMap:
name: envoy-conf
- name: certs
secret:
secretName: envoy-certs
- name: jwt
configMap:
name: jwt-keyYou can choose to remove the debug command by removing the lines “command” and “args”. Please also make sure that the names are correct.
- And finally the envoy service to make it accessible outside of the cluster:
apiVersion: v1
kind: Service
metadata:
name: envoy
spec:
type: NodePort
selector:
app: envoy
ports:
- port: 9080
targetPort: 8000If you’re not using minikube you can replace NodePort by LoadBalancer
Now that we have everything, we can deploy our infrastructure:
kubectl apply -f jwt-configmap.yaml
kubectl apply -f envoy-configmap.yaml
kubectl apply -f deployment-boostore.yaml
kubectl apply -f bookstore-svc.yaml
kubectl apply -f envoy-deployment.yaml
kubectl apply -f envoy-service.yamlYou should be able to see everything deployed on your minikube dashboard or by running “kubectl get pod” and “kubectl get svc”, something like that:
Now it’s time to test the application. First we need to get the port on which minikube allow our envoy service to listen:
minikube service envoy --urlThen we just need to use the client given by Google:
python3 bookstore_client.py --host [Your IP] --port [Your Port] --use_tls USE_TLS --auth_token [Your own token]The auth_token is the token you generated in the set-up part. You should a result like this:
You can also deploy this application adding my helm repo: https://sindvero.github.io/distributed-grpc-app/ and using my helm chart.
So, now we know how to deploy a high available, scalable and secure application using Envoy proxy, Kubernetes and gRPC protocol. It would be interesting to deploy the same infrastructure using an Infrastructure as Code (IaC) like Terraform or Pulumi.
