Building Globally Distributed Services using Kubernetes Cluster Federation
Editor's note: Today’s post is by Allan Naim, Product Manager, and Quinton Hoole, Staff Engineer at Google, showing how to deploy a multi-homed service behind a global load balancer and have requests sent to the closest cluster.
In Kubernetes 1.3, we announced Kubernetes Cluster Federation and introduced the concept of Cross Cluster Service Discovery, enabling developers to deploy a service that was sharded across a federation of clusters spanning different zones, regions or cloud providers. This enables developers to achieve higher availability for their applications, without sacrificing quality of service, as detailed in our
In the latest release,
Federated Replica Sets leverage the same configuration as non-federated Kubernetes Replica Sets and automatically distribute Pods across one or more federated clusters. By default, replicas are evenly distributed across all clusters, but for cases where that is not the desired behavior, we've introduced Replica Set preferences, which allow replicas to be distributed across only some clusters, or in non-equal proportions (). Starting with Google Cloud Platform (GCP), we’ve introduced Federated Ingress as a Kubernetes 1.4 alpha feature which enables external clients point to a single IP address and have requests sent to the closest cluster with usable capacity in any region, zone of the Federation. Federated Secrets automatically create and manage secrets across all clusters in a Federation, automatically ensuring that these are kept globally consistent and up-to-date, even if some clusters are offline when the original updates are applied. Federated Namespaces are similar to the traditional Kubernetes Namespaces providing the same functionality. Creating them in the Federation control plane ensures that they are synchronized across all the clusters in Federation. Federated Events are similar to the traditional Kubernetes Events providing the same functionality. Federation Events are stored only in Federation control plane and are not passed on to the underlying kubernetes clusters. Let’s walk through how all this stuff works. We’re going to provision 3 clusters per region, spanning 3 continents (Europe, North America and Asia). The next step is to federate these clusters. Kelsey Hightower developed a
The rest of the blog post assumes that you have a running Kubernetes Cluster Federation provisioned. Let’s verify that we have 9 clusters in 3 regions running.$ kubectl --context=federation-cluster get clusters
NAME STATUS AGE
gce-asia-east1-a Ready 17m
gce-asia-east1-b Ready 15m
gce-asia-east1-c Ready 10m
gce-europe-west1-b Ready 7m
gce-europe-west1-c Ready 7m
gce-europe-west1-d Ready 4m
gce-us-central1-a Ready 1m
gce-us-central1-b Ready 53s
gce-us-central1-c Ready 39s
In our example, we’ll be deploying the service and ingress object using the federated control plane. The ConfigMap object isn’t currently supported by Federation, so we’ll be deploying it manually in each of the underlying Federation clusters. Our cluster deployment will look as follows:
We’re going to deploy a Service that is sharded across our 9 clusters. The backend deployment will consist of a Pod with 2 containers:
- busybox container that fetches the zone and outputs an HTML with the zone embedded in it into a Pod volume mount path
- nginx container that reads from that Pod volume mount path and serves an HTML containing the zone it’s running in
Let’s start by creating a federated service object in the federation-cluster context.
$ kubectl --context=federation-cluster create -f services/nginx.yaml
It will take a few minutes for the service to propagate across the 9 clusters.
$ kubectl --context=federation-cluster describe services nginx
Name: nginx
Namespace: default
Labels: app=nginx
Selector: app=nginx
Type: LoadBalancer
IP:
LoadBalancer Ingress: 108.59.xx.xxx, 104.199.xxx.xxx, ...
Port: http 80/TCP
NodePort: http 30061/TCP
Endpoints: <none>
Session Affinity: None
Let’s now create a Federated Ingress. Federated Ingresses are created in much that same way as traditional Kubernetes Ingresses: by making an API call which specifies the desired properties of your logical ingress point. In the case of Federated Ingress, this API call is directed to the Federation API endpoint, rather than a Kubernetes cluster API endpoint. The API for Federated Ingress is 100% compatible with the API for traditional Kubernetes Services.
$ cat ingress/ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx
spec:
backend:
serviceName: nginx
servicePort: 80
$ kubectl --context=federation-cluster create -f ingress/ingress.yaml
ingress "nginx" created
Once created, the Federated Ingress controller automatically:
- 1.creates matching Kubernetes Ingress objects in every cluster underlying your Cluster Federation
- 2.ensures that all of these in-cluster ingress objects share the same logical global L7 (i.e. HTTP(S)) load balancer and IP address
- 3.monitors the health and capacity of the service “shards” (i.e. your Pods) behind this ingress in each cluster
- 4.ensures that all client connections are routed to an appropriate healthy backend service endpoint at all times, even in the event of Pod, cluster, availability zone or regional outages We can verify the ingress objects are matching in the underlying clusters. Notice the ingress IP addresses for all 9 clusters is the same.
$ for c in $(kubectl config view -o jsonpath='{.contexts[*].name}'); do kubectl --context=$c get ingress; done
NAME HOSTS ADDRESS PORTS AGE
nginx \* 80 1h
NAME HOSTS ADDRESS PORTS AGE
nginx \* 130.211.40.xxx 80 40m
NAME HOSTS ADDRESS PORTS AGE
nginx \* 130.211.40.xxx 80 1h
NAME HOSTS ADDRESS PORTS AGE
nginx \* 130.211.40.xxx 80 26m
NAME HOSTS ADDRESS PORTS AGE
nginx \* 130.211.40.xxx 80 1h
NAME HOSTS ADDRESS PORTS AGE
nginx \* 130.211.40.xxx 80 25m
NAME HOSTS ADDRESS PORTS AGE
nginx \* 130.211.40.xxx 80 38m
NAME HOSTS ADDRESS PORTS AGE
nginx \* 130.211.40.xxx 80 3m
NAME HOSTS ADDRESS PORTS AGE
nginx \* 130.211.40.xxx 80 57m
NAME HOSTS ADDRESS PORTS AGE
nginx \* 130.211.40.xxx 80 56m
