HA for kubeapi-load-balancer
The standard deployment of Charmed Kubernetes includes a single instance of the kube-api-loadbalancer. For many use cases this is perfectly adequate, but in a production environment you should be keen to eliminate any single point of failure.
The recommended way to provide a failover for the kube-api-loadbalancer on bare metal or MAAS is by using keepalived. This is available as a Juju charm and can be deployed into your Charmed Kubernetes model and configured as follows:
juju deploy cs:~containers/keepalived
Add the required relations:
juju add-relation keepalived:juju-info kubeapi-load-balancer:juju-info juju add-relation keepalived:lb-sink kubeapi-load-balancer:website juju add-relation keepalived:loadbalancer kubernetes-master:loadbalancer juju add-relation keepalived:website kubernetes-worker:kube-api-endpoint
This redirects both the Kubernetes master and worker units to point at the keepalived service rather than the api-endpoint directly.
Configure the keepalived application. You should substitute a suitable IP address and FQDN in the example below:
export VIP=10.10.74.250 export VIP_HOSTNAME=test.example.com juju config keepalived virtual_ip=$VIP juju config keepalived vip_hostname=$VIP_HOSTNAME
Allocating a VIP and ensuring that it can route to all of the instances is a manual process which depends on your infrastructure. It does require that the VIP be able to route to each instance, and that the VRRP protocol is allowed on the network. While this should be the case on bare metal and MAAS, and can be made to work on OpenStack, it will generally not be possible on public clouds. Thus, it is generally better to in those cases to replace kubeapi-load-balancer with a cloud-provided load balancer with health checks, such as Octavia or ELB.
Add both the new hostname and VIP to the API server certificate. This is done by specifying additional SANs:
juju config kubeapi-load-balancer extra_sans="$VIP $VIP_HOSTNAME" juju config kubernetes-master extra_sans="$VIP $VIP_HOSTNAME"
Wait for the new service to settle. You can check the status of the
keepalivedapplication by running:
juju status keepalived
Once the application reports a 'ready' status, continue to the next step.
Remove unneeded relations:
juju remove-relation kubernetes-worker:kube-api-endpoint kubeapi-load-balancer:website juju remove-relation kubernetes-master:loadbalancer kubeapi-load-balancer:loadbalancer
Scale up the
kubeapi-load-balancer. You can specify as many units as your situation requires. In this example, we add two additional units for a total of three:
juju add-unit kubeapi-load-balancer -n 2
Check for correct functionality by using kubectl and verifying it returns results. You can also check the SANs listed in the certificate returned by the VIP.
kubectl get pods --all-namespaces openssl s_client -connect $VIP:443 | openssl x509 -noout -text
Note that the
keepalived application is a
subordinate charm - it does not require a machine of its
own to run on, but rather runs alongside the
kubeapi-load-balancer charm. If
for any reason you need to view logs or troubleshoot this
application, it will be found co-located on the machines running the load