Private Docker Registries and the Canonical Distribution of Kubernetes

This originally appeared on Tim Van Steenburgh’s blog

How do I use a private image registry with my Kubernetes cluster? How do I set up my own registry? Let’s look at how to perform these tasks on the Canonical Distribution of Kubernetes (CDK).

Using an Existing Insecure Registry

In order to connect to an insecure registry, the Docker daemon must be reconfigured and an --insecure-registry option must be added.

This can be done directly via Juju, using the command:

juju config kubernetes-worker docker-config=”--insecure-registry registry.domain.com:5000"

Creating a Secure CDK Registry

CDK provides an option to deploy a secure Docker registry within the cluster, and expose it via an ingress.

Note: The registry provided is not a production grade registry, and should not be used in a production context.

Requirements

To deploy and use the provided registry, you will need:

  • A DNS entry (registry.acme.com) pointing at the ingress of the cluster (directly, via DNS round robin or with a load balancer)
  • A valid TLS certificate and key for registry.acme.com (registry.crt and registry.key)
  • A set of usernames and passwords stored in a file for htpasswd authentication (format: username:password, one user per line)

Considering a htpasswd.cleartxt file filled with users, the following loop will generate an encoded version of it:

while read line
do
  USER=$(echo ${line} | cut -f1 -d':')
  PASS=$(echo ${line} | cut -f2 -d':')
  docker run \
    --rm \
    xmartlabs/htpasswd \
    ${USER} ${PASS} \
    | tee -a htpasswd.enc
done < htpasswd.cleartxt
sed -i "/^$/d" htpasswd.enc

Deployment

To deploy the registry, run:

juju run-action kubernetes-worker/0 registry \
  domain=registry.acme.com \
  htpasswd=”$(base64 -w0 htpasswd.enc)” \
  htpasswd-plain=”$(base64 -w0 htpasswd.cleartxt)” \
  tlscert=”$(base64 -w0 registry.crt)” \
  tlskey=”$(base64 -w0 registry.key)” \
  ingress=true

Tear down

To tear down the registry, run

juju run-action kubernetes-worker/0 registry \
  delete=true \
  ingress=true

Storage

The registry provided by CDK will use a /srv/registry hostPath to store the images. This means that in case of a rescheduling of the registry (failure, overload…), if the new pod is scheduled on a different host, you will lose your images.

Alternatively, you can use a network mount such as NFS on all workers to benefit from a single point of storage for the images.

Ingress Configuration

The CDK registry action makes the assumption that the ingress running is nginx and will enforce a change of the configuration to increase the client_max_body_size from 1MB to 1GB. This is done via a patch, hence will not overwrite other configuration keys.

If you are using another ingress, deploy with ingress=false and make sure your ingress will support image upload (typical images are ~300MB, and typical CUDA images are 1 to 4GB)

Alternatives

If you want a similar setup but with flexibility on the storage and management via native Kubernetes tools you will find a derived work delivered via a Helm chart on https://github.com/madeden/charts

If you’d like to follow along more closely with CDK development, you can do so in the following places:

Until next time!

Ubuntu cloud

Ubuntu offers all the training, software infrastructure, tools, services and support you need for your public and private clouds.

Newsletter signup

Select topics you’re
interested in

In submitting this form, I confirm that I have read and agree to Canonical’s Privacy Notice and Privacy Policy.

Related posts

Declarative vs Imperative: DevOps done right

Deciding whether to automate workloads, while designing your ICT infrastructure, is trivial. It’s 2019 and automation is everywhere around. However, deciding which DevOps paradigm to choose and which tool to use, may not be that obvious. …

Charmed Kubernetes update for upstream API server vulnerability

An upstream Kubernetes vulnerability (CVE-2019-11247) has been identified where the API server mistakenly allows access to a cluster-scoped custom resource, if the request is made as if the resource were namespaced. Authorisations for the resource accessed …

Edge computing monitoring with Kubernetes

This blog demonstrates how to easily deploy monitoring tools at the edge using Kubernetes. In IoT scenarios, such a deployment brings the benefits of privacy, latency and minimal bandwidth cost. Luckily, MicroK8s the single node Kubernetes, …