Historical Archive Mirror Setup

Identifying the Challenges

After understanding our architectural requirements, the next step was to figure out how to manage each individual Ubuntu Archive deployment. A non exhaustive list of challenges we needed to address were:

• How do we manage the lifecycle of each Ubuntu Archive Mirror service?
• Can we be sure that each environment is consistent and that the user experience remains the same no matter which Archive Mirror environment you are working on?
• How can we test proposed changes before upgrading production services?
• How do we respond to incidents when things go wrong with any of our environments and how do we ensure each environment is isolated from each other to avoid unnecessary outages?

Our engineers can’t get bogged down with the semantics of each cloud. The best approach is to have a uniform interface that applies to any platform. This saves engineering time by making our playbooks more succinct and easier to digest, there is less room for error when there are minimal commands to run, a consistent UX reduces mean time to recovery as an engineer can be confident that the same troubleshooting methodology can be applied no matter which environment they are in, finally, environment specifications stored in version control allow us to keep track of changes, and easily test and rollback revisions that do not work.

ubuntu-repository-cache lp:ubuntu-repository-cache;overwrite=True
haproxy                 cs:haproxy
nrpe                    cs:nrpe

The actual service and relation specifications detail how each charm is related and any configuration options, i.e. Squid cache size and HAProxy monitor settings. For a look at the specification please see the appendix at the end of the post. Mojo uses codetree to pull down the charms, from either the charmstore or version control (e.g. launchpad.net or github.com), to the host and copies them to a charm repository for deployment. A phased approach protects us from incorrectly configured specifications as Mojo will not proceed if a phase fails. At the end of this process we have a new validated environment deployed and ready for use.

You can try this out for yourself, download the example mojo spec.

1. Before we begin you will need to have Juju, Mojo and a Juju controller bootstrapped. You can find a tutorial on how to set this up here.
2. Configure a Mojo project and workspace which we will use to build and test our archive mirror environment locally before it gets deployed into our cloud.

export MOJO_STAGE=production
export MOJO_WORKSPACE=archive-mirror
export MOJO_PROJECT=mojo-archive-mirror-proj
export MOJO_SERIES=xenial
export MOJO_SPEC=~/mojo/mojo-archive-mirror-spec/
export MOJO_LOCAL=~/mojo/LOCAL/mojo-archive-mirror-spec/

juju bootstrap

mkdir -p ${MOJO_SPEC}; mkdir -p ${MOJO_LOCAL};
(cd ~/mojo; curl https://ubuntu.com/wp-content/uploads/db69/mojo-archive-mirror-spec.tar.gz | tar xv)

mojo project-new -s $MOJO_SERIES -c lxd
mojo workspace-new --project $MOJO_PROJECT -s $MOJO_SERIES $MOJO_SPEC $MOJO_WORKSPACE
mojo run

Mojo has now orchestrated a deployment of our Archive Mirror environment. Below details the instances that were created and the applications installed onto them.

$ juju status

We know our new Archive Mirror  is functioning because the verify phase in our manifest has curl’d the HAProxy unit and downloaded the list of supported Ubuntu series.

2018-07-16 06:53:34 [INFO] Running script verify-deploy
2018-07-16 06:53:35 [INFO] /ubuntu/
artful-backports/
artful-proposed/
artful-security/
artful-updates/
artful/
bionic-backports/
bionic-proposed/
bionic-security/
bionic-updates/
bionic/
[...]
2018-07-16 06:53:35 [INFO] Completed script verify-deploy in 0s (0.31s)

We keep track of each Archive Mirror deployment that we manage with stages embedded in the root of the Mojo spec directory. These stages are nothing more than sub-directories that Mojo looks out for. Under each stage is the information used by each Mojo phase.

The directory layout looks a little something like this:

mojo-archive-mirror-spec # repository root.
├── manifest # the manifest script, this is what mojo run is looking for.
└── production
├── collect # Contains the list of charms that the collect phase will download.
├── haproxy-services.yaml
├── relations # Specifies the contracts between each service.
└── services # Details how each service should be configured.

Mojo uses this approach when sourcing local significant configuration. This comes in handy when you want to do something like configure a service with dummy credentials for the CI deployment. If we wanted to add a new stage, like ci to the list of deployments we manage, we only need to create a new directory and populate the required files:

mojo-archive-mirror-spec
├── manifest
└── production
├── collect
[...]
└── ci
├── collect


Any changes made to the Archive Mirror specification are tested. Our testing environment is a mixture of technologies: Jenkins, Openstack, Mojo and Juju. The testing procedure is broken down as follows:

• Jenkins job pulls the latest Mojo specification from version control.
• A new juju model is created for the revision of the specification. The model is deployed into one of our private OpenStack clouds.
• The Jenkins script then executes a Mojo run which deploys the Archive Mirror service to the new Juju model.
• Jenkins monitors the progress and provides feedback on any errors.

This process gives us assurance that the changes we have made are deployable albeit only within an Openstack context.