Setting up multi-site replication
Here we discuss setting up RADOS Gateway multi-site replication in a charmed Ceph deployment. Native replication between ceph-radosgw applications is supported via
juju relations. By default, both primary and secondary RGW applications accept write operations (i.e. active-active replication is configured).
Multi-site replication is supported by upstream starting with Ceph Luminous.
In order to setup multi-site, we need two Ceph clusters (one primary and one secondary) both backing separate ceph-radosgw applications. For simplicity we will refer to these as primary-ceph-radosgw and secondary-ceph-radosgw.
Multi-site configuration is done using charm configs for the respective primary and secondary ceph-radosgw application.
As will be shown in the below sections, multi-site can either be deployed directly or by having an existing cluster migrated to multi-site by means of a new secondary cluster.
Configuring multi-site as initial deployment
To configure multi-site in an initial deployment, ensure that the following charm configurations are set for both the primary and secondary ceph-radosgw applications in the bundle file.
# Primary Ceph rgw config primary-ceph-radosgw: realm: replicated zonegroup: us zone: us-east # Secondary Ceph rgw config secondary-ceph-radosgw: realm: replicated zonegroup: us zone: us-west
Multi-site migration for an existing Ceph cluster
To configure multi-site for an existing Ceph cluster, configuration can be done directly via Juju commands:
# Primary Ceph configuration juju config primary-ceph-radosgw realm=replicated juju config primary-ceph-radosgw zonegroup=us juju config primary-ceph-radosgw zone=us-east # Secondary Ceph configuration juju config primary-ceph-radosgw realm=replicated juju config primary-ceph-radosgw zonegroup=us juju config primary-ceph-radosgw zone=us-west
The realm and zonegroup configuration must be identical between instances of the primary and secondary ceph-radosgw application participating in the multi-site deployment; the zone configuration must be unique per application.
Relating the primary and secondary RGW applications
Once the ceph-radosgw applications are configured, they will go into a blocked state until the primary/secondary relation is added.
primary-ceph-radosgw/0* blocked idle 0 10.5.2.103 80/tcp multi-site configuration but primary/secondary relation missing secondary-ceph-radosgw/0* blocked idle 1 10.5.2.197 80/tcp multi-site configuration but primary/secondary relation missing
The multi-site relation can be added as:
juju add-relation primary-ceph-radosgw:primary secondary-ceph-radosgw:secondary
Once the relation has been added the realm, zonegroup and zone configuration will be created in the primary deployment and then synced to the secondary deployment.
The current sync status can be validated from either rgw app using the
radosgw-admin sync status command:
sudo radosgw-admin --id=rgw.$(hostname -s) sync status realm 3a7acb0c-c7a3-4b74-8d52-43a330c7a9f0 (replicated) zonegroup 170a884a-6610-40f8-aec8-24117aa7f088 (us) zone 77218517-8bf7-4c29-af0e-bb5312861cf8 (us-east) metadata sync no sync (zone is master) data sync source: aac90d5a-f4ac-4dd7-acbe-61879b7ea4b8 (us-west) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source
Cross model relations
When the Ceph clusters are managed through separate Juju models, we would need a cross model relationship to establish the relation between the primary and secondary RGW applications.
One ceph-radosgw application acts as the initial primary for the deployment - setup the primary relation endpoint as the provider of the offer for the cross-model relation (here for simplicity,
smodel are taken to be the respective primary and secondary model names):
juju offer -m pmodel primary-ceph-radosgw:primary
The cross-model relation offer can then be consumed in the other model and related to the secondary ceph-radosgw application:
juju consume -m smodel admin/pmodel.primary-ceph-radosgw juju add-relation -m smodel secondary-ceph-radosgw:secondary primary-ceph-radosgw:primary
Once the deployment is complete, the default zone and zonegroup can optionally be tidied using the ‘tidydefaults’ action:
juju run-action -m smodel --unit secondary-ceph-radosgw/0 tidydefaults
Read/write vs read-only
By default all zones within a deployment will be read/write capable but this can be configured according to need using the readonly and readwrite actions.
Non-primary zones can be made as read-only by using the ‘readonly’ action:
juju run-action -m smodel --wait secondary-ceph-radosgw/0 readonly
A zone that is currently read-only can be switched to read/write mode by using the ‘readwrite’ action:
juju run-action -m smodel --wait secondary-ceph-radosgw/0 readwrite