---
title: "Setting up multi-site replication\n    | Ubuntu"
description: Ubuntu is an open source software operating system that runs from the
  desktop, to the cloud, to all your internet connected things.
url: https://ubuntu.com/ceph/docs/setting-up-multi-site-replication?format=md
keywords: index, follow
---

*Submit*

# Setting up multi-site replication

## [Overview](https://ubuntu.com/ceph/docs/setting-up-multi-site-replication?format=md#overview)

Here we discuss setting up [RADOS Gateway multi-site replication](https://ubuntu.com/ceph/docs/multi-site-replication) in a charmed Ceph deployment. Native replication between ceph-radosgw applications is supported via `juju relations`. By default, both primary and secondary RGW applications accept write operations (i.e. active-active replication is configured).

## [Prerequisites](https://ubuntu.com/ceph/docs/setting-up-multi-site-replication?format=md#prerequisites)

In order to setup multi-site, we need two Ceph clusters (one primary and one secondary) both backing separate ceph-radosgw applications. For simplicity we will refer to these as primary-ceph-radosgw and secondary-ceph-radosgw.

## [Multi-site configuration](https://ubuntu.com/ceph/docs/setting-up-multi-site-replication?format=md#multi-site-configuration)

Multi-site configuration is done using charm configs for the respective primary and secondary ceph-radosgw application.

Example bundles (\*-multisite.yaml) with primary and secondary Ceph applications can be found at [/test/bundles/](https://opendev.org/openstack/charm-ceph-radosgw/src/branch/master/tests/bundles) subdirectory of the [ceph-radosgw charm](https://opendev.org/openstack/charm-ceph-radosgw).

As will be shown in the below sections, multi-site can either be deployed directly or by having an existing cluster migrated to multi-site by means of a new secondary cluster.

### [Configuring multi-site as initial deployment](https://ubuntu.com/ceph/docs/setting-up-multi-site-replication?format=md#configuring-multi-site-as-initial-deployment)

To configure multi-site in an initial deployment, ensure that the following charm configurations are set for both the primary and secondary ceph-radosgw applications in the bundle file.

```
    # Primary Ceph rgw config
    primary-ceph-radosgw:
        realm: replicated
        zonegroup: us
        zone: us-east
    # Secondary Ceph rgw config
    secondary-ceph-radosgw:
        realm: replicated
        zonegroup: us
        zone: us-west
```

### [Multi-site migration for an existing Ceph cluster](https://ubuntu.com/ceph/docs/setting-up-multi-site-replication?format=md#multi-site-migration-for-an-existing-ceph-cluster)

To configure multi-site for an existing Ceph cluster, configuration can be done directly via Juju commands:

```
# Primary Ceph configuration
juju config primary-ceph-radosgw realm=replicated
juju config primary-ceph-radosgw zonegroup=us
juju config primary-ceph-radosgw zone=us-east

# Secondary Ceph configuration
juju config primary-ceph-radosgw realm=replicated
juju config primary-ceph-radosgw zonegroup=us
juju config primary-ceph-radosgw zone=us-west
```

The realm and zonegroup configuration must be identical between instances of the primary and secondary ceph-radosgw application participating in the multi-site deployment; the zone configuration must be unique per application.

## [Relating the primary and secondary RGW applications](https://ubuntu.com/ceph/docs/setting-up-multi-site-replication?format=md#relating-the-primary-and-secondary-rgw-applications)

Once the ceph-radosgw applications are configured, they will go into a blocked state until the primary/secondary relation is added.

```
primary-ceph-radosgw/0*    blocked  idle  0  10.5.2.103  80/tcp  multi-site configuration but primary/secondary relation missing
secondary-ceph-radosgw/0*  blocked  idle  1  10.5.2.197  80/tcp  multi-site configuration but primary/secondary relation missing
```

The multi-site relation can be added as:

```
juju integrate primary-ceph-radosgw:primary secondary-ceph-radosgw:secondary
```

Once the integration has been added the realm, zonegroup and zone configuration will be created in the primary deployment and then synced to the secondary deployment.

The current sync status can be validated from either rgw app using the `radosgw-admin sync status` command:

```
sudo radosgw-admin --id=rgw.$(hostname -s) sync status
          realm 3a7acb0c-c7a3-4b74-8d52-43a330c7a9f0 (replicated)
          zonegroup 170a884a-6610-40f8-aec8-24117aa7f088 (us)
          zone 77218517-8bf7-4c29-af0e-bb5312861cf8 (us-east)
    metadata sync no sync (zone is master)
    data sync source: aac90d5a-f4ac-4dd7-acbe-61879b7ea4b8 (us-west)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        data is caught up with source
```

## [Cross model relations](https://ubuntu.com/ceph/docs/setting-up-multi-site-replication?format=md#cross-model-relations)

When the Ceph clusters are managed through separate Juju models, we would need a cross model relationship to establish the relation between the primary and secondary RGW applications.

One ceph-radosgw application acts as the initial primary for the deployment - setup the primary relation endpoint as the provider of the offer for the cross-model relation (here for simplicity, `pmodel` and `smodel` are taken to be the respective primary and secondary model names):

```
juju offer -m pmodel primary-ceph-radosgw:primary
```

The cross-model relation offer can then be consumed in the other model and related to the secondary ceph-radosgw application:

```
juju consume -m smodel admin/pmodel.primary-ceph-radosgw
juju integrate -m smodel secondary-ceph-radosgw:secondary primary-ceph-radosgw:primary
```

Once the deployment is complete, the default zone and zonegroup can optionally be tidied using the ‘tidydefaults’ action:

```
juju run -m smodel --wait secondary-ceph-radosgw/0 tidydefaults
```

## [Read/write vs read-only](https://ubuntu.com/ceph/docs/setting-up-multi-site-replication?format=md#readwrite-vs-read-only)

By default all zones within a deployment will be read/write capable but this can be configured according to need using the readonly and readwrite actions.

Non-primary zones can be made as read-only by using the ‘readonly’ action:

```
juju run -m smodel --wait secondary-ceph-radosgw/0 readonly
```

A zone that is currently read-only can be switched to read/write mode by using the ‘readwrite’ action:

```
juju run -m smodel --wait secondary-ceph-radosgw/0 readwrite
```

---

[Previous

Managing Ceph cluster:
Removing MONs](https://ubuntu.com/ceph/docs/removing-mons)
[Next

Recovering from an outage with multi-site replication](https://ubuntu.com/ceph/docs/recovering-from-an-outage-with-multi-site-replication)

This page was last modified 2 years ago. [Help improve this document in the forum](https://discourse.ubuntu.com/t/setting-up-multi-site-replication/36025).