Charmed MySQL K8s - Async Replication

WARNING: it is an internal article. Do NOT use it in production! Contact Canonical Data Platform team if you are interested in the topic.

Creating a MySQL Cluster Set

1 - Deploy two MySQL clusters, named ROME and LISBON:

ROME:

juju switch az1
juju deploy mysql-k8s db1 --trust --channel=8.0/edge/arepl --config profile=testing --config cluster-name=ROME --base ubuntu@22.04

LISBON:

juju switch az2
juju deploy mysql-k8s db2 --trust --channel=8.0/edge/arepl --config profile=testing --config cluster-name=LISBON --base ubuntu@22.04

2 - Create and consume an offer

It’s required to define the roles that the clusters will play in the cluster-set, i.e. which will be the primary and the standby. For this setup, the application db1 will be setup as the primary cluster and db2 will be the replica/standby cluster.

NOTE: The side of the relation is used for the setup phase only. In an event of a planned switchover or a failover, a standby cluster can be promoted to active independently of the relation side.

To create a cluster set from this two clusters, we need to create a relation, which uses the async_replication interface, through the async-primaryand async-replica relation names. But first it’s necessary to create and consume an offer:

juju switch az1
juju offer db1:async-primary async-primary
juju switch az2
juju consume az1.async-primary

3 - Relate the applications

juju switch az2
juju relate async-primary db2:async-replica

And wait until the process is finished. Behind the scenes the replica cluster will be setup as a replica, cloning data from the primary cluster and rejoining all units to it.

4 - Checking cluster-set status

Run the get-cluster-status with the cluster-set=Trueflag:

juju run db1/0 get-cluster-status cluster-set=True

5 - Scaling clusters

The two clusters works independently, this means that it’s possible to independently scaling in/out each cluster without much hassle, e.g.:

juju scale-application db1 5

NOTE: resource usage configurations are also independent.

Safe removing a cluster from the cluster set

For removing a given cluster from the cluster set, you need be sure that the given cluster is not the primary cluster. Case it is, there’s a provided action that can execute a safe switchover:

1 - (optional) Safely promote to active

juju run db2/leader promote-standby-cluster cluster-set-name=my-cluster-set

It’s required to provide the cluster-set-name option as a foolproof method.

2 - Remove the relation

When removing a async_replication relation, the primary cluster will keep working as is, while the replica cluster will be dissolved with all it’s units in standalone read-only mode.

The side of the relation (primary/replica) does not matter, since only the current role of the cluster will be considered.

juju remove-relation async-primary db2

3 - Recovering blocked cluster

To recover a replica cluster after the relation is removed, there’s a provided action:

juju run db1/leader recreate-cluster

The action will recover the cluster as a standalone cluster and with the data from the cluster-set.

Failover

When a failover of the primary is required, it’s necessary to manually set cluster roles.

1 - Promote to active

To promote the standby cluster to active/primary, it’s necessary to run the action with the force flag set.

juju run db2/leader promote-standby-cluster cluster-set-name=my-cluster-set force=True

The force will cause the old primary to be invalidated. It’s required to provide the cluster-set-name option as a foolproof method.

2 - Fence writes from old primary

To avoid a split brain scenario, where more than one cluster is set as primary, it’s important to fence all write traffic from the failed primary cluster. For doing so there’s an action:

juju run db1/0 fence-writes cluster-set-name=my-cluster-set

The action can be run against any of the cluster units.

Case the old primary is reestablished and/or have all transactions reconciled, one can resume write traffic to it, by using the unfence-writes action, e.g.:

juju run db1/0 unfence-writes cluster-set-name=my-cluster-set