Configuration backup and restore
Apache Kafka configuration is distributed using Apache ZooKeeper.
An Apache ZooKeeper backup can be stored on any S3-compatible storage.
S3 access and configurations are managed with the s3-integrator charm, that can be integrated with Apache ZooKeeper.
This guide contains step-by-step instructions on how to deploy and configure the s3-integrator charm for AWS S3, send the configurations to Charmed Apache ZooKeeper, and finally manage your Apache ZooKeeper backups.
Configure s3-integrator
First, deploy the s3-integrator charm:
juju deploy s3-integrator
juju run s3-integrator/leader sync-s3-credentials access-key=<aws-access-key-id> secret-key=<aws-secret-key>
Then, use juju config to add your configuration parameters. For example:
juju config s3-integrator \
endpoint="https://s3.us-west-2.amazonaws.com" \
bucket="zk-backups-bucket-1" \
path="/zk-backups" \
region="us-west-2"
The only mandatory configuration parameter in the command above is the bucket.
Integrate with Charmed Apache ZooKeeper
To pass these configurations to Charmed Apache ZooKeeper, integrate the two applications:
juju integrate s3-integrator zookeeper
You can create, list, and restore backups now:
juju run zookeeper/leader create-backup
juju run zookeeper/leader list-backups
juju run zookeeper/leader restore backup-id=<backup-id-here>
Create a backup
Check that Charmed Apache ZooKeeper deployment with configurations set for S3 storage is active and idle with the juju status command. Once it’s active, create a backup with the create-backup command:
juju run zookeeper/leader create-backup
Apache ZooKeeper backups created with the command above will always be full backups: a copy of all the Apache Kafka configuration will be stored in S3.
The command will output the ID of the newly created backup:
Backup created
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Id ┃ Log-sequence-number ┃ Path ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 2024-09-12T14:32:46Z │ 8589934621 │ zookeeper_backups/2024-09-12T1 │
│ │ │ 4:32:46Z/snapshot │
└──────────────────────┴─────────────────────┴────────────────────────────────┘
List backups
To list available backups, run the list-backups command:
juju run zookeeper/leader list-backups
This command shows available backups, for example:
Backups
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Id ┃ Log-sequence-number ┃ Path ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 2024-09-12T14:32:46Z │ 8589934621 │ zookeeper_backups/2024-09-12T1 │
│ │ │ 4:32:46Z/snapshot │
│ 2024-09-12T14:32:00Z │ 8589934621 │ zookeeper_backups/2024-09-12T1 │
│ │ │ 4:32:00Z/snapshot │
│ 2024-09-12T14:26:12Z │ 8589934621 │ zookeeper_backups/2024-09-12T1 │
│ │ │ 4:26:12Z/snapshot │
└──────────────────────┴─────────────────────┴────────────────────────────────┘
Below is a list of parameters shown for each backup:
Id: identifier of the backup.Log-Sequence-number: a database-specific number to identify its state. Learn more about the Zxid on Apache ZooKeeper documentation.Path: path of the snapshot file in the S3 repository.
Restore a backup
This operation puts you at risk of losing unsaved configuration data. We recommend creating a backup first.
To restore from backup, run the restore command and pass the backup-id (in the YYYY-MM-DDTHH:MM:SSZ format) that is listed in the list-backups action output:
juju run zookeeper/leader restore backup-id=<backup-id-here>
The restore will then proceed. Follow its progress using juju status.