Charmed Kafka Documentation - How to deploy on GCE

How to deploy on GCE

Google Compute Engine is a popular subsidiary of Google that provides on-demand cloud computing resources on a metered pay-as-you-go basis. Access the GCloud web console at console.cloud.google.com.

Summary


Install Google Cloud CLI and Juju tooling

Install Juju via snap:

sudo snap install juju
sudo snap install google-cloud-cli --classic

Check the official the Google Cloud (GCloud) CLI documentation about other installation options.

To check they are all correctly installed, you can run the commands demonstrated below with sample outputs:

~$ juju version
...
3.5.4-genericlinux-amd64

~$ gcloud --version
Google Cloud SDK 498.0.0
...

Authenticate

Login to GCloud:

gcloud auth login

Follow the instructions in the browser and enter the verification code provided. Once successful, you should see a message similar to this:

You are now logged in as [user@domain.com].
...

Create a new project

To manage cloud resources more efficiently, it is advised that you create a separate project. Run the following command in order to create a new project named juju-kafka in Google Cloud.

~$ gcloud projects create juju-kafka
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/juju-kafka].
Waiting for [operations/cp.7114461363952122971] to finish...done.
Enabling service [cloudapis.googleapis.com] on project [juju-kafka]...
Operation "operations/acat.p2-626433822898-1be21ace-ce42-4a84-973f-6246d9549297" finished successfully.

Switch to the newly created project for the next steps:

~$ gcloud config set project juju-kafka
Updated property [core/project].

Authorize

Create an IAM service account for Juju to operate GCE; The steps would be:

  • Create a service account
  • Create a private key for the mentioned service account
  • Add IAM policy-binding for the service account and grant it the compute.admin role
> gcloud iam service-accounts create juju-gce-account --display-name="Juju GCE service account"
Created service account [juju-gce-account].

> gcloud iam service-accounts list
DISPLAY NAME              EMAIL                                                DISABLED
Juju GCE service account  juju-gce-account@juju-kafka.iam.gserviceaccount.com  False

> gcloud iam service-accounts keys create sa-private-key.json \
 --iam-account=juju-gce-account@juju-kafka.iam.gserviceaccount.com
created key [...redacted...] of type [json] as [sa-private-key.json] for [juju-gce-account@juju-kafka.iam.gserviceaccount.com]

> gcloud projects add-iam-policy-binding juju-kafka --role=roles/compute.admin \
--member serviceAccount:juju-gce-account@canonical-data-123456.iam.gserviceaccount.com

Bootstrap Juju controller on GCE

Note: move newly exported GCloud jsonfile into SNAP accessible folder due to the known Juju issue.

sudo mv sa-private-key.json /var/snap/juju/common/sa-private-key.json
sudo chmod a+r /var/snap/juju/common/sa-private-key.json

Add GCE credentials to Juju:

> juju add-credential google
...
Enter credential name: juju-gce-account
...

Auth Types
  jsonfile
  oauth2

Select auth type [jsonfile]: jsonfile

Enter path to the .json file containing a service account key for your project
Path: /var/snap/juju/common/sa-private-key.json

Credential "juju-gce-account" added locally for cloud "google".

Bootstrap Juju controller (check all supported configuration options):

juju bootstrap google gce
Output example
> juju bootstrap google gce
Creating Juju controller "gce" on google/us-east1
Looking for packaged Juju agent version 3.5.4 for amd64
Located Juju agent version 3.5.4-ubuntu-amd64 at https://streams.canonical.com/juju/tools/agent/3.5.4/juju-3.5.4-linux-amd64.tgz
Launching controller instance(s) on google/us-east1...
 - juju-367a0f-0 (arch=amd64 mem=3.6G cores=4)
Installing Juju agent on bootstrap instance
Waiting for address
Attempting to connect to 35.227.15.149:22
Attempting to connect to 10.142.0.2:22
Connected to 35.227.15.149
Running machine configuration script...
Bootstrap agent now started
Contacting Juju controller at 35.227.15.149 to verify accessibility...

Bootstrap complete, controller "gce" is now available
Controller machines are in the "controller" model

Now you can run
        juju add-model <model-name>
to create a new model to deploy workloads.

You can check the GCE instance availability (ensure the right GCloud project chosen!):

Deploy charms

Create a new Juju model, if needed:

juju add-model <MODEL_NAME>

(Optional) Increase the debug level if you are troubleshooting charms:

juju model-config logging-config='<root>=INFO;unit=DEBUG'

Then, Charmed Kafka can be deployed as usual. Please note that the default instance types on GCP Compute may not be suitable for Kafka deployments. We recommend selecting an instance type that provides at the very least 8 GB of RAM and 4 cores. For more guidance on production environment sizing, see the Requirements page. You can find more information about the available instance types in the Azure documentation.

Please also note that you may need to increase your project’s quotas to be able to spin up properly sized instances.

In order to deploy and integrate Kafka and ZooKeeper, you could run:

juju deploy zookeeper -n3 --channel 3/stable
juju deploy kafka -n3 --constraints "cores=4 mem=8G" --channel 3/stable
juju integrate kafka zookeeper

We also recommend to deploy Data Integrator in order to create an admin user to manage the content of the Kafka cluster:

juju deploy data-integrator admin --channel edge \
  --config extra-user-roles=admin \
  --config topic-name=admin-topic

And integrate it with the Kafka application:

juju integrate kafka admin

For more information on Data Integrator and how to use it, please refer to the how-to manage applications guide.

Clean up

Always clean GCE resources that are no longer necessary - they could be costly!

To destroy the Juju controller and remove GCE instance (warning: all your data will be permanently removed):

> juju controllers
Controller  Model    User   Access     Cloud/Region     Models  Nodes    HA  Version
gce*        welcome  admin  superuser  google/us-east1       2      1  none  3.5.4  

> juju destroy-controller gce --destroy-all-models --destroy-storage --force

Next, check and manually delete all unnecessary GCloud resources, to show the list of all your GCE instances run the following command (make sure the correct region used!):

gcloud compute instances list
Output example
Listed 0 items.

List your Juju credentials:

> juju credentials
...
Client Credentials:
Cloud        Credentials
google       juju-gce-account
...

Remove GCloud credentials from Juju:

> juju remove-credential google juju-gce-account

Finally, remove GCloud jsonfile user credentials (to avoid forgetting and leaking):

rm -f /var/snap/juju/common/sa-private-key.json