Spark Client Snap How-to - Run on Charmed Kubernetes

doc tutorials
, ,
1

Launch Jobs on Charmed Kubernetes

Setup

In order to setup a Charmed Kubernetes (assuming that you don’t have a Kubernetes cluster yet), you need to install Juju first. Afterwards you can install Charmed Kubernetes by following the Installation Guide and then the Setup Guide.

Once the Charmed Kubernetes cluster is up and running, you can enable and run Spark jobs like this.

First thing is to set up the spark-client snap.

$ sudo snap install spark-client

Then you need to set up a service account for Spark job submission. Let’s create an account called spark in the default namespace.

$ spark-client.service-account-registry create --username spark --namespace default

Note For spark-submit, spark-shell and pyspark to work correctly, make sure DNS is enabled.

Spark Job Submission To Kubernetes Cluster

Let’s use the spark-submit utility (that’s part of the snap) to submit a Spark Pi job example.

$ spark-client.spark-submit --username spark --class org.apache.spark.examples.SparkPi local:///opt/spark/examples/jars/spark-examples_2.12-3.3.2.jar 100

Spark Shell

To invoke a Spark shell, run the following command.

$ spark-client.spark-shell --username spark

PySpark Shell

To launch a pyspark shell, run the following command.

$ spark-client.pyspark --username spark

Note Make sure the namespace provided to spark-submit is valid and the service account provided belongs to that namespace. To keep it simple, use the setup-spark-k8s script to create the account.