Charmed Apache Spark K8s Documentation - How to use the Python API

charm
, , ,
1

Manage Service Accounts using the Python API

The spark-client snap relies on the spark8t toolkit. spark8t provides both a CLI and a programmatic interface to enhanced Apache Spark client functionalities.

Here we describe how to use the spark8t toolkit (as part of the spark-client snap) to manage service accounts using Python.

Preparation

The spark8t package is already part of the SNAP. However, if the python package is used outside of the SNAP context, please make sure that environment settings (described on the tool’s README) are correctly configured.

Furthermore, you need to make sure that PYTHONPATH contains the location where the spark8t libraries were installed within the snap (something like /snap/spark-client/current/lib/python3.10/site-packages)

Bind to Kubernetes

The following snipped allows you to import relevant environment variables into a confined object, among which there should an auto-inference of your kubeconfig file location.

import os
from spark8t.domain import Defaults
from spark8t.services import KubeInterface

# Defaults for spark-client
defaults = Defaults(dict(os.environ))  # General defaults

# Create a interface connection to k8s
kube_interface = KubeInterface(defaults.kube_config)

Note that if you want to override some of these settings, you can extend the Default class accordingly.

Alternatively, you can also use auto-inference using the kubectl command via

from spark8t.services import KubeInterface

kube_interface = KubeInterface.autodetect(kubectl_cmd="kubectl")

Once bound to the k8s cluster, you have some properties of the connection readily available, e.g.

kube_interface.namespace
kube_interface.api_server

You can also issue some kubectl commands, using the exec method

service_accounts = kube_interface.exec("get sa -A")
service_accounts_namespace = kube_interface.exec(
    "get sa", namespace=kube_interface.namespace
)

Manage Spark Service Accounts

All functionalities for managing Apache Spark service accounts are embedded within the K8sServiceAccountRegistry that can be instantiated using the kube_interface object we defined above

from spark8t.services import K8sServiceAccountRegistry

registry = K8sServiceAccountRegistry(kube_interface)

Once this object is instantiated we can perform several operations, as outlined in the sections below

Create new Apache Spark service accounts

New Apache Spark service accounts can be created by first creating a ServiceAccount domain object, and optionally specifying extra-properties, e.g.

from spark8t.domain import PropertyFile, ServiceAccount

configurations = PropertyFile({"my-key": "my-value"})
service_account = ServiceAccount(
    name="my-spark",
    namespace="default",
    api_server=kube_interface.api_server,
    primary=False,
    extra_confs=configurations,
)

The account can then be created using the registry

service_account_id = registry.create(service_account)

This returns an id, which is effectively the {namespace}:{username}, e.g. “default:my-spark”.

Listing spark service accounts

Once Apache Spark service accounts have been created, these can be listed via

spark_service_accounts = registry.all()

or retrieved using their ids

retrieved_account = registry.get(service_account_id)

Delete service account

The registry can also be used to delete existing service accounts

registry.delete(primary_account_id)

or using an already existing ServiceAccount object:

registry.delete(service_account.id)

Manage Primary Accounts

spark8t and spark-client snap have the notation of the so-called ‘primary’ service account, the one that would be chosen by default if no specific account is provided. The primary Apache Spark service account can be set up using:

registry.set_primary(service_account_id)

or using an already existing ServiceAccount object:

registry.set_primary(service_account.id)

The primary Apache Spark service account can be retrieved using

primary_account = registry.get_primary()

Manage configurations of Spark service accounts

Apache Spark service accounts can have a configuration that is provided (unless overridden) during each execution of Spark jobs. This configuration is stored in the PropertyFile object, which can be provided on the creation of a ServiceAccount object (extra_confs argument).

The PropertyFile object can either be created from a dictionary, as done above

from spark8t.domain import PropertyFile

static_property = PropertyFile({"my-key": "my-value"})

or also read from a file, e.g.:

from spark8t.domain import PropertyFile

static_property = PropertyFile.read(defaults.static_conf_file)

PropertyFile objects can be merged using the + operator:

merged_property = static_property + service_account.extra_confs

And ServiceAccount properties can be updated using new “merged” properties via the API provided by the registry:

registry.set_configurations(service_account.id, merged_property)

Alternatively, you can also store these properties in files:

with open("my-file", "w") as fid:
    merged_property.log().write(fid)