Understanding authorisation in Charmed Kubeflow

charm
,
1

Charmed Kubeflow has multi-tenant authorisation features out of the box. This page will explore how authorisation works in Charmed Kubeflow and step through setting it up for multi-tenant use. The examples will require a running Kubeflow environment. See the the quickstart guide to get up and running on MicroK8s if you need a cluster to test with.

Contents:

An overview of the Kubeflow permissions model

Kubeflow is a solution that is deeply integrated with Kubernetes, and thus Kubeflow makes use of the features of Kubernetes to implement authorisation of different kinds of access to resources. In Kubernetes, a resource is a thing that you want to interact with. Typical examples of Kubernetes resources are pods, services, persistentVolumes and secrets.

Typically, you interact with resources in different ways, for example you might get, delete, edit or describe a Kubernetes resource.

The services that Kubeflow offers are also essentially resources. For example, an experiment, pipeline or notebook are all resources that Kubeflow offers for users to interact with. So in Kubeflow, access to those resources is controlled by an integration with Kubernetes, so that the Kubeflow resources are evaluated by the Kubernetes platform, like any other Kubernetes resource, when considering whether to grant or deny access to a user.

Kubernetes offers three different options for implementing the rules that govern access to Kubernetes resources - ABAC, RBAC and Webhooks.

  • ABAC (Attribute Based Access Control) allows Kubernetes authorisation policies to be declared in policy files
  • RBAC (Role Based Access Control) allows Kubernetes authorisation policies to be declared using roles that you assign to groups of users or individual users
  • Webhooks allow you to send authorisation requests to a custom policy service that you provide, which decides whether a resource access request should be granted or denied.

You can learn more about these three different options in the Kubernetes documentation.

Charmed Kubeflow implements RBAC based policy for governing access to Kubeflow resources, for example for accessing Katib experiments. Policies are configured automatically, so that users can only access things associated with their workspace, and may not access resources associated with another user’s workspace.

For example, if user1 tries to list the experiments that are associated with the workspace of user2, their action will be denied by policy:

curl 'http://kubeflow.ubuntu.local/pipeline/apis/v1beta1/experiments?resource_reference_key.type=NAMESPACE&resource_reference_key.id=user2' -H 'Cookie: authservice_session=XXXX'
{"error":"Failed to authorize with API resource references: Failed to authorize with API resource references: PermissionDenied: User 'user1@example.org' is not authorized with reason: (request: ResourceAttributes{Namespace:test-user1,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:experiments,Subresource:,Name:,}): Unauthorized access","code":7,"message":"Failed to authorize with API resource references: Failed to authorize with API resource references: PermissionDenied...

However, the same list request against the user’s own experiments succeeds:

curl 'http://kubeflow.ubuntu.local/pipeline/apis/v1beta1/experiments?resource_reference_key.type=NAMESPACE&resource_reference_key.id=user1' -H 'Cookie: authservice_session=XXXXX'
{}

It doesn’t matter how you try to access the resources - whether via curl or the python SDK, or via kubectl, or through the browser, Charmed Kubeflow has a consistent way to regulate access using the underlying mechanisms of Kubernetes.

Important to know In Kubeflow, pipelines are currently considered a shared resource, so they are always visible for all users.

Getting set up for authorisation

If your Kubernetes environment has RBAC authorisation enabled, you will already be able to use the authorisation features of Charmed Kubeflow. But if not, you will need to enable RBAC for the cluster. In MicroK8s that’s as straightforward as running the following command:

microk8s enable rbac

That’s pretty much all you need to do!

Now follow the guide to configuring Charmed Kubeflow with OpenLDAP and you will have a multi-tenant MLOps platform with pretty robust role-based access controls ready-configured.

Testing who can access what

Testing what resources users can and cannot access can be done with kubectl, the Swiss army knife of Kubernetes:

microk8s.kubectl auth can-i list "experiments.kubeflow.org" -nuser2 --as test.user2@example.org
yes
microk8s.kubectl auth can-i list "experiments.kubeflow.org" -nuser1 --as test.user2@example.org
no

User authorisation is a straightforward, out-of-the-box experience in Charmed Kubeflow. Hopefully now you’ll have a better understanding of how the Charmed Kubeflow MLOps platform implements authorisation. But if you have questions, don’t worry - head over to the forum to ask a question and get in touch with the community.

Further reading

1 Like