This guide describes authorisation in Charmed Kubeflow (CKF) and how to set it up for multi-tenant use.
The provided examples require a running Kubeflow environment. See Get started for more details.
Kubeflow permissions model
Kubeflow is a solution that is deeply integrated with Kubernetes, and thus Kubeflow makes use of the features of Kubernetes to implement authorisation of different kinds of access to resources. In Kubernetes, a resource is a thing that you want to interact with. Typical examples of Kubernetes resources are pods, services, persistentVolumes and secrets.
Typically, you interact with resources in different ways, for example you might get, delete, edit or describe a Kubernetes resource.
The services that Kubeflow offers are also essentially resources. For example, an experiment, pipeline or notebook are all resources that Kubeflow offers for users to interact with. So in Kubeflow, access to those resources is controlled by an integration with Kubernetes, so that the Kubeflow resources are evaluated by the Kubernetes platform, like any other Kubernetes resource, when considering whether to grant or deny access to a user.
Kubernetes offers three different options for implementing the rules that govern access to Kubernetes resources - ABAC, RBAC and Webhooks.
- ABAC (Attribute Based Access Control) allows Kubernetes authorisation policies to be declared in policy files
- RBAC (Role Based Access Control) allows Kubernetes authorisation policies to be declared using roles that you assign to groups of users or individual users
- Webhooks allow you to send authorisation requests to a custom policy service that you provide, which decides whether a resource access request should be granted or denied.
You can learn more about these three different options in the Kubernetes documentation.
Charmed Kubeflow implements RBAC based policy for governing access to Kubeflow resources, for example for accessing Katib experiments. Policies are configured automatically, so that users can only access things associated with their workspace, and may not access resources associated with another user’s workspace.
For example, if user1 tries to list the experiments that are associated with the workspace of user2, their action will be denied by policy:
curl 'http://kubeflow.ubuntu.local/pipeline/apis/v1beta1/experiments?resource_reference_key.type=NAMESPACE&resource_reference_key.id=user2' -H 'Cookie: authservice_session=XXXX'
{"error":"Failed to authorize with API resource references: Failed to authorize with API resource references: PermissionDenied: User 'user1@example.org' is not authorized with reason: (request: ResourceAttributes{Namespace:test-user1,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:experiments,Subresource:,Name:,}): Unauthorized access","code":7,"message":"Failed to authorize with API resource references: Failed to authorize with API resource references: PermissionDenied...
However, the same list request against the user’s own experiments succeeds:
curl 'http://kubeflow.ubuntu.local/pipeline/apis/v1beta1/experiments?resource_reference_key.type=NAMESPACE&resource_reference_key.id=user1' -H 'Cookie: authservice_session=XXXXX'
{}
It doesn’t matter how you try to access the resources - whether via curl or the python SDK, or via kubectl, or through the browser, Charmed Kubeflow has a consistent way to regulate access using the underlying mechanisms of Kubernetes.
In Kubeflow, pipelines are currently considered a shared resource, so they are always visible for all users.
Enable authorisation
If your Kubernetes environment has RBAC authorisation enabled, you will already be able to use the authorisation features of CKF. But if not, you will need to enable RBAC for the cluster. In MicroK8s that’s as straightforward as running the following command:
microk8s enable rbac
Test user access
Testing user access can be done with kubectl:
microk8s.kubectl auth can-i list "experiments.kubeflow.org" -nuser2 --as test.user2@example.org
yes
microk8s.kubectl auth can-i list "experiments.kubeflow.org" -nuser1 --as test.user2@example.org
no