Version 1.6 of Kubeflow was released in September 2022, together with the bundles and charms which make up Charmed Kubeflow.
However, version 1.6 introduced some known breaking changes, so the upgrade path is not as straightforward as it might otherwise have been. It is recommended to simply deploy the latest version of Kubeflow rather than upgrade an existing installation. If there are compelling reasons to upgrade though, the following procedure can be followed.
Kubeflow 1.4 supported Kubernetes 1.21. Starting from Kubeflow 1.6, Kubernetes 1.22 is supported as well. If you wish to upgrade both, follow the Kubeflow procedure first and then upgrade your k8s cluster. See more info about microk8s upgrade here.
As this upgrade process removes the relations between the Kubeflow components, there will be service interruption until the upgrade is complete.
Contents:
- Remove existing relations
- Remove outdated charms
- Upgrade the charms
- Deploy additional charms
- Add relations
- Configure authentication
- Known post-upgrade issues
Remove existing relations
Check you are using the correct Juju model where Kubeflow is deployed:
juju models
The corresponding output will highlight the current model. You can switch to the appropriate model if required with:
juju switch kubeflow
The existing relations need to be removed then. You can see the current relations by running:
juju status --relations
Rather than manually removing the relations one by one, it is faster to invoke a short script:
juju status --relations | grep regular | awk '{print $1" "$2}' | xargs -l juju remove-relation
Remove outdated charms
To ensure a smooth upgrade, we will need to remove charms that introduced breaking changes.
It is recommended to back up the dex configuration first:
dex_username=$(juju config dex-auth static-username)
dex_password=$(juju config dex-auth static-password)
dex_public_url=$(juju config dex-auth public-url)
dex_connectors=$(juju config dex-auth connectors)
The charms can then be safely removed:
juju remove-application istio-ingressgateway --force --no-wait
juju remove-application istio-pilot --force --no-wait
juju remove-application dex-auth --force --no-wait
Upgrade the charms
Individual charms can now be upgraded to the latest versions. The charm upgrades shown here target a specific channel of the charms (the default 1.6 release stable version).
To find out the current channel and revision of a charm, run juju status.
juju refresh admission-webhook --channel 1.6/stable
juju refresh argo-controller --channel 3.3/stable
juju refresh argo-server --channel 3.3/stable
juju refresh jupyter-controller --channel 1.6/stable
juju refresh jupyter-ui --channel 1.6/stable
juju refresh kfp-api --channel 2.0/stable
juju refresh kfp-persistence --channel 2.0/stable
juju refresh kfp-profile-controller --channel 2.0/stable
juju refresh kfp-schedwf --channel 2.0/stable
juju refresh kfp-ui --channel 2.0/stable
juju refresh kfp-viewer --channel 2.0/stable
juju refresh kfp-viz --channel 2.0/stable
juju refresh kubeflow-dashboard --channel 1.6/stable
juju refresh kubeflow-profiles --channel 1.6/stable
juju refresh kubeflow-roles --channel 1.6/stable
juju refresh kubeflow-volumes --channel 1.6/stable
juju refresh metacontroller-operator --channel 2.0/stable
juju refresh minio --channel latest/stable
juju refresh oidc-gatekeeper --channel ckf-1.6/stable
juju refresh seldon-controller-manager --channel 1.14/stable
juju refresh training-operator --channel 1.5/stable
# Charms not included in the lite bundle
juju refresh katib-controller --channel 0.14/stable
juju refresh katib-db-manager --channel 0.14/stable
juju refresh katib-ui --channel 0.14/stable
juju refresh tensorboard-controller --channel 1.6/stable
juju refresh tensorboards-web-app --channel 1.6/stable
Deploy additional charms
Version 1.6 requires some charms to be deployed again:
juju deploy istio-gateway istio-ingressgateway --channel 1.11/stable --trust --config kind=ingress
juju deploy istio-pilot --channel 1.11/stable --trust --config default-gateway=kubeflow-gateway
juju deploy dex-auth --channel 2.31/stable --trust
Add relations
The charms now require relations to be added again to complete a working deployment. You do not need to wait for all the charms to be deployed to add these relations.
juju relate argo-controller minio
juju relate dex-auth:oidc-client oidc-gatekeeper:oidc-client
juju relate istio-pilot:ingress dex-auth:ingress
juju relate istio-pilot:ingress jupyter-ui:ingress
juju relate istio-pilot:ingress kfp-ui:ingress
juju relate istio-pilot:ingress kubeflow-dashboard:ingress
juju relate istio-pilot:ingress kubeflow-volumes:ingress
juju relate istio-pilot:ingress oidc-gatekeeper:ingress
juju relate istio-pilot:ingress-auth oidc-gatekeeper:ingress-auth
juju relate istio-pilot:istio-pilot istio-ingressgateway:istio-pilot
juju relate kfp-api kfp-db
juju relate kfp-api:kfp-api kfp-persistence:kfp-api
juju relate kfp-api:kfp-api kfp-ui:kfp-api
juju relate kfp-api:kfp-viz kfp-viz:kfp-viz
juju relate kfp-api:object-storage minio:object-storage
juju relate kfp-profile-controller:object-storage minio:object-storage
juju relate kfp-ui:object-storage minio:object-storage
juju relate kubeflow-profiles kubeflow-dashboard
# Relations not included in the lite bundle
juju relate istio-pilot:ingress tensorboards-web-app:ingress
juju relate istio-pilot:gateway tensorboard-controller:gateway
juju relate istio-pilot:ingress katib-ui:ingress
juju relate katib-db-manager katib-db
You can control the progress of the update by running:
watch -c juju status --color
Configure authentication
For authentication and allowing access to the dashboard service, some configuration will need to be added. You can set up new credentials or use the previous config:
juju config dex-auth static-username=$dex_username
juju config dex-auth static-password=$dex_password
juju config dex-auth public-url=$dex_public_url
juju config dex-auth connectors=$dex_connectors
If you have a different setup for MicroK8s, or you are adapting this tutorial for a different Kubernetes, you can find the URL required by examining the IP address of the istio-ingressgateway service. For example, you can determinine this information using kubectl:
microk8s kubectl -n kubeflow get svc istio-ingressgateway -o
jsonpath='{.status.loadBalancer.ingress[0].ip}'
The dex-auth URL has to match oidc-gatekeeper’s URL.
Known post-upgrade issues
Training-operator charm may get blocked with message Patching resources failed with code 404. In that case, run the following commands:
wget https://raw.githubusercontent.com/canonical/bundle-kubeflow/4bc484456a60b69e60e6a28fe2d639ae829d24a0/docs/upgrade/mpijobs.yaml
microk8s kubectl apply -f mpijobs.yaml
juju run --unit training-operator/0 -- "export JUJU_DISPATCH_PATH=hooks/install; ./dispatch"