New installation Kubeflow on microk8s - error

When i install kubeflow

ops.pebble.ExecError: non-zero exit code 1 executing ['/usr/bin/mysqlsh', '--passwords-from-stdin', '--uri=serverconfig@katib-db-0.katib-db-endpoints.kubeflow.svc.cluster.local.:33062', '--python', '--verbose=0', '-c', "shell.options.set('useWizards', False)\nprint('###')\ncluster = dba.create_cluster('cluster-28203b3910d690dd5badc53a77b2d60e', {'communicationStack': 'MySQL'})\ncluster.set_instance_option('katib-db-0.katib-db-endpoints.kubeflow.svc.cluster.local.', 'label', 'katib-db-0')"], stdout="\x1b[1mPlease provide the password for 'serverconfig@katib-db-0.katib-db-endpoints.kubeflow.svc.cluster.local.:33062': \x1b[0m###\n+----------------------------------------+---------------+----------------+--------------------------------------------------+\n| Variable                               | Current Value | Required Value | Note                                             |+----------------------------------------+---------------+----------------+--------------------------------------------------+\n| binlog_transaction_dependency_tracking | COMMIT_ORDER  | WRITESET       | Update the server variable                       || server_id                              | 1             | <unique ID>    | Update read-only variable and restart the server |", stderr='Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\nA new InnoDB Cluster will be created on instance \'katib-db-0.katib-db-endpoints.kubeflow.svc.cluster.local.:3306\'.\n\nValidating instance configuration at katib-db-0.katib-db-endpoints.kubeflow.svc.cluster.local.:33062...\n\nThis instance reports its own address as \x1b[1mkatib-db-0.katib-db-endpoints.kubeflow.svc.cluster.local.:3306\x1b[0m\n\n\x1b[36mNOTE: \x1b[0mSome configuration options need to be fixed:\n\n\n\n+----------------------------------------+---------------+----------------+--------------------------------------------------+\n\nSome variables need to be changed, but cannot be done dynamically on the server.\n\x1b[36mNOTE: \x1b[0mPlease use the dba.configure_instance() command to repair these issues.\n\n\x1b[31mERROR: \x1b[0mInstance must be configured and validated with dba.check_instance_configuration() and dba.configure_instance() before it can be used in an InnoDB cluster.\nTraceback (most recent call last):\n  File "<string>", line 3, in <module>\nRuntimeError: Dba.' [truncated]

microk8s config | juju add-k8s ml-k8s --client --storage=nfs-csi6
juju bootstrap ml-k8s ctlmk8s
juju add-model kubeflow
juju deploy kubeflow --trust --channel=1.10/stable

juju status
Model     Controller  Cloud/Region      Version  SLA          Timestamp
kubeflow  ctlmk8s     ml-k8s/localhost  3.6.9    unsupported  08:17:15+03:00

App                      Version                  Status       Scale  Charm                    Channel           Rev  Address         Exposed  Message
admission-webhook                                 active           1  admission-webhook        1.10/stable       467  10.152.183.120  no       
argo-controller                                   active           1  argo-controller          3.5/stable        818  10.152.183.91   no       
dex-auth                                          active           1  dex-auth                 2.41/stable       699  10.152.183.55   no       
envoy                                             active           1  envoy                    2.4/stable        406  10.152.183.53   no       
istio-ingressgateway                              active           1  istio-gateway            1.24/stable      1474  10.152.183.63   no       
istio-pilot                                       active           1  istio-pilot              1.24/stable      1417  10.152.183.175  no       
jupyter-controller                                active           1  jupyter-controller       1.10/stable      1301  10.152.183.166  no       
jupyter-ui                                        active           1  jupyter-ui               1.10/stable      1222  10.152.183.116  no       
katib-controller                                  active           1  katib-controller         0.18/stable      1163  10.152.183.118  no       
katib-db                 8.0.41-0ubuntu0.22.04.1  maintenance      1  mysql-k8s                8.0/stable        255  10.152.183.245  no       Starting mysqld
katib-db-manager                                  waiting          1  katib-db-manager         0.18/stable      1123  10.152.183.219  no       Incorrect/incomplete data found in relation relational-db. See logs
katib-ui                                          active           1  katib-ui                 0.18/stable      1122  10.152.183.229  no       
kfp-api                                           waiting          1  kfp-api                  2.5/stable       2240  10.152.183.22   no       Incorrect/incomplete data found in relation relational-db. See logs
kfp-db                   8.0.41-0ubuntu0.22.04.1  maintenance      1  mysql-k8s                8.0/stable        255  10.152.183.117  no       Starting mysqld
kfp-metadata-writer                               active           1  kfp-metadata-writer      2.5/stable       1310  10.152.183.30   no       
kfp-persistence                                   blocked          1  kfp-persistence          2.5/stable       2251  10.152.183.69   no       [relation:kfp-api] Expected data from exactly 1 related applications - got 0.
kfp-profile-controller                            active           1  kfp-profile-controller   2.5/stable       2211  10.152.183.36   no       
kfp-schedwf                                       active           1  kfp-schedwf              2.5/stable       2261  10.152.183.26   no       
kfp-ui                                            blocked          1  kfp-ui                   2.5/stable       2253  10.152.183.56   no       [relation:kfp-api] Expected data from exactly 1 related applications - got 0.
kfp-viewer                                        active           1  kfp-viewer               2.5/stable       2281  10.152.183.192  no       
kfp-viz                                           active           1  kfp-viz                  2.5/stable       2197  10.152.183.222  no       
knative-eventing                                  active           1  knative-eventing         1.16/stable       866  10.152.183.160  no       
knative-operator                                  active           1  knative-operator         1.16/stable       827  10.152.183.77   no       
knative-serving                                   active           1  knative-serving          1.16/stable       865  10.152.183.237  no       
kserve-controller                                 active           1  kserve-controller        0.14/stable       951  10.152.183.24   no       
kubeflow-dashboard                                active           1  kubeflow-dashboard       1.10/stable       847  10.152.183.234  no       
kubeflow-profiles                                 active           1  kubeflow-profiles        1.10/stable       643  10.152.183.203  no       
kubeflow-roles                                    active           1  kubeflow-roles           1.10/stable       357  10.152.183.216  no       
kubeflow-volumes                                  active           1  kubeflow-volumes         1.10/stable       514  10.152.183.235  no       
metacontroller-operator                           active           1  metacontroller-operator  4.11/stable       510  10.152.183.248  no       
minio                    res:oci-image@7f2474f    active           1  minio                    ckf-1.10/stable   583  10.152.183.122  no       
mlmd                                              active           1  mlmd                     ckf-1.10/stable   344  10.152.183.182  no       
oidc-gatekeeper                                   active           1  oidc-gatekeeper          ckf-1.10/stable   556  10.152.183.251  no       
pvcviewer-operator                                active           1  pvcviewer-operator       1.10/stable       293  10.152.183.252  no       
tensorboard-controller                            active           1  tensorboard-controller   1.10/stable       557  10.152.183.129  no       
tensorboards-web-app                              active           1  tensorboards-web-app     1.10/stable       543  10.152.183.221  no       
training-operator                                 active           1  training-operator        1.9/stable        653  10.152.183.230  no       

Unit                        Workload     Agent      Address      Ports          Message
admission-webhook/0*        active       idle       10.1.149.1                  
argo-controller/0*          active       idle       10.1.149.2                  
dex-auth/0*                 active       idle       10.1.149.6                  
envoy/0*                    active       idle       10.1.149.36                 
istio-ingressgateway/0*     active       idle       10.1.149.61                 
istio-pilot/0*              active       idle       10.1.149.18                 
jupyter-controller/0*       active       idle       10.1.149.23                 
jupyter-ui/0*               active       idle       10.1.149.52                 
katib-controller/0*         active       idle       10.1.149.58                 
katib-db-manager/0*         waiting      idle       10.1.149.49                 Incorrect/incomplete data found in relation relational-db. See logs
katib-db/0*                 maintenance  executing  10.1.149.26                 Starting mysqld
katib-ui/0*                 active       idle       10.1.149.24                 
kfp-api/0*                  waiting      idle       10.1.149.28                 Incorrect/incomplete data found in relation relational-db. See logs
kfp-db/0*                   maintenance  executing  10.1.149.20                 Starting mysqld
kfp-metadata-writer/0*      active       idle       10.1.149.11                 
kfp-persistence/0*          blocked      idle       10.1.149.51                 [relation:kfp-api] Expected data from exactly 1 related applications - got 0.
kfp-profile-controller/0*   active       idle       10.1.149.34                 
kfp-schedwf/0*              active       idle       10.1.149.21                 
kfp-ui/0*                   blocked      idle       10.1.149.48                 [relation:kfp-api] Expected data from exactly 1 related applications - got 0.
kfp-viewer/0*               active       idle       10.1.149.45                 
kfp-viz/0*                  active       idle       10.1.149.31                 
knative-eventing/0*         active       idle       10.1.149.9                  
knative-operator/0*         active       idle       10.1.149.39                 
knative-serving/0*          active       idle       10.1.149.38                 
kserve-controller/0*        active       idle       10.1.149.4                  
kubeflow-dashboard/0*       active       idle       10.1.149.46                 
kubeflow-profiles/0*        active       idle       10.1.149.41                 
kubeflow-roles/0*           active       idle       10.1.149.10                 
kubeflow-volumes/0*         active       idle       10.1.149.43                 
metacontroller-operator/0*  active       idle       10.1.149.7                  
minio/0*                    active       idle       10.1.149.8   9000-9001/TCP  
mlmd/0*                     active       idle       10.1.149.32                 
oidc-gatekeeper/0*          active       idle       10.1.149.60                 
pvcviewer-operator/0*       active       idle       10.1.149.54                 
tensorboard-controller/0*   active       idle       10.1.149.12                 
tensorboards-web-app/0*     active       idle       10.1.149.27                 
training-operator/0*        active       idle       10.1.149.62  

administrator@dtc1-ml4-n01:~$ microk8s version
MicroK8s v1.32.9 revision 8511
administrator@dtc1-ml4-n01:~$ juju version
3.6.9-genericlinux-amd64


Hi! Thank you for raising this.

There seems to be an issue with the MySQL charm. I will check with the team developing the MySQL operator, but meanwhile, could you give it a go to deploy this using Terraform?

YAML are being deprecated in the Juju ecosystem, and the current recommended way to deploy is by using Terraform. We are therefore currently focusing more on providing support for Terraform (with continuous testing and integration with other teams) over maintaining that legacy bundle

Thanks Enrico