MySQL error in Pipeline Creation

juju
1

Hello everyone, When I am trying to create a Pipeline or an Experiment or a Run (that has already run successfully in the past), I receive the following error: The kubeflow is installed using charmed-kubeflow, on a microk8s cluster

Failed to retry run Failed to retry run: f8b571bd-132d-4d2b-8739-efa2f00f3a36 with error: “{“error”:“InternalServerError: Failed to update the database entry.: InternalServerError: Failed to update run f8b571bd-132d-4d2b-8739-efa2f00f3a36. error: ‘Error 1290: The MySQL server is running with the --read-only option so it cannot execute this statement’: Error 1290: The MySQL server is running with the --read-only option so it cannot execute this statement”,“code”:13,“message”:“InternalServerError: Failed to update the database entry.: InternalServerError: Failed to update run f8b571bd-132d-4d2b-8739-efa2f00f3a36. error: ‘Error 1290: The MySQL server is running with the --read-only option so it cannot execute this statement’: Error 1290: The MySQL server is running with the --read-only option so it cannot execute this statement”,“details”:[{”@type":" type .googleapis.com/api. Error",“error_message”:“Internal Server Error”,“error_details”:“InternalServerError: Failed to update the database entry.: InternalServerError: Failed to update run f8b571bd-132d-4d2b-8739-efa2f00f3a36. error: ‘Error 1290: The MySQL server is running with the --read-only option so it cannot execute this statement’: Error 1290: The MySQL server is running with the --read-only option so it cannot execute this statement”}]}" ```

The juju status command shows that kserve-controller is in error state (hook failed :install)

The logs of the kserve-controller juju unit are represented below:

2025-02-25T13:36:21.061Z [container-agent] 2025-02-25 13:36:21 WARNING install Generating RSA private key, 2048 bit long modulus (2 primes) 2025-02-25T13:36:21.116Z [container-agent] 2025-02-25 13:36:21 WARNING install …+++++ 2025-02-25T13:36:21.120Z [container-agent] 2025-02-25 13:36:21 WARNING install …+++++ 2025-02-25T13:36:21.121Z [container-agent] 2025-02-25 13:36:21 WARNING install e is 65537 (0x010001) 2025-02-25T13:36:21.130Z [container-agent] 2025-02-25 13:36:21 WARNING install Generating RSA private key, 2048 bit long modulus (2 primes) 2025-02-25T13:36:21.149Z [container-agent] 2025-02-25 13:36:21 WARNING install …+++++ 2025-02-25T13:36:21.286Z [container-agent] 2025-02-25 13:36:21 WARNING install …+++++ 2025-02-25T13:36:21.286Z [container-agent] 2025-02-25 13:36:21 WARNING install e is 65537 (0x010001) 2025-02-25T13:36:21.316Z [container-agent] 2025-02-25 13:36:21 WARNING install Signature ok 2025-02-25T13:36:21.316Z [container-agent] 2025-02-25 13:36:21 WARNING install subject=C = GB, ST = Canonical, L = Canonical, O = Canonical, OU = Canonical, CN = 127.0.0.1 2025-02-25T13:36:21.316Z [container-agent] 2025-02-25 13:36:21 WARNING install Getting CA Private Key 2025-02-25T13:36:21.388Z [container-agent] 2025-02-25 13:36:21 INFO juju-log Rendering manifests 2025-02-25T13:36:22.928Z [pebble] Check “readiness” failure 10 (threshold 3): received non-20x status code 418 2025-02-25T13:36:24.345Z [container-agent] 2025-02-25 13:36:24 INFO juju-log HTTP Request: PATCH https:// 10.152.183.1 /apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterservingruntimes.serving.kserve.io?fieldManager=lightkube “HTTP/1.1 409 Conflict” 2025-02-25T13:36:24.395Z [container-agent] 2025-02-25 13:36:24 ERROR juju-log Apply failed with 2 conflicts: conflicts with “kubectl-client-side-apply” using apiextensions.k8s.io/v1: 2025-02-25T13:36:24.395Z [container-agent] - .metadata.annotations.controller-gen.kubebuilder.io/version 2025-02-25T13:36:24.395Z [container-agent] - .spec.versions 2025-02-25T13:36:24.407Z [container-agent] 2025-02-25 13:36:24 ERROR juju-log Uncaught exception while in charm code: 2025-02-25T13:36:24.407Z [container-agent] Traceback (most recent call last): 2025-02-25T13:36:24.407Z [container-agent] File “/var/lib/juju/agents/unit-kserve-controller-0/charm/venv/lightkube/core/generic_client.py”, line 188, in raise_for_status 2025-02-25T13:36:24.407Z [container-agent] resp.raise_for_status() 2025-02-25T13:36:24.407Z [container-agent] File “/var/lib/juju/agents/unit-kserve-controller-0/charm/venv/httpx/_models.py”, line 749, in raise_for_status 2025-02-25T13:36:24.407Z [container-agent] raise HTTPStatusError(message, request=request, response=self) 2025-02-25T13:36:24.407Z [container-agent] httpx.HTTPStatusError: Client error ‘409 Conflict’ for url ‘[https:// 10.152.183.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterservingruntimes.serving.kserve.io?fieldManager=lightkube’]( https:// 10.152.183.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterservingruntimes.serving.kserve.io?fieldManager=lightkube%27) 2025-02-25T13:36:24.407Z [container-agent] For more information check: https:// httpstatuses,com /409 2025-02-25T13:36:24.407Z [container-agent] 2025-02-25T13:36:24.407Z [container-agent] During handling of the above exception, another exception occurred: 2025-02-25T13:36:24.407Z [container-agent] 2025-02-25T13:36:24.407Z [container-agent] Traceback (most recent call last): 2025-02-25T13:36:24.407Z [container-agent] File “./src/charm.py”, line 553, in 2025-02-25T13:36:24.407Z [container-agent] main(KServeControllerCharm) 2025-02-25T13:36:24.407Z [container-agent] File “/var/lib/juju/agents/unit-kserve-controller-0/charm/venv/ops/main.py”, line 439, in main 2025-02-25T13:36:24.407Z [container-agent] framework.reemit() 2025-02-25T13:36:24.407Z [container-agent] File “/var/lib/juju/agents/unit-kserve-controller-0/charm/venv/ops/framework.py”, line 843, in reemit 2025-02-25T13:36:24.407Z [container-agent] self._reemit() 2025-02-25T13:36:24.407Z [container-agent] File “/var/lib/juju/agents/unit-kserve-controller-0/charm/venv/ops/framework.py”, line 922, in _reemit 2025-02-25T13:36:24.407Z [container-agent] custom_handler(event) 2025-02-25T13:36:24.407Z [container-agent] File “./src/charm.py”, line 340, in _on_install 2025-02-25T13:36:24.407Z [container-agent] self.k8s_resource_handler.apply() 2025-02-25T13:36:24.407Z [container-agent] File “/var/lib/juju/agents/unit-kserve-controller-0/charm/venv/charmed_kubeflow_chisme/kubernetes/_kubernetes_resource_handler.py”, line 348, in apply 2025-02-25T13:36:24.407Z [container-agent] raise e 2025-02-25T13:36:24.407Z [container-agent] File “/var/lib/juju/agents/unit-kserve-controller-0/charm/venv/charmed_kubeflow_chisme/kubernetes/_kubernetes_resource_handler.py”, line 333, in apply 2025-02-25T13:36:24.407Z [container-agent] apply_many(client=self.lightkube_client, objs=resources, force=force) 2025-02-25T13:36:24.407Z [container-agent] File “/var/lib/juju/agents/unit-kserve-controller-0/charm/venv/charmed_kubeflow_chisme/lightkube/batch/_many.py”, line 64, in apply_many 2025-02-25T13:36:24.407Z [container-agent] returns[i] = client.apply( 2025-02-25T13:36:24.407Z [container-agent] File “/var/lib/juju/agents/unit-kserve-controller-0/charm/venv/lightkube/core/client.py”, line 457, in apply 2025-02-25T13:36:24.407Z [container-agent] return self.patch(type(obj), name, obj, namespace=namespace, 2025-02-25T13:36:24.407Z [container-agent] File “/var/lib/juju/agents/unit-kserve-controller-0/charm/venv/lightkube/core/client.py”, line 325, in patch 2025-02-25T13:36:24.407Z [container-agent] return self._client.request(“patch”, res=res, name=name, namespace=namespace, obj=obj, 2025-02-25T13:36:24.407Z [container-agent] File “/var/lib/juju/agents/unit-kserve-controller-0/charm/venv/lightkube/core/generic_client.py”, line 245, in request 2025-02-25T13:36:24.407Z [container-agent] return self.handle_response(method, resp, br) 2025-02-25T13:36:24.407Z [container-agent] File “/var/lib/juju/agents/unit-kserve-controller-0/charm/venv/lightkube/core/generic_client.py”, line 196, in handle_response 2025-02-25T13:36:24.407Z [container-agent] self.raise_for_status(resp) 2025-02-25T13:36:24.407Z [container-agent] File “/var/lib/juju/agents/unit-kserve-controller-0/charm/venv/lightkube/core/generic_client.py”, line 190, in raise_for_status 2025-02-25T13:36:24.407Z [container-agent] raise transform_exception(e) 2025-02-25T13:36:24.407Z [container-agent] lightkube.core.exceptions.ApiError: Apply failed with 2 conflicts: conflicts with “kubectl-client-side-apply” using apiextensions.k8s.io/v1: 2025-02-25T13:36:24.407Z [container-agent] - .metadata.annotations.controller-gen.kubebuilder.io/version 2025-02-25T13:36:24.407Z [container-agent] - .spec.versions 2025-02-25T13:36:24.682Z [container-agent] 2025-02-25 13:36:24 ERROR juju.worker.uniter.operation runhook.go:153 hook “install” (via hook dispatching script: dispatch) failed: exit status 1 2025-02-25T13:36:24.683Z [container-agent] 2025-02-25 13:36:24 INFO juju.worker.uniter resolver.go:155 awaiting error resolution for “install” hook 2025-02-25T13:36:32.928Z [pebble] Check “readiness” failure 11 (threshold 3): received non-20x status code 418 2025-02-25T13:36:42.925Z [pebble] Check “readiness” failure 12 (threshold 3): received non-20x status code 418 2025-02-25T13:36:52.927Z [pebble] Check “readiness” failure 13 (threshold 3): received non-20x status code 418 2025-02-25T13:37:02.928Z [pebble] Check “readiness” failure 14 (threshold 3): received non-20x status code 418

I do not have previous experience with both juju and kubeflow. What may cause this issue?

Thank you in advance

2

Hi @geopape, thanks for the report!

Ideally issue reports are done in our repositories, which are monitored constantly. You can find our issue trackers:

My understanding from the post is that these are two different issues, so I’ll treat them separately.

  1. For the issue with Pipelines, Experiments, and Pipeline Runs - it looks like the database for kfp and katib (kfp-db and katib-db respectively) were restarted. When this happens, they come back up as read-only and subsequentially they recover on the next update-status event and should happen automatically within minutes of the restart. This can explain why you are getting the ‘Error 1290: The MySQL server is running with the --read-only option so it cannot execute this statement’ message.

In order to understand the issue better, though, we need to understand the status of the model. Could you please provide the output of the following:

juju status
juju debug-log -i kfp-db --replay

Also, please help us understand what version of Charmed Kubeflow you are running and how did you deploy it (please include as many details as possible).

  1. Regarding kserve-controller being in an error state - the error message Request: PATCH https:// 10.152.183.1 /apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterservingruntimes.serving.kserve.io?fieldManager=lightkube “HTTP/1.1 409 Conflict” 2025-02-25T13:36:24.395Z [container-agent] 2025-02-25 13:36:24 ERROR juju-log Apply failed with 2 conflicts: conflicts with “kubectl-client-side-apply” using apiextensions.k8s.io/v1: 2025-02-25T13:36:24.395Z suggests the cluster was not clean before deploying Charmed Kubeflow. Our charms would potentially fail if you are re-deploying in a cluster where Charmed Kubeflow was previously deployed and not cleaned up thoroughly (e.g. leaving behind CRDs, Deployments, or other resources). Is that your case?

To understand this other issue better, please also send us the following:

juju status kserve-controller
kubectl get logs -nkubeflow kserve-controller-0 -c kserve-controller
3

I believe that @geopape has also opened a github issue here: MySQL error in Pipeline Creation · Issue #1218 · canonical/bundle-kubeflow · GitHub

We can probably continue the discussion there. Please @geopape, provide in the github issue more information following @dnplas instructions

4

Hello @deusebio this issue opened from my colleague. I will continue the discussion there.

1 Like