I am deploying my 3rd charmed Openstack rack in 4 months and ceph dashboard will not install now

I have used the same basic yml file to deploy openstack HA with ceph-dashboard. it has worked every time but now it is stuck saying install failed.

    2022-09-27 17:58:52 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "relation-changed" hook
2022-09-27 17:58:55 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "relation-changed" hook
2022-09-27 17:59:00 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "relation-changed" hook
2022-09-27 17:59:00 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "relation-changed" hook
2022-09-27 17:59:01 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "relation-changed" hook
2022-09-27 17:59:21 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "relation-changed" hook
2022-09-27 17:59:21 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "relation-changed" hook
2022-09-27 17:59:50 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "relation-changed" hook
2022-09-27 18:00:16 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "relation-changed" hook
2022-09-27 18:03:13 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "relation-changed" hook
2022-09-27 18:03:14 INFO unit.ceph-dashboard/0.juju-log server.go:316 dashboard:27: Requesting a CA certificate. Common name: juju-74d22a-2-lxd-0.dev, SANS: ['10.0.7.26', '10.0.7.65', 'juju-74d22a-2-lxd-0']
2022-09-27 18:03:15 ERROR unit.ceph-dashboard/0.juju-log server.go:316 dashboard:27: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 608, in <module>
    main(CephDashboardCharm)
  File "/var/lib/juju/agents/unit-ceph-dashboard-0/charm/venv/ops/main.py", line 406, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-ceph-dashboard-0/charm/venv/ops/main.py", line 140, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-ceph-dashboard-0/charm/venv/ops/framework.py", line 278, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-ceph-dashboard-0/charm/venv/ops/framework.py", line 722, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-ceph-dashboard-0/charm/venv/ops/framework.py", line 767, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-ceph-dashboard-0/charm/src/interface_dashboard.py", line 50, in on_changed
    self.on.mon_ready.emit()
  File "/var/lib/juju/agents/unit-ceph-dashboard-0/charm/venv/ops/framework.py", line 278, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-ceph-dashboard-0/charm/venv/ops/framework.py", line 722, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-ceph-dashboard-0/charm/venv/ops/framework.py", line 767, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 424, in _configure_dashboard
    if self.unit.is_leader() and not ceph_utils.is_dashboard_enabled():
  File "/var/lib/juju/agents/unit-ceph-dashboard-0/charm/venv/charms_ceph/utils.py", line 3440, in is_mgr_module_enabled
    return module in enabled_manager_modules()
  File "/var/lib/juju/agents/unit-ceph-dashboard-0/charm/venv/charms_ceph/utils.py", line 3428, in enabled_manager_modules
    modules = json.loads(modules)
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2022-09-27 18:03:15 ERROR juju.worker.uniter.operation runhook.go:140 hook "dashboard-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1
2022-09-27 18:03:15 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "relation-changed" hook
2022-09-27 18:03:31 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "relation-changed" hook
2022-09-27 18:03:31 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "relation-changed" hook
2022-09-27 18:04:32 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "relation-changed" hook
2022-09-27 18:04:32 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "relation-changed" hook
2022-09-27 18:04:33 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "relation-changed" hook
2022-09-27 18:04:33 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "relation-changed" hook
2022-09-27 18:04:34 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "relation-changed" hook
2022-09-27 18:04:34 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "relation-changed" hook
2022-09-27 18:04:34 INFO juju.worker.uniter uniter.go:310 unit "ceph-dashboard/0" shutting down: catacomb 0xc0001e5200 is dying
2022-09-27 18:04:34 INFO juju.worker.logger logger.go:136 logger worker stopped

I dont know if anything changed. i attempted the manual remove and re-install which has worked in the past but the same error is happening. Does anyone know what the problem could be. vault is open and there is an active cert. all other systems are online and working properly.

thanks for any help!

1 Like

I have exactly the same error on a very basic install…

022-10-20 02:41:45 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "install" hook
2022-10-20 02:41:46 INFO unit.ceph-dashboard/6.juju-log server.go:316 Running legacy hooks/install.
2022-10-20 02:41:46 INFO unit.ceph-dashboard/6.juju-log server.go:316 Installing packages
2022-10-20 02:41:48 INFO unit.ceph-dashboard/6.juju-log server.go:316 Installing ['ceph-mgr-dashboard'] with options: ['--option=Dpkg::Options::=--force-confold']
2022-10-20 02:41:48 INFO unit.ceph-dashboard/6.juju-log server.go:316 Updating status
2022-10-20 02:41:49 ERROR unit.ceph-dashboard/6.juju-log server.go:316 Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 608, in <module>
    main(CephDashboardCharm)
  File "/var/lib/juju/agents/unit-ceph-dashboard-6/charm/venv/ops/main.py", line 404, in main
    framework.reemit()
  File "/var/lib/juju/agents/unit-ceph-dashboard-6/charm/venv/ops/framework.py", line 732, in reemit
    self._reemit()
  File "/var/lib/juju/agents/unit-ceph-dashboard-6/charm/venv/ops/framework.py", line 767, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-ceph-dashboard-6/charm/venv/ops_openstack/core.py", line 105, in on_install
    self.install_pkgs()
  File "/var/lib/juju/agents/unit-ceph-dashboard-6/charm/venv/ops_openstack/core.py", line 102, in install_pkgs
    self.update_status()
  File "/var/lib/juju/agents/unit-ceph-dashboard-6/charm/venv/ops_openstack/core.py", line 144, in update_status
    _result = check()
  File "./src/charm.py", line 363, in check_dashboard
    if not check_f():
  File "/var/lib/juju/agents/unit-ceph-dashboard-6/charm/venv/charms_ceph/utils.py", line 3440, in is_mgr_module_enabled
    return module in enabled_manager_modules()
  File "/var/lib/juju/agents/unit-ceph-dashboard-6/charm/venv/charms_ceph/utils.py", line 3428, in enabled_manager_modules
    modules = json.loads(modules)
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2022-10-20 02:41:49 ERROR juju.worker.uniter.operation runhook.go:140 hook "install" (via hook dispatching script: dispatch) failed: exit status 1
2022-10-20 02:41:49 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "install" hook

juju status:

kutovoy@gigaby-minipc:~/tutorial$ juju status
Model  Controller      Cloud/Region    Version  SLA          Timestamp
ceph   kcloud-default  kcloud/default  2.9.35   unsupported  19:53:42-07:00

App                               Version  Status  Scale  Charm                   Channel       Rev  Exposed  Message
ceph-dashboard                             error       3  ceph-dashboard          stable          2  no       hook failed: "install"
ceph-mon                          17.2.0   active      3  ceph-mon                stable         73  no       Unit is ready and clustered
ceph-osd                          17.2.0   active      3  ceph-osd                stable        513  no       Unit is ready (1 OSD)
mysql-innodb-cluster              8.0.30   active      3  mysql-innodb-cluster    8.0/stable     30  no       Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
openstack-loadbalancer                     active      3  openstack-loadbalancer  jammy/stable   10  no       Unit is ready
openstack-loadbalancer-hacluster           active      3  hacluster               stable         83  no       Unit is ready and clustered
vault                             1.10.6   active      1  vault                   stable         54  no       Unit is ready (active: true, mlock: disabled)
vault-mysql-router                8.0.30   active      1  mysql-router            8.0/stable     35  no       Unit is ready

Unit                                   Workload  Agent  Machine  Public address  Ports     Message
ceph-mon/6                             active    idle   7/lxd/0  10.0.1.24                 Unit is ready and clustered
  ceph-dashboard/6                     error     idle            10.0.1.24                 hook failed: "install"
ceph-mon/7*                            active    idle   8/lxd/0  10.0.1.4                  Unit is ready and clustered
  ceph-dashboard/7*                    error     idle            10.0.1.4                  hook failed: "install"
ceph-mon/8                             active    idle   9/lxd/0  10.0.1.44                 Unit is ready and clustered
  ceph-dashboard/8                     error     idle            10.0.1.44                 hook failed: "install"
ceph-osd/6                             active    idle   7        10.0.1.252                Unit is ready (1 OSD)
ceph-osd/7*                            active    idle   8        10.0.1.18                 Unit is ready (1 OSD)
ceph-osd/8                             active    idle   9        10.0.1.17                 Unit is ready (1 OSD)
mysql-innodb-cluster/6                 active    idle   7/lxd/1  10.0.1.36                 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
mysql-innodb-cluster/7                 active    idle   8/lxd/1  10.0.1.9                  Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
mysql-innodb-cluster/8*                active    idle   9/lxd/1  10.0.1.35                 Unit is ready: Mode: R/W, Cluster is ONLINE and can tolerate up to ONE failure.
openstack-loadbalancer/3*              active    idle   7/lxd/3  10.0.1.102                Unit is ready
  openstack-loadbalancer-hacluster/0*  active    idle            10.0.1.102                Unit is ready and clustered
openstack-loadbalancer/4               active    idle   8/lxd/2  10.0.1.253                Unit is ready
  openstack-loadbalancer-hacluster/2   active    idle            10.0.1.253                Unit is ready and clustered
openstack-loadbalancer/5               active    idle   9/lxd/2  10.0.1.57                 Unit is ready
  openstack-loadbalancer-hacluster/1   active    idle            10.0.1.57                 Unit is ready and clustered
vault/2*                               active    idle   7/lxd/2  10.0.1.101      8200/tcp  Unit is ready (active: true, mlock: disabled)
  vault-mysql-router/1*                active    idle            10.0.1.101                Unit is ready

Machine  State    Address     Inst id              Series  AZ       Message
7        started  10.0.1.252  kcloud-3             focal   default  Deployed
7/lxd/0  started  10.0.1.24   juju-41872b-7-lxd-0  focal   default  Container started
7/lxd/1  started  10.0.1.36   juju-41872b-7-lxd-1  focal   default  Container started
7/lxd/2  started  10.0.1.101  juju-41872b-7-lxd-2  focal   default  Container started
7/lxd/3  started  10.0.1.102  juju-41872b-7-lxd-3  focal   default  Container started
8        started  10.0.1.18   kcloud-4             focal   default  Deployed
8/lxd/0  started  10.0.1.4    juju-41872b-8-lxd-0  focal   default  Container started
8/lxd/1  started  10.0.1.9    juju-41872b-8-lxd-1  focal   default  Container started
8/lxd/2  started  10.0.1.253  juju-41872b-8-lxd-2  focal   default  Container started
9        started  10.0.1.17   kcloud-5             focal   default  Deployed
9/lxd/0  started  10.0.1.44   juju-41872b-9-lxd-0  focal   default  Container started
9/lxd/1  started  10.0.1.35   juju-41872b-9-lxd-1  focal   default  Container started
9/lxd/2  started  10.0.1.57   juju-41872b-9-lxd-2  focal   default  Container started

OK - I found a solution: