Ceph-osd : No block devices detected using current configuration

juju
,
#1

So, this is probably gonna be a stupid question, please bear with me haha…

I’m installing Openstack using juju and MAAS(from this guide https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/install-openstack.html) and I’m currently facing an error I’m unsure how to go about solving. I’ve installed ceph-osd and I’m getting the error in the title. My osd-devices in ceph-osd.yaml file is /dev/sdb/

I sshd into the machines and passed lsblk… this was the output : https://imgur.com/a/N5rHXBC

As you can see there’s no /dev/sdb/…just a /dev/sda and it’s being used by my MAAS nodes root drive so it’s not available to ceph.

How would I go about solving this? Do I use fdisk and partition sda? Or do I need a new drive completely or something…how do I get /dev/sdb for ceph/ can i get it done in dev/sda?

And would I have to delete my entire model and start again for this?

#2

Welcome Rohan. It is recommended that each OSD daemon has a dedicated drive. So add a completely new disk to each corresponding MAAS node.

#5

Hello, So I used WD mycloud and added a /dev/sdb to my nodes

https://imgur.com/a/fhGQbWW

I then passed the command

juju config ceph-osd osd-devices=’/dev/sda /dev/sdb’ (since I changed it earlier)

And then

juju resolve --no-retry ceph-osd/0-3

But nothing seems to have changed

https://imgur.com/a/gjs8Myi

Any suggestions on what I can do to fix this?

#6

Just to be sure, can you give the output to:

juju config ceph-osd osd-devices
#7

The output is:

/dev/sda /dev/sdb

I changed the command to

juju config ceph-osd osd-devices=’/dev/sdb’

(The output of juju config ceph-osd osd-devices was simply /dev/sdb)

And did juju resolve but no change.

I also tried (just for the sake of it)

juju config ceph-osd osd-devices=’/dev/sdb1’

Output for what you asked here was /dev/sdb1

As well as

juju config ceph-osd osd-devices=’/dev/sdb1 /dev/sdb2 /dev/sdb3 /dev/sdb4’

To which the output was /dev/sdb1 /dev/sdb2 /dev/sdb3 /dev/sdb4

And performed juju resolve command.

But the result for all of this was the same. Status doesn’t change to active at all.

My ceph-osd.yaml file says osd-devices= /dev/sdb throughout all this by the way.

#8

If you look in the ceph-osd unit logs I bet you will find the reason why the devices are being rejected. You may have to use the zap-disk action, but look in the logs first:

juju debug-log --replay --no-tail -i ceph-osd/1
#9

Hey, so this is basically the output of the command…not sure what it means

also tried

juju run-action --wait ceph-osd/0-3 zap-disk i-really-mean-it=true devices=/dev/sdb

didnt change anything

Also, i removed the extra partitions and just have /dev/sdb1 now

And while fdisk -l and lsblk shows /dev/sdb and /dev/sdb1

df -h does not. Do i need to mount a file system onto it? Could that be the issue?

#10

So, generally, more information is better, juju config ceph-osd and juju config ceph-mon would be quite helpful in debugging issues.

That said, the disks you’re passing to ceph-osd shouldn’t have any partitions on them.

#11

output for juju config ceph-mon

output for juju config ceph-osd

So basically only have /dev/sdb? I have removed all partitions now and tried to resolve it but no change.

#12

Looking through the error log a bit, it looks like the keys don’t have necessary permissions:

RADOS permission denied (error connecting to the cluster)

Could you run:

juju ssh ceph-mon/0 'sudo ceph auth list | grep -v key:'`

That will let us see the key names and permissions

#13

juju ssh ceph-osd/0 ‘sudo ceph auth list | grep -v key:’

(changed it to osd not mon…mon says ceph-mon0 not found, its status is still waiting since ceph-osd hasnt reached the expected osd count)

output: ceph auth list - Pastebin.com

#14

The status for the mon can be waiting, that’s fine; the OSD doesn’t have access to the full auth list, which is why I asked for it on the mon, if ceph-mon/0 isn’t around, can you run the command targeting a different ceph-mon?

#15

ah yes my bad sorry.

juju ssh ceph-mon/11 ‘sudo ceph auth list | grep -v key:’

#16

It looks like it should have the permissions; another thing I’m noticing is that it’s trying to do an upgrade. Did you try installing a different version (Octopus / focal’s distro) and then upgraded to Pacific / focal-xena?

#17

umm…i dont think so(im a bit new to all this)…I basically just followed the openstack documentation word for word.

https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/install-openstack.html

But I ended up not having a block device for ceph-osd so I continued on with the documentation and after everything I used a WD MyCloud device to add a block device to my nodes.

Edit: Not sure if it means anything but when I pass fdisk -l the block device /dev/sdb has the disk label “dos”

#18

So, I think there’s an issue with your deployment that has nothing to do with the OSD config. The OSD charm will happily ignore bad devices so the fact that you’re getting hook errors is a bad thing, and the trace suggests that it’s related to an attempt to upgrade. To summarize, all of the following options should work fine, and if the named thing exists, the charm will use it as an OSD:

osd-devices='/dev/vdb /dev/sda /dev/sdb3 /dev/not-real'

Can you provide the output of juju status ceph-mon and juju status ceph-osd, and confirm which commit you’re using openstack-base from? Also, it would be great if you can confirm that the bundle I just linked is the one you’re using.

#19

ceph-osd status:

ceph-mon status:

I installed Openstack via individual charm and as per the documentation it says it uses Openstack Xena(cloud:focal-xena). The link you said is the exact same one in the documentation for installing Openstack via charm bundle so it should be the same(except that for the charm bundle the expected osd count is 3 whereas for individual charm installation its 4)

Also under the yaml files of ceph-osd and ceph-mon the source is cloud:focal-xena. so I’m sure i didn’t use a different version before

#20

I’m about to drop offline for the day, but I wanted to confirm that you’re using a bundle to deploy everything? The charm-deployment-guide page linked to in your first post documents two distinct ways to approach deploying OpenStack via the Charms:

  1. With individual charms, where you’d run things like juju deploy -n 4 --config ceph-osd.yaml --constraints tags=compute ceph-osd
  2. Via a bundle, where you’d run something like: juju deploy ./bundle.yaml
#21

Nope! I’m installing it via individual charm not through the charm bundle. so I ran things like option 1.

I’m doing the individual charm method as I’m trying to learn how it all works and well this is a proof of concept project im working on. So yea I was hoping to get this working without using the charm bundle. Had to reinstall everything multiple times…im guessing i might have to do it again but yea was hoping I could solve this without having to start from scratch.

Thanks for your time, appreciate it. If you have any other suggestions do let me know whenever you’re free next haha.

#22

Hi, just wanted to ask if there’s anything else I could try before deleting the openastack model and starting again?