Use Fibre Channel storage in ceph-osd for Openstack

charm Ceph
,
1

Hi,

Let’s say I have 3 machines that will be used for ceph cluster.

Each machine has 1 FC storage device (LUN Storage) with multipath configured. For example:

# multipath.conf

defaults {
    polling_interval            5
    no_path_retry               4
    user_friendly_names         no
    flush_on_last_del           yes
    fast_io_fail_tmo            5
    dev_loss_tmo                30
    max_fds                     4096
}

blacklist {
        protocol "(scsi:adt|scsi:sbp)"
}

overrides {
      no_path_retry            4
}

devices {
      device {
               path_grouping_policy    "multibus"
               path_checker            "tur"
               prio                    "const"
               path_selector           "round-robin 0"
               failback                "immediate"
               features                "0"
               no_path_retry           "fail"
           }
}

# multipath -ll

3xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx dm-1
size=50G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 15:0:0:2 sdc  8:32   active ready running
  |- 17:0:0:2 sds  65:32  active ready running
  |- 15:0:1:2 sdg  8:96   active ready running
  |- 17:0:1:2 sdw  65:96  active ready running
  |- 15:0:2:2 sdk  8:160  active ready running
  |- 17:0:2:2 sdaa 65:160 active ready running
  |- 15:0:3:2 sdo  8:224  active ready running
  `- 17:0:3:2 sdae 65:224 active ready running

# ls -l /dev/disk/by-id/* | grep 3xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

lrwxrwxrwx. 1 root root /dev/disk/by-id/dm-name-3xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -> ../../dm-1
lrwxrwxrwx. 1 root root /dev/disk/by-id/dm-uuid-mpath-3xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -> ../../dm-1
lrwxrwxrwx. 1 root root /dev/disk/by-id/scsi-3xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -> ../../dm-1

Refer to ceph-osd charm , The list of all possible storage devices for the cluster is defined by the osd-devices option. For example:

ceph-osd:
  osd-devices: /dev/vdb /dev/vdc /dev/vdd

Since FC storage is more unique and commonly uses “by-ID” for better implementation,

  1. How to define the FC/LUN storage path on the osd-devices option ?
  2. How to define the osd-devices for a specific machine ? like: Machine-A is /dev/disk/by-id/3AAA... , Machine-B is /dev/disk/by-id/3BBB..., and so on.
  3. Is there any best practice ?

Thanks.