I’m trying to automate some Ansible to do the underlying provisioning and using a preseed/manifest file to do the bootstrap & configure.
I get nothing but OpenStack timeout issues.
I’m utilizing either:
- Jammy-Current
- or a Jammy Release from August last year as the host OS
I have two nics:
- one that gets assigned a static IP
- the other is up, but doesn’t get assigned anything
preseed/manifest
bootstrap:
management_cidr: 172.19.A.B/21
addons:
metallb: 172.19.A.B-172.19.A.B
user:
run_demo_setup: True
username: demo
password: testtesttest
cidr: 192.168.122.0/24
nameservers: 8.8.8.8
security_group_rules: True
remote_access_location: local
external_network:
cidr: 172.19.A.B/21
gateway: 172.19.A.B
start: 172.19.A.B
end: 172.19.A.B
network_type: flat
segmentation_id: 0
nic: enp2s0
physical_network: physnet1
microceph_config:
opnstk-server-vm:
osd_devices: /dev/vdc /dev/vdd /dev/vde
I’ve been iterating on this for something around 100+ times via just firing off a Jenkins pipeline.
The Ansible is:
- name: Add groups to ubuntu user
become: true
ansible.builtin.user:
name: ubuntu
groups: ubuntu, adm, avahi, audio, snap_microk8s, cdrom, dialout, dip, floppy, kvm, lxd, netdev, plugdev, sudo, video, snap_daemon
- name: Add groups to stack user
become: true
ansible.builtin.user:
name: stack
groups: ubuntu, adm, avahi, audio, snap_microk8s, cdrom, dialout, dip, floppy, kvm, lxd, netdev, plugdev, sudo, video, snap_daemon
- name: increase inotify watchers
become_user: ubuntu
ansible.builtin.shell: |
echo "fs.inotify.max_user_instances = 1024" | sudo tee /etc/sysctl.d/80-sunbeam.conf && sudo sysctl -q -p /etc/sysctl.d/80-sunbeam.conf
- name: connect snap to ssh keys
become_user: ubuntu
ansible.builtin.shell: |
sudo /usr/bin/snap connect openstack:ssh-keys
- name: generate keypair
become_user: ubuntu
ansible.builtin.shell: |
ssh-keygen -b 4096 -f /home/ubuntu/.ssh/id_rsa -t rsa -N ""
- name: make auth key
become_user: ubuntu
ansible.builtin.shell: |
cat /home/ubuntu/.ssh/id_rsa.pub >> /home/ubuntu/.ssh/authorized_keys
- name: keyscan
become_user: ubuntu
ansible.builtin.shell: |
ssh-keyscan -H $(hostname --all-ip-addresses) >> /home/ubuntu/.ssh/known_hosts
- name: mkdirs
become_user: ubuntu
ansible.builtin.shell: |
mkdir -p /home/ubuntu/.local/share && mkdir -p /home/ubuntu/.config/openstack
- name: Template generate-preseed.yaml.j2
become_user: ubuntu
ansible.builtin.template:
src: generated-preseed.yaml.j2
dest: /home/ubuntu/generated-preseed.yaml
- name: bootstrap microceph
become_user: ubuntu
ansible.builtin.shell: |
sudo /snap/bin/microceph cluster bootstrap --verbose
- name: flush second nic
ansible.builtin.shell: |
ip addr flush dev enp2s0
- name: make sure microk8s is ready
become_user: ubuntu
ansible.builtin.shell: |
sudo /snap/bin/microk8s status --wait-ready
- name: bootstrap openstack
become_user: ubuntu
ansible.builtin.shell: |
echo "n" | /snap/bin/sunbeam cluster bootstrap --manifest /home/ubuntu/generated-preseed.yaml
- name: configure openstack
become_user: ubuntu
ansible.builtin.shell: |
/snap/bin/sunbeam configure --manifest /home/ubuntu/generated-preseed.yaml --openrc demo-openrc
My cloud-config / cloud-init user-data takes the shape of:
#cloud-config
final_message: "Canonical Sunbeam: Openstack Cluster Initialization of Packages finished in $UPTIME"
ssh_pwauth: True
bootcmd:
- /usr/sbin/ip link set enp1s0 up
- /usr/sbin/ip link set enp2s0 up
package_update: true
users:
- default
- name: root
lock_passwd: false
passwd: HASHED-PASSWORD
- name: stack
sudo: ALL=(ALL) NOPASSWD:ALL
passwd: HASHED-PASSWORD
lock_passwd: false
shell: /bin/bash
homedir: /home/stack
snap:
commands:
00: ['snap', 'refresh', 'lxd', '--channel=latest/stable']
01: ['snap', 'install', 'juju', '--channel=3.2/stable']
02: ['snap', 'install', 'juju-db', '--channel=4.4/stable']
03: ['snap', 'install', 'microk8s', '--channel=1.26-strict/stable']
04: ['snap', 'install', 'microceph', '--channel=latest/stable']
05: ['snap', 'install', 'openstack', '--channel=2023.2/edge']
packages:
- qemu-guest-agent
- apt-transport-https
- neovim
- wget
- ca-certificates
- python3
- python3-pip
- jq
- curl
- gnupg-agent
- gnupg
- lsb-release
- htop
- parted
- fdisk
- gnupg2
- neovim
- software-properties-common
- coreutils
- sshpass
- tmux
- net-tools
- unzip
- debian-keyring
- debian-archive-keyring
- apt-transport-https
- build-essential
- libguestfs-tools
- avahi-daemon
- avahi-utils
- curl
- jq
- openssh-server
- openssl
- plocate
- sshpass
- ubuntu-advantage-tools
- unzip
- vim
- wget
- whois
timezone: 'America/Los_Angeles'
locale: 'en_US.UTF-8'
runcmd:
- - systemctl
- enable
- --now
- qemu-guest-agent.service
- systemctl stop apparmor
- systemctl disable apparmor
- ufw disable
Whereas, the cloud-config / cloud-init network data takes the shape of:
#cloud-config
network:
version: 1
config:
- type: nameserver
address:
- ${var.OPENSTACK_DNS_SERVER_TO_USE}
search:
- exemplary
- type: physical
name: enp1s0
subnets:
- type: static
address: 172.19.A.B/21
gateway: 172.19.A.B
Juju status of openstack module is:
ubuntu@opnstk-server-vm:~$ juju status -m openstack
Model Controller Cloud/Region Version SLA Timestamp
openstack sunbeam-controller sunbeam-microk8s/localhost 3.2.4 unsupported 09:37:20-07:00
App Version Status Scale Charm Channel Rev Address Exposed Message
certificate-authority active 1 self-signed-certificates latest/beta 109 10.152.183.49 no
cinder waiting 1 cinder-k8s 2023.2/edge 63 10.152.183.45 no installing agent
cinder-ceph waiting 1 cinder-ceph-k8s 2023.2/edge 57 10.152.183.210 no installing agent
cinder-ceph-mysql-router 8.0.35-0ubuntu0.22.04.1 active 1 mysql-router-k8s 8.0/candidate 96 10.152.183.211 no
cinder-mysql 8.0.35-0ubuntu0.22.04.1 active 1 mysql-k8s 8.0/candidate 127 10.152.183.28 no
cinder-mysql-router 8.0.35-0ubuntu0.22.04.1 active 1 mysql-router-k8s 8.0/candidate 96 10.152.183.146 no
glance waiting 1 glance-k8s 2023.2/edge 78 10.152.183.31 no installing agent
glance-mysql 8.0.35-0ubuntu0.22.04.1 active 1 mysql-k8s 8.0/candidate 127 10.152.183.142 no
glance-mysql-router 8.0.35-0ubuntu0.22.04.1 active 1 mysql-router-k8s 8.0/candidate 96 10.152.183.111 no
horizon waiting 1 horizon-k8s 2023.2/edge 75 10.152.183.138 no installing agent
horizon-mysql waiting 1 mysql-k8s 8.0/candidate 127 10.152.183.117 no installing agent
horizon-mysql-router 8.0.35-0ubuntu0.22.04.1 waiting 1 mysql-router-k8s 8.0/candidate 96 10.152.183.42 no installing agent
keystone waiting 1 keystone-k8s 2023.2/edge 169 10.152.183.205 no installing agent
keystone-mysql 8.0.35-0ubuntu0.22.04.1 active 1 mysql-k8s 8.0/candidate 127 10.152.183.79 no
keystone-mysql-router 8.0.35-0ubuntu0.22.04.1 active 1 mysql-router-k8s 8.0/candidate 96 10.152.183.95 no
neutron waiting 1 neutron-k8s 2023.2/edge 70 10.152.183.109 no installing agent
neutron-mysql 8.0.35-0ubuntu0.22.04.1 active 1 mysql-k8s 8.0/candidate 127 10.152.183.212 no
neutron-mysql-router 8.0.35-0ubuntu0.22.04.1 active 1 mysql-router-k8s 8.0/candidate 96 10.152.183.53 no
nova waiting 1 nova-k8s 2023.2/edge 66 10.152.183.162 no installing agent
nova-api-mysql-router 8.0.35-0ubuntu0.22.04.1 active 1 mysql-router-k8s 8.0/candidate 96 10.152.183.104 no
nova-cell-mysql-router 8.0.35-0ubuntu0.22.04.1 active 1 mysql-router-k8s 8.0/candidate 96 10.152.183.75 no
nova-mysql 8.0.35-0ubuntu0.22.04.1 active 1 mysql-k8s 8.0/candidate 127 10.152.183.85 no
nova-mysql-router 8.0.35-0ubuntu0.22.04.1 active 1 mysql-router-k8s 8.0/candidate 96 10.152.183.40 no
ovn-central active 1 ovn-central-k8s 23.09/edge 75 10.152.183.59 no
ovn-relay active 1 ovn-relay-k8s 23.09/edge 62 10.20.21.10 no
placement waiting 1 placement-k8s 2023.2/edge 58 10.152.183.127 no installing agent
placement-mysql 8.0.35-0ubuntu0.22.04.1 waiting 1 mysql-k8s 8.0/candidate 127 10.152.183.36 no installing agent
placement-mysql-router 8.0.35-0ubuntu0.22.04.1 waiting 1 mysql-router-k8s 8.0/candidate 96 10.152.183.86 no installing agent
rabbitmq waiting 1 rabbitmq-k8s 3.12/edge 34 10.20.21.12 no installing agent
traefik 2.10.4 active 1 traefik-k8s 1.0/edge 164 10.20.21.11 no
traefik-public 2.10.4 active 1 traefik-k8s 1.0/edge 164 10.20.21.13 no
Unit Workload Agent Address Ports Message
certificate-authority/0* active idle 10.1.105.135
cinder-ceph-mysql-router/0* active idle 10.1.105.142
cinder-ceph/0* blocked idle 10.1.105.146 (ceph) integration missing
cinder-mysql-router/0* active idle 10.1.105.154
cinder-mysql/0* active idle 10.1.105.138 Primary
cinder/0* waiting idle 10.1.105.161 (workload) Not all relations are ready
glance-mysql-router/0* active idle 10.1.105.145
glance-mysql/0* active idle 10.1.105.149 Primary
glance/0* waiting idle 10.1.105.171 (amqp) integration incomplete
horizon-mysql-router/0* waiting idle 10.1.105.157
horizon-mysql/0* unknown idle 10.1.105.174
horizon/0* waiting idle 10.1.105.168 (workload) Not all relations are ready
keystone-mysql-router/0* active idle 10.1.105.143
keystone-mysql/0* active idle 10.1.105.178 Primary
keystone/0* waiting idle 10.1.105.175 (amqp) integration incomplete
neutron-mysql-router/0* active idle 10.1.105.150
neutron-mysql/0* active idle 10.1.105.179 Primary
neutron/0* waiting idle 10.1.105.159 (workload) Not all relations are ready
nova-api-mysql-router/0* active idle 10.1.105.147
nova-cell-mysql-router/0* active idle 10.1.105.151
nova-mysql-router/0* active idle 10.1.105.152
nova-mysql/0* active idle 10.1.105.180 Primary
nova/0* waiting idle 10.1.105.162 (workload) Not all relations are ready
ovn-central/0* active idle 10.1.105.183
ovn-relay/0* active idle 10.1.105.160
placement-mysql-router/0* waiting idle 10.1.105.158
placement-mysql/0* unknown idle 10.1.105.153
placement/0* waiting idle 10.1.105.163 (workload) Not all relations are ready
rabbitmq/0* maintenance executing 10.1.105.181 installing charm software
traefik-public/0* active idle 10.1.105.182
traefik/0* active idle 10.1.105.140
Offer Application Charm Rev Connected Endpoint Interface Role
cert-distributor keystone keystone-k8s 169 0/0 send-ca-cert certificate_transfer provider
certificate-authority certificate-authority self-signed-certificates 109 0/0 certificates tls-certificates provider
cinder-ceph cinder-ceph cinder-ceph-k8s 57 0/0 ceph-access cinder-ceph-key provider
keystone keystone keystone-k8s 169 0/0 identity-credentials keystone-credentials provider
ovn-relay ovn-relay ovn-relay-k8s 62 0/0 ovsdb-cms-relay ovsdb-cms provider
rabbitmq rabbitmq rabbitmq-k8s 34 0/0 amqp rabbitmq provider
Neutron does spit back some warnings with:
ubuntu@opnstk-server-vm:~$ microk8s.kubectl logs svc/neutron -n openstack | grep -ie warning
Defaulted container "charm" out of: charm, neutron-server, charm-init (init)
2024-04-10T23:26:52.775Z [container-agent] 2024-04-10 23:26:52 WARNING juju.worker.proxyupdater proxyupdater.go:241 unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
2024-04-10T23:42:54.758Z [container-agent] 2024-04-10 23:42:54 WARNING juju-log Charm is waiting in section 'Bootstrapping' due to 'Not all relations are ready'
2024-04-10T23:43:04.374Z [container-agent] 2024-04-10 23:43:04 WARNING juju-log Charm is waiting in section 'Bootstrapping' due to 'Not all relations are ready'
2024-04-10T23:43:21.389Z [container-agent] 2024-04-10 23:43:21 WARNING juju-log certificates:67: Charm is waiting in section 'Bootstrapping' due to 'Not all relations are ready'
2024-04-10T23:43:41.049Z [container-agent] 2024-04-10 23:43:41 WARNING juju-log ingress-internal:77: 'app' expected but not received.
2024-04-10T23:43:41.055Z [container-agent] 2024-04-10 23:43:41 WARNING juju-log ingress-internal:77: 'app_name' expected in snapshot but not found.
2024-04-10T23:43:42.248Z [container-agent] 2024-04-10 23:43:42 WARNING juju-log ingress-internal:77: Charm is waiting in section 'Bootstrapping' due to 'Not all relations are ready'
2024-04-10T23:44:20.967Z [container-agent] 2024-04-10 23:44:20 WARNING juju-log ovsdb-cms:99: Charm is waiting in section 'Bootstrapping' due to 'Not all relations are ready'
2024-04-10T23:44:34.540Z [container-agent] 2024-04-10 23:44:34 WARNING juju-log certificates:67: Charm is waiting in section 'Bootstrapping' due to 'Not all relations are ready'
2024-04-10T23:45:01.946Z [container-agent] 2024-04-10 23:45:01 WARNING update-status /var/lib/juju/agents/unit-neutron-0/charm/lib/charms/tls_certificates_interface/v1/tls_certificates.py:1251: CryptographyDeprecationWarning: Properties that return a naïve datetime object have been deprecated. Please switch to not_valid_after_utc.
2024-04-10T23:45:01.946Z [container-agent] 2024-04-10 23:45:01 WARNING update-status time_difference = certificate_object.not_valid_after - datetime.utcnow()
2024-04-10T23:46:45.373Z [container-agent] 2024-04-10 23:46:45 WARNING juju-log ingress-public:82: 'app' expected but not received.
2024-04-10T23:46:45.381Z [container-agent] 2024-04-10 23:46:45 WARNING juju-log ingress-public:82: 'app_name' expected in snapshot but not found.
2024-04-10T23:46:46.317Z [container-agent] 2024-04-10 23:46:46 WARNING juju-log ingress-public:82: Charm is waiting in section 'Bootstrapping' due to 'Not all relations are ready'
2024-04-10T23:49:08.636Z [container-agent] 2024-04-10 23:49:08 WARNING update-status /var/lib/juju/agents/unit-neutron-0/charm/lib/charms/tls_certificates_interface/v1/tls_certificates.py:1251: CryptographyDeprecationWarning: Properties that return a naïve datetime object have been deprecated. Please switch to not_valid_after_utc.
2024-04-10T23:49:08.636Z [container-agent] 2024-04-10 23:49:08 WARNING update-status time_difference = certificate_object.not_valid_after - datetime.utcnow()
2024-04-10T23:49:48.708Z [container-agent] 2024-04-10 23:49:48 WARNING juju-log identity-service:100: Charm is waiting in section 'Bootstrapping' due to 'Not all relations are ready'
2024-04-10T23:50:02.800Z [container-agent] 2024-04-10 23:50:02 WARNING juju-log receive-ca-cert:105: Provider relation data did not pass JSON Schema validation: {'egress-subnets': '10.152.183.205/32', 'ingress-address': '10.152.183.205', 'private-address': '10.152.183.205'}
2024-04-10T23:53:11.879Z [container-agent] 2024-04-10 23:53:11 WARNING juju-log receive-ca-cert:105: Charm is waiting in section 'Bootstrapping' due to 'Not all relations are ready'
2024-04-10T23:53:55.343Z [container-agent] 2024-04-10 23:53:55 WARNING update-status /var/lib/juju/agents/unit-neutron-0/charm/lib/charms/tls_certificates_interface/v1/tls_certificates.py:1251: CryptographyDeprecationWarning: Properties that return a naïve datetime object have been deprecated. Please switch to not_valid_after_utc.
Additionally I guess rabbitmq is seeing some issues with 1/2 pods not ready:
arm 2024-04-11T16:43:38.077Z [pebble] Check "readiness" failure 6188 (threshold 3): received non-20x status code 418
charm 2024-04-11T16:43:38.112Z [container-agent] 2024-04-11 16:43:38 INFO juju-log Waiting for RabbitMQ to start
charm 2024-04-11T16:43:48.080Z [pebble] Check "readiness" failure 6189 (threshold 3): received non-20x status code 418
charm 2024-04-11T16:43:48.137Z [container-agent] 2024-04-11 16:43:48 INFO juju-log Waiting for RabbitMQ to start
charm 2024-04-11T16:43:58.080Z [pebble] Check "readiness" failure 6190 (threshold 3): received non-20x status code 418
charm 2024-04-11T16:43:58.160Z [container-agent] 2024-04-11 16:43:58 INFO juju-log Waiting for RabbitMQ to start
charm 2024-04-11T16:44:08.078Z [pebble] Check "readiness" failure 6191 (threshold 3): received non-20x status code 418
charm 2024-04-11T16:44:08.188Z [container-agent] 2024-04-11 16:44:08 INFO juju-log Waiting for RabbitMQ to start
charm 2024-04-11T16:44:18.080Z [pebble] Check "readiness" failure 6192 (threshold 3): received non-20x status code 418
charm 2024-04-11T16:44:18.209Z [container-agent] 2024-04-11 16:44:18 INFO juju-log Waiting for RabbitMQ to start
charm-init starting containeragent init command
Stream closed EOF for openstack/rabbitmq-0 (charm-init)
charm 2024-04-11T16:44:28.077Z [pebble] Check "readiness" failure 6193 (threshold 3): received non-20x status code 418
charm 2024-04-11T16:44:28.233Z [container-agent] 2024-04-11 16:44:28 INFO juju-log Waiting for RabbitMQ to start
charm 2024-04-11T16:44:38.080Z [pebble] Check "readiness" failure 6194 (threshold 3): received non-20x status code 418
charm 2024-04-11T16:44:38.261Z [container-agent] 2024-04-11 16:44:38 INFO juju-log Waiting for RabbitMQ to start
Seemingly having an impact on things like nova-mysql:
nova-mysql-0 mysql 2024-04-11T16:41:31.963Z [pebble] POST /v1/files 45.067109ms 200
nova-mysql-0 mysql 2024-04-11T16:41:32.107Z [pebble] POST /v1/exec 143.218648ms 202
nova-mysql-0 charm 2024-04-11T16:08:03.098Z [container-agent] File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
nova-mysql-0 mysql 2024-04-11T16:41:32.260Z [pebble] GET /v1/tasks/80/websocket/control 151.577318ms 200
nova-mysql-0 mysql 2024-04-11T16:41:32.261Z [pebble] GET /v1/tasks/80/websocket/stdio 117.561µs 200
nova-mysql-0 mysql 2024-04-11T16:41:32.262Z [pebble] GET /v1/tasks/80/websocket/stderr 74.156µs 200
nova-mysql-0 mysql 2024-04-11T16:41:33.265Z [pebble] GET /v1/changes/80/wait?timeout=4.000s 1.001919126s 200
nova-mysql-0 charm 2024-04-11T16:08:03.098Z [container-agent] result = func(*args)
nova-mysql-0 mysql 2024-04-11T16:42:02.170Z [pebble] POST /v1/exec 573.289926ms 202
nova-mysql-0 charm 2024-04-11T16:08:03.098Z [container-agent] File "/var/lib/juju/agents/unit-nova-mysql-0/charm/venv/ops/pebble.py", line 326, in http_open
nova-mysql-0 charm 2024-04-11T16:08:03.098Z [container-agent] return self.do_open(_UnixSocketConnection, req, # type:ignore
nova-mysql-0 charm 2024-04-11T16:08:03.098Z [container-agent] File "/usr/lib/python3.10/urllib/request.py", line 1352, in do_open
nova-mysql-0 charm 2024-04-11T16:08:03.098Z [container-agent] r = h.getresponse()
nova-mysql-0 charm 2024-04-11T16:08:03.098Z [container-agent] File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
nova-mysql-0 charm 2024-04-11T16:08:03.098Z [container-agent] response.begin()
nova-mysql-0 mysql 2024-04-11T16:42:02.171Z [pebble] GET /v1/tasks/81/websocket/control 117.224µs 200
nova-mysql-0 charm 2024-04-11T16:08:03.098Z [container-agent] File "/usr/lib/python3.10/http/client.py", line 318, in begin
nova-mysql-0 charm 2024-04-11T16:08:03.098Z [container-agent] version, status, reason = self._read_status()
nova-mysql-0 charm 2024-04-11T16:08:03.098Z [container-agent] File "/usr/lib/python3.10/http/client.py", line 287, in _read_status
nova-mysql-0 mysql 2024-04-11T16:42:02.172Z [pebble] GET /v1/tasks/81/websocket/stdio 97.749µs 200
nova-mysql-0 mysql 2024-04-11T16:42:02.173Z [pebble] GET /v1/tasks/81/websocket/stderr 94.566µs 200
nova-mysql-0 mysql 2024-04-11T16:42:02.770Z [pebble] GET /v1/changes/81/wait?timeout=4.000s 595.68225ms 200
nova-mysql-0 mysql 2024-04-11T16:42:02.837Z [pebble] POST /v1/files 64.017966ms 200
nova-mysql-0 charm 2024-04-11T16:08:03.098Z [container-agent] raise RemoteDisconnected("Remote end closed connection without"
nova-mysql-0 charm 2024-04-11T16:08:03.098Z [container-agent] http.client.RemoteDisconnected: Remote end closed connection without response
nova-mysql-0 mysql 2024-04-11T16:42:03.147Z [pebble] POST /v1/exec 308.884932ms 202
nova-mysql-0 mysql 2024-04-11T16:42:03.360Z [pebble] GET /v1/tasks/82/websocket/control 212.185569ms 200
It just seems like either 2023.2/edge or 2023.2/stable I can’t get it to provision w/ a preseed/manifest file.
I’ll pivot back and run another iteration on 2023.2/stable. I wasn’t able to have any success on 2023.1/stable iirc.
Thanks for any insight / help into MicroStack Timeout Issues on 2023.2/stable & 2023.2/edge!