Juju provisioner - failed to aquire node

1

Trying to deploy an openstack bundle that i wrote i am receiving the following error

controller-2: 07:23:52 WARNING juju.worker.provisioner failed to start machine 62 (failed to acquire node: unexpected: ServerError: 400 Bad Request ({"interfaces": ["Invalid label name: '-1' (Must begin with an alphanumeric character, and include only alphanumeric characters, dashes, and underscores.)"]})), retrying in 10s (3 more attempts)

Interfaces do also look ok.

I am selecting the machine using a tag and the name is ok. Is there any chance to increase the logging in order to understand what is wrong ?

###########

juju --version
2.7.6-focal-amd64

juju show-controller
xxxxxx:
details:
uuid: xxxxxxxx
controller-uuid: xxxxxxxxxx
cloud: maas
region: default
agent-version: 2.8.7
agent-git-commit: ee2cfeb2c8c716af763a011e184ddea879c0985d
controller-model-version: 2.8.7
mongo-version: 3.6.8

###########

2

Hi

Can you attach the bundle that is causing the problem (sanitized as needed). It’s probably just a typo, but it would be good to see it to confirm it.

Thanks

3

There you go

local_overlay_enabled: true
series: focal

variables:
openstack-origin: &openstack-origin cloud:focal-victoria
data-port: &data-port br-ex:enp1s0
worker-multiplier: &worker-multiplier 0.25
osd-devices: &osd-devices /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf
expected-osd-count: &expected-osd-count 6
expected-mon-count: &expected-mon-count 6
maas_url: &maas_url http://x.x.x.23:5240/MAAS/api/2.0
maas_credentials: &maas_credentials
zone: &zone us-west-1
series: &series focal
machines:
‘1’:
constraints: arch=amd64 tags=ceph01 zones=us-west-1
series: *series
‘2’:
constraints: arch=amd64 tags=ceph02 zones=us-west-1
series: *series
‘3’:
constraints: arch=amd64 tags=ceph03 zones=us-west-1
series: *series
‘4’:
constraints: arch=amd64 tags=ceph04 zones=us-west-1
series: *series
‘5’:
constraints: arch=amd64 tags=ceph05 zones=us-west-1
series: *series
‘6’:
constraints: arch=amd64 tags=ceph06 zones=us-west-1
series: *series
‘7’:
constraints: arch=amd64 tags=controller01 zones=us-west-1
series: *series
‘8’:
constraints: arch=amd64 tags=controller02 zones=us-west-1
series: *series
‘9’:
constraints: arch=amd64 tags=controller03 zones=us-west-1
series: *series
‘10’:
constraints: arch=amd64 tags=compute01 zones=us-west-1
series: *series
‘11’:
constraints: arch=amd64 tags=compute02 zones=us-west-1
series: *series
‘12’:
constraints: arch=amd64 tags=compute03 zones=us-west-1
series: *series
‘13’:
constraints: arch=amd64 tags=compute04 zones=us-west-1
series: *series

relations:

CEPH

Ceph radosgw

    • ceph-radosgw:mon
    • ceph-mon:radosgw
    • ceph-radosgw:identity-service
    • keystone:identity-service

Ceph mon

    • ceph-mon:client
    • nova-compute:ceph
    • ceph-mon:client
    • cinder-ceph:ceph
    • ceph-mon:client
    • glance:ceph

Ceph osd

    • ceph-osd:mon
    • ceph-mon:osd

Nova compute

    • nova-compute:amqp
    • rabbitmq-server:amqp
    • nova-compute:image-service
    • glance:image-service
    • nova-compute:ceph-access
    • cinder-ceph:ceph-access

Ovn Central

    • ovn-central:certificates
    • vault:certificates
    • ovn-central:ovsdb-cms
    • neutron-api-plugin-ovn:ovsdb-cms

Ovn Chassis

    • ovn-chassis
    • nova-compute
    • ovn-chassis:certificates
    • vault:certificates
    • ovn-chassis:ovsdb
    • ovn-central:ovsdb

Neutron api

    • hacluster-neutron:ha
    • neutron-api:ha
    • neutron-api:identity-service
    • keystone:identity-service
    • neutron-api:shared-db
    • neutron-mysql-router:shared-db
    • neutron-api:amqp
    • rabbitmq-server:amqp
    • neutron-mysql-router:db-router
    • mysql-innodb-cluster:db-router
    • neutron-api-plugin-ovn:neutron-plugin
    • neutron-api:neutron-plugin-api-subordinate

Keystone

    • hacluster-keystone:ha
    • keystone:ha
    • keystone:shared-db
    • keystone-mysql-router:shared-db
    • keystone-mysql-router:db-router
    • mysql-innodb-cluster:db-router

Glance

    • hacluster-glance:ha
    • glance:ha
    • glance:identity-service
    • keystone:identity-service
    • glance:shared-db
    • glance-mysql-router:shared-db
    • glance:amqp
    • rabbitmq-server:amqp
    • glance-mysql-router:db-router
    • mysql-innodb-cluster:db-router

Nova Cloud Controller

    • hacluster-nova:ha
    • nova-cloud-controller:ha
    • nova-cloud-controller:identity-service
    • keystone:identity-service
    • nova-cloud-controller:image-service
    • glance:image-service
    • nova-cloud-controller:cloud-compute
    • nova-compute:cloud-compute
    • nova-cloud-controller:amqp
    • rabbitmq-server:amqp
    • nova-cloud-controller:shared-db
    • nova-mysql-router:shared-db
    • nova-cloud-controller:neutron-api
    • neutron-api:neutron-api
    • nova-mysql-router:db-router
    • mysql-innodb-cluster:db-router
    • nova-cloud-controller:memcache
    • memcached

Cinder

    • hacluster-cinder:ha
    • cinder:ha
    • cinder:image-service
    • glance:image-service
    • cinder:amqp
    • rabbitmq-server:amqp
    • cinder:identity-service
    • keystone:identity-service
    • cinder:cinder-volume-service
    • nova-cloud-controller:cinder-volume-service
    • cinder-ceph:storage-backend
    • cinder:storage-backend
    • cinder:shared-db
    • cinder-mysql-router:shared-db
    • cinder-mysql-router:db-router
    • mysql-innodb-cluster:db-router

Dashboard

    • hacluster-horizon:ha
    • openstack-dashboard:ha
    • openstack-dashboard:identity-service
    • keystone:identity-service
    • openstack-dashboard:shared-db
    • dashboard-mysql-router:shared-db
    • dashboard-mysql-router:db-router
    • mysql-innodb-cluster:db-router

NTP

    • ntp:juju-info
    • nova-compute:juju-info

# Ceilometer

- - ceilometer:ha

- hacluster-ceilometer:ha

- - ceilometer-agent

- nova-compute

- - ceilometer:amqp

- rabbitmq-server

- - ceilometer:identity-credentials

- keystone:identity-credentials

- - ceilometer

- keystone:identity-notifications

- - ceilometer:ceilometer-service

- ceilometer-agent:ceilometer-service

# Heat

- - heat:shared-db

- mysql:shared-db

- - heat:identity-service

- keystone:identity-service

- - heat:ha

- hacluster-heat:ha

- - heat:amqp

- rabbitmq-server:amqp

# Gnocchi

- - gnocchi:shared-db

- mysql:shared-db

- - gnocchi:identity-service

- keystone:identity-service

- - gnocchi:ha

- hacluster-gnocchi:ha

- - gnocchi:storage-ceph

- ceph-mon

- - gnocchi

- ceilometer

- - gnocchi:coordinator-memcached

- memcached

# Barbican

- - barbican:shared-db

- mysql:shared-db

- - barbican:ha

- hacluster-barbican:ha

- - barbican:identity-service

- keystone:identity-service

- - barbican:amqp

- rabbitmq-server:amqp

Placement

    • hacluster-placement:ha
    • placement:ha
    • placement:shared-db
    • placement-mysql-router:shared-db
    • placement:identity-service
    • keystone:identity-service
    • placement:placement
    • nova-cloud-controller:placement
    • placement-mysql-router:db-router
    • mysql-innodb-cluster:db-router

Vault

    • hacluster-vault:ha
    • vault:ha
    • vault:shared-db
    • vault-mysql-router:shared-db
    • vault-mysql-router:db-router
    • mysql-innodb-cluster:db-router
    • vault:certificates
    • neutron-api-plugin-ovn:certificates
    • vault:certificates
    • cinder:certificates
    • vault:certificates
    • glance:certificates
    • vault:certificates
    • keystone:certificates
    • vault:certificates
    • nova-cloud-controller:certificates
    • vault:certificates
    • openstack-dashboard:certificates
    • etcd:db
    • vault:etcd
    • etcd:certificates
    • easyrsa:client

services:
hacluster-horizon:
annotations:
“gui-x”: “629.4802856445312”
“gui-y”: “-187.46556091308594”
charm: cs:focal/hacluster-72
options:
maas_source: ppa:maas/2.9
corosync_key: |

cluster_count: 3
maas_url: *maas_url
maas_credentials: *maas_credentials

hacluster-cinder:
annotations:
“gui-x”: “1113.6179809570312”
“gui-y”: “-182.85367584228516”
charm: cs:focal/hacluster-72
options:
maas_source: ppa:maas/2.9
corosync_key: |

cluster_count: 3
maas_url: *maas_url
maas_credentials: *maas_credentials

hacluster-keystone:
annotations:
“gui-x”: “364.57623291015625”
“gui-y”: “-183.88646697998047”
charm: cs:focal/hacluster-72
options:
maas_source: ppa:maas/2.9
corosync_key: |

cluster_count: 3
maas_url: *maas_url
maas_credentials: *maas_credentials

hacluster-nova:
annotations:
“gui-x”: “-500.8421058654785”
“gui-y”: “-188.8864517211914”
charm: cs:focal/hacluster-72
options:
maas_source: ppa:maas/2.9
corosync_key: |

cluster_count: 3
maas_url: *maas_url
maas_credentials: *maas_credentials

hacluster-glance:
annotations:
“gui-x”: “125.77049398422241”
“gui-y”: “-182.24116134643555”
charm: cs:focal/hacluster-72
options:
maas_source: ppa:maas/2.9
corosync_key: |

cluster_count: 3
maas_url: *maas_url
maas_credentials: *maas_credentials

hacluster-neutron:
annotations:
“gui-x”: “-244.68324279785156”
“gui-y”: “-180.88827514648438”
charm: cs:focal/hacluster-72
options:
maas_source: ppa:maas/2.9
corosync_key: |

cluster_count: 3
maas_url: *maas_url
maas_credentials: *maas_credentials

hacluster-vault:
annotations:
“gui-x”: “-244.68324279785156”
“gui-y”: “-180.88827514648438”
charm: cs:focal/hacluster-72
options:
maas_source: ppa:maas/2.9
corosync_key: |

cluster_count: 3
maas_url: *maas_url
maas_credentials: *maas_credentials

hacluster-placement:
annotations:
“gui-x”: “-244.68324279785156”
“gui-y”: “-180.88827514648438”
charm: cs:focal/hacluster-72
options:
maas_source: ppa:maas/2.9
corosync_key: |

cluster_count: 3
maas_url: *maas_url
maas_credentials: *maas_credentials

ceph-mon:
annotations:
“gui-x”: “-31.47058868408203”
“gui-y”: “1241.4407653808594”
charm: cs:focal/ceph-mon
num_units: 6
options:
expected-osd-count: *expected-osd-count
monitor-count: *expected-mon-count
#fsid: 51f54ee0-829b-11e7-89d3-cf1d259b9b2c
monitor-secret:
to:
- ‘1’
- ‘2’
- ‘3’
- ‘4’
- ‘5’
- ‘6’

ceph-osd:
annotations:
“gui-x”: “-350.8948059082031”
“gui-y”: “1241.34326171875”
charm: cs:focal/ceph-osd
num_units: 6
options:
osd-devices: *osd-devices
#osd-reformat: ‘yes’
to:
- ‘1’
- ‘2’
- ‘3’
- ‘4’
- ‘5’
- ‘6’

ceph-radosgw:
annotations:
“gui-x”: “330.8906555175781”
“gui-y”: “1242.5653076171875”
charm: cs:focal/ceph-radosgw
num_units: 3
# options:
# use-embedded-webserver: true
to:
- lxd:1
- lxd:2
- lxd:3

cinder:
annotations:
“gui-x”: “1079.5557861328125”
“gui-y”: “354.4442138671875”
charm: cs:focal/cinder
num_units: 3
options:
openstack-origin: *openstack-origin
dns-ha: true
os-internal-network: x.x.x.0/24
os-public-hostname: “cinder..com”
os-internal-hostname: “cinder..internal”
os-admin-hostname: “cinder..internal”
block-device: None
glance-api-version: 2
worker-multiplier: *worker-multiplier
to:
- lxd:7
- lxd:8
- lxd:9

cinder-ceph:
annotations:
“gui-x”: “1110.9674072265625”
“gui-y”: “707.72314453125”
charm: cs:focal/cinder-ceph
num_units: 0

cinder-mysql-router:
annotations:
gui-x: ‘900’
gui-y: ‘1400’
charm: cs:focal/mysql-router
num_units: 0

glance-mysql-router:
annotations:
gui-x: ‘-290’
gui-y: ‘1400’
charm: cs:focal/mysql-router
num_units: 0

glance:
annotations:
“gui-x”: “91.16439819335938”
“gui-y”: “93.23036193847656”
charm: cs:focal/glance
options:
dns-ha: true
openstack-origin: *openstack-origin
os-internal-network: x.x.x.0/24
os-public-hostname: “glance..com”
os-internal-hostname: “glance..internal”
os-admin-hostname: “glance..internal”
num_units: 3
to:
- lxd:7
- lxd:8
- lxd:9

keystone-mysql-router:
annotations:
gui-x: ‘230’
gui-y: ‘1400’
charm: cs:focal/mysql-router
num_units: 0

keystone:
annotations:
“gui-x”: “336.1510925292969”
“gui-y”: “87.35755920410156”
charm: cs:focal/keystone
num_units: 3
options:
openstack-origin: *openstack-origin
admin-user:
admin-password:
dns-ha: true
os-internal-network: x.x.x.0/24
os-public-hostname: “keystone..com”
os-internal-hostname: “keystone..internal”
os-admin-hostname: “keystone..internal”
to:
- lxd:7
- lxd:8
- lxd:9

mysql-innodb-cluster:
annotations:
“gui-x”: “169.0853271484375”
“gui-y”: “669.0663452148438”
charm: cs:focal/mysql-innodb-cluster
num_units: 3
options:
innodb-buffer-pool-size: 512M
max-connections: 50000
os-db-router-hostname: “dbserver..internal”
to:
- lxd:7
- lxd:8
- lxd:9

neutron-mysql-router:
annotations:
gui-x: ‘505’
gui-y: ‘1385’
charm: cs:focal/mysql-router
num_units: 0

neutron-api-plugin-ovn:
annotations:
gui-x: ‘690’
gui-y: ‘1385’
charm: cs:focal/neutron-api-plugin-ovn

neutron-api:
annotations:
“gui-x”: “-276.74143981933594”
“gui-y”: “97.98399353027344”
charm: cs:focal/neutron-api
num_units: 3
options:
flat-network-providers: physnet1
neutron-security-groups: true
openstack-origin: *openstack-origin
worker-multiplier: *worker-multiplier
dns-ha: true
os-internal-network: x.x.x.0/24
os-public-hostname: “neutron-api..com”
os-internal-hostname: “neutron-api..internal”
os-admin-hostname: “neutron-api..internal”
to:
- lxd:7
- lxd:8
- lxd:9

ovn-central:
annotations:
gui-x: ‘70’
gui-y: ‘1550’
charm: cs:focal/ovn-central
num_units: 3
to:
- lxd:7
- lxd:8
- lxd:9

ovn-chassis:
annotations:
gui-x: ‘120’
gui-y: ‘1030’
charm: cs:focal/ovn-chassis
num_units: 0
options:
ovn-bridge-mappings: physnet1:br-ex
bridge-interface-mappings: *data-port

nova-mysql-router:
annotations:
gui-x: ‘-30’
gui-y: ‘1385’
charm: cs:focal/mysql-router
num_units: 0

nova-cloud-controller:
annotations:
“gui-x”: “-523.3897094726562”
“gui-y”: “97.57771301269531”
charm: cs:focal/nova-cloud-controller
num_units: 3
options:
network-manager: Neutron
openstack-origin: *openstack-origin
worker-multiplier: *worker-multiplier
console-access-protocol: novnc
console-proxy-ip: “cloud..com”
#config-flags: “ssl_only=true,novncproxy_host=0.0.0.0,novncproxy_port=6080,novncproxy_base_url=https://cloud..com:6080/vnc_auto.html”
config-flags: “novncproxy_host=0.0.0.0,novncproxy_port=6080,novncproxy_base_url=https://cloud..com:6080/vnc_auto.html”
dns-ha: true
os-internal-network: x.x.x.0/24
os-public-hostname: “nova..com”
os-internal-hostname: “nova..internal”
os-admin-hostname: “nova..internal”
cpu-allocation-ratio: 2.5
ram-allocation-ratio: 1.5
to:
- lxd:7
- lxd:8
- lxd:9

placement-mysql-router:
annotations:
gui-x: ‘1320’
gui-y: ‘1385’
charm: cs:focal/mysql-router
num_units: 0

placement:
annotations:
“gui-x”: “-523.3897094726562”
“gui-y”: “97.57771301269531”
charm: cs:focal/placement
num_units: 3
options:
os-internal-network: x.x.x.0/24
os-public-hostname: “placement..com”
os-internal-hostname: “placement..internal”
os-admin-hostname: “placement..internal”
openstack-origin: *openstack-origin
to:
- lxd:7
- lxd:8
- lxd:9

nova-compute:
annotations:
“gui-x”: “-1058.333251953125”
“gui-y”: “328.2070007324219”
charm: cs:focal/nova-compute
num_units: 4
options:
openstack-origin: *openstack-origin
enable-live-migration: true
enable-resize: true
migration-auth-type: ssh
config-flags: default_ephemeral_format=ext4
to:
- ‘10’
- ‘11’
- ‘12’
- ‘13’

ntp:
annotations:
“gui-x”: “-1034.0950012207031”
“gui-y”: “635.7695922851562”
charm: cs:focal/ntp
num_units: 0

dashboard-mysql-router:
annotations:
gui-x: ‘510’
gui-y: ‘1030’
charm: cs:focal/mysql-router
num_units: 0

openstack-dashboard:
annotations:
“gui-x”: “599.20654296875”
“gui-y”: “81.98934936523438”
charm: cs:focal/openstack-dashboard
num_units: 3
options:
openstack-origin: *openstack-origin
neutron-network-vpn: true
neutron-network-lb: true
ubuntu-theme: “no”
cinder-backup: true
neutron-network-l3ha: true
webroot: “/”
dns-ha: true
os-public-hostname: “sws..com”
endpoint-type: internalURL
#os-internal-hostname: “horizon..internal”
#os-admin-hostname: “horizon..internal”
to:
- lxd:7
- lxd:8
- lxd:9

rabbitmq-server:
annotations:
“gui-x”: “-184”
“gui-y”: “634”
charm: cs:focal/rabbitmq-server
num_units: 3
options:
min-cluster-size: 3
to:
- lxd:7
- lxd:8
- lxd:9

vault-mysql-router:
annotations:
gui-x: ‘1535’
gui-y: ‘1560’
charm: cs:focal/mysql-router
num_units: 0

vault:
annotations:
gui-x: ‘1610’
gui-y: ‘1430’
charm: cs:vault
num_units: 3
options:
dns-ha-access-record: “vault..internal”
hostname: “horizon..internal”
to:
- lxd:7
- lxd:8
- lxd:9

Memcached is used for nova-cloud-controller

memcached:
annotations:
“gui-x”: “1647.76416015625”
“gui-y”: “82.0185546875”
charm: cs:focal/memcached
num_units: 3
options:
repcached: False ## Repcached replication only can take place between two units. If cluster is > 2 then this should be false
repcached_origin: ‘ppa:iacobs/memcached-repcached’
annotations:
gui-x: “2528.00146484375”
gui-y: “794.591796875”
to:
- lxd:7
- lxd:8
- lxd:9

Etcd is used for vault ha

etcd:
annotations:
“gui-x”: “1647.76416015625”
“gui-y”: “82.0185546875”
charm: cs:focal/etcd
num_units: 3
annotations:
gui-x: “2528.00146484375”
gui-y: “794.591796875”
to:
- lxd:7
- lxd:8
- lxd:9

Etcd is used for vault ha

easyrsa:
annotations:
“gui-x”: “1647.76416015625”
“gui-y”: “82.0185546875”
charm: cs:~containers/easyrsa
num_units: 1
annotations:
gui-x: “2528.00146484375”
gui-y: “794.591796875”
to:
- lxd:7

4

Unfortunately, I think the formatting has messed up the paste of the bundle (perhaps try it as a file). e.g. I couldn’t find machine 62 in the bundle anywhere. Maybe a bit more of the log to get context of what is going on?

5

There is no machine 62 cause it does not exist. The way the bundle will work is that it will find the appropriate machine with the tag and then assign the number. The numbers inside the bundle have nothing to do with the actual assigned number.

6

Yes, I aware of that, although allocating to machine tends to make things clearer. My point (badly made) is that there isn’t enough detail in what you’ve posted so far to work out what is going on. Perhaps if you could please attach/upload (not copy/paste into the message body) the bundle and the debug log from the deploy, and the juju status (text is easiest to read). (e.g. from juju status).

Is the bundle you posted above the one you used to deploy with? (e.g. juju deploy <some-bundle>)

  • Where did the bundle come from?
  • What version of juju are you using?
  • What/where is the controller bootstrapped to? (e.g. the version of MaaS = 2.9?)
  • Is there a bit more context from the log? (i.e. could you add the whole thing?)

Unfortunately, the formatting of the message above makes it very difficult to see what the bundle is actually saying (although I can get the gist of it).

I suspect that it something to do with the configuration of MaaS, but it’s hard to tell at the moment. The error is coming from within the MaaS provisioner (in validate_constraint_label_name(...)) so it’s what’s being handed to MaaS that is causing the problem.

Thanks.