Adding manual machine fails on "address not valid"

1

I’m trying to add a physical machine manually to a model and it seems to work:

image
image1178×817 230 KB

However, the machine never leaves the pending state.

This is in /var/log/machine-1.log

2022-03-08 17:20:37 DEBUG juju.network network.go:142 no lxc bridge addresses to filter for machine
2022-03-08 17:20:37 DEBUG juju.worker.logger logger.go:93 reconfiguring logging from "<root>=DEBUG" to "<root>=INFO"
2022-03-08 17:20:37 INFO juju.worker.machiner machiner.go:162 setting addresses for "machine-1" to [local-machine:127.0.0.1 local-machine:::1]
2022-03-08 17:20:38 INFO juju.worker.upgradeseries worker.go:161 no series upgrade lock present
2022-03-08 17:20:38 INFO juju.worker.authenticationworker worker.go:103 "machine-1" key updater worker started
2022-03-08 17:20:38 INFO juju.worker.logger logger.go:136 logger worker stopped
2022-03-08 17:20:38 ERROR juju.worker.dependency engine.go:693 "api-caller" manifold worker returned unexpected error: [fb0a48] "machine-1" cannot open api: validating info for opening an API connection: missing addresses not valid
2022-03-08 17:20:38 ERROR juju.worker.dependency engine.go:693 "api-caller" manifold worker returned unexpected error: [fb0a48] "machine-1" cannot open api: validating info for opening an API connection: missing addresses not valid
2022-03-08 17:20:41 ERROR juju.worker.dependency engine.go:693 "api-caller" manifold worker returned unexpected error: [fb0a48] "machine-1" cannot open api: validating info for opening an API connection: missing addresses not valid
2022-03-08 17:20:46 ERROR juju.worker.dependency engine.go:693 "api-caller" manifold worker returned unexpected error: [fb0a48] "machine-1" cannot open api: validating info for opening an API connection: missing addresses not valid
2022-03-08 17:20:51 ERROR juju.worker.dependency engine.go:693 "api-caller" manifold worker returned unexpected error: [fb0a48] "machine-1" cannot open api: validating info for opening an API connection: missing addresses not valid
2022-03-08 17:20:57 ERROR juju.worker.dependency engine.go:693 "api-caller" manifold worker returned unexpected error: [fb0a48] "machine-1" cannot open api: validating info for opening an API connection: missing addresses not valid
2022-03-08 17:21:04 ERROR juju.worker.dependency engine.go:693 "api-caller" manifold worker returned unexpected error: [fb0a48] "machine-1" cannot open api: validating info for opening an API connection: missing addresses not valid
2022-03-08 17:21:13 ERROR juju.worker.dependency engine.go:693 "api-caller" manifold worker returned unexpected error: [fb0a48] "machine-1" cannot open api: validating info for opening an API connection: missing addresses not valid
2022-03-08 17:21:24 ERROR juju.worker.dependency engine.go:693 "api-caller" manifold worker returned unexpected error: [fb0a48] "machine-1" cannot open api: validating info for opening an API connection: missing addresses not valid

I’ve tried with juju version 2.9.22 and also with 2.9.25 (controller, client and model)

Cloud is lxd

2

Interesting, I’m unable to reproduce with 2.9.25, and lxd cloud and the manual machine is a lxd container running focal.

What type of machine is the manual one?

3

Its a hardware, x86_64…

This is the network config:

cat /etc/netplan/00-installer-config.yaml

This is the network config written by ‘subiquity’

network:
  version: 2
  ethernets:
    ens5f0:
      dhcp4: no
      dhcp6: no
      addresses: [ 192.168.2.3/24 ]
  bridges:
    lxdbr0:
      interfaces: [ vlan2 ]
      addresses: [ 192.168.2.2/24 ]
      gateway4: 192.168.2.1
      nameservers:
        addresses:
        - 192.168.2.1
        search:
        - garage.lonroth.net
      parameters:
        stp: true
        forward-delay: 4
      dhcp4: no
      dhcp6: no
 
  vlans:
    vlan2:
      id: 2
      link: ens5f0
      dhcp4: no
      dhcp6: no

Never mind the vlan config, its not working on a similar setup without.

This is what I get in /var/log/juju/machine-3.log

2022-03-10 18:58:04 ERROR juju.worker.dependency engine.go:693 "api-caller" manifold worker returned unexpected error: [fb0a48] "machine-3" cannot open api: validating info for opening an API connection: missing addresses not valid

4

The other thing to look for is if there is anything interesting in /var/lib/juju/agent/machine-*/agent.conf It at least sounds like it is missing lines around:

apiaddresses:
- 10.5.24.10:17070

That particular error comes from: api/interface.go

func (info *Info) Validate() error {
        if len(info.Addrs) == 0 {
                return errors.NotValidf("missing addresses")
        }

I don’t know why add-machine would have resulted in the newly added machine not having any IP addresses for the controller. Is there anything interesting around your controller setup? (config-mgmt-space comes to mind)

5

You should be able to add an apiaddress in that field, and the agent should then use that to connect to the controller, and if the controller is HA, it will pick up any additional addresses. IIRC we did have a bug in the past where the controller would filter out addresses incorrectly, but that was quite a while ago.

6

No apiaddresses in there:

cat /var/lib/juju/agents/machine-3/agent.conf

# format 2.0
tag: machine-3
datadir: /var/lib/juju
transient-datadir: /var/run/juju
logdir: /var/log/juju
metricsspooldir: /var/lib/juju/metricspool
nonce: manual:192.168.2.3:6bb994ca-15d2-4b73-877c-a0630dc19b4d
jobs:
- JobHostUnits
upgradedToVersion: 2.9.25
cacert: |
  -----BEGIN CERTIFICATE-----
 ...
  -----END CERTIFICATE-----
controller: controller-37475adf-9af1-421e-8032-318cd6f47bab
model: model-fb0a48a3-72b3-430f-88c0-ed246604a5eb
apipassword: ****************
oldpassword: ****************
loggingconfig: <root>=INFO
values:
  AGENT_SERVICE_NAME: jujud-machine-3
  CONTAINER_TYPE: ""
  NUMA_CTL_PREFERENCE: "false"
  PROVIDER_TYPE: lxd
agent-logfile-max-size: 100
agent-logfile-max-backups: 2
mongoversion: "0.0"
7 
erik@pixel:~ > juju controller-config config-mgmt-space
ERROR key "config-mgmt-space" not found in "iceberg-default" controller
erik@pixel:~ >
8

Sorry, the key is juju-mgmt-space but I wouldn’t particularly expect it to be set (it only really gets set when you have a controller machine that has multiple network devices).

1 Like
9

@jameinel

I did set the apiaddresses to my controller IP and then systemctl restart jujud-machine-3.service

It didn’t solve the issue.

It seemed to overwrite that config, because when I looked again in the config file, the entry was gone. Is jujud rewriting the config from cache?

I’ll try stop, update file, start …

10

I somehow managed to get it in there now, but now I get this error:

2022-03-10 19:30:44 ERROR juju.worker.apicaller connect.go:209 Failed to connect to controller: invalid entity name or password (unauthorized access)

11

this sounds like

and/or
1 Like
12

After editing that file, getting the apiaddress in there, I get this

2022-03-10 19:35:32 ERROR juju.worker.apicaller connect.go:209 Failed to connect to controller: invalid entity name or password (unauthorized access)

I saw that the apipassword was lost from the configuration file also now after restart…

@jameinel I experience that after restart of the machine agent, either the apiaddress or the apipassword keys are removed from the agent config ( /var/lib/juju/agents/machine-*/agent.conf)

Something really weird is going on here.

13

@jameinel I’ve added a new bug with my findings…

14

I’m still stuck here. I have no idea why the machine is stuck in pending.

I just tested again with juju 2.9.27 - still not able to get through this.