Question about juju add-machine

Hi all

I observed a strange behaviour when adding machine via add-machine command ssh:ubuntu@hostname.

The machine gets a fqdn in field address instead of first adapter ip address which then causes charms like mysql innodb cluster to fail installation.

Is this the desired behavior? Are you planning on introducing some kind of mapping between fqdn and ip?

Thanks in advance

What are you using for a cloud? MaaS, AWS, etc… Also what shows up as the hostname?

I use maas cloud in general, but the machines I am adding are lxc containers generated via lxc launch e.g

lxc launch images:ubuntu/jammy/cloud juju-test -p container-vlan10 --target infra3 -c user.network-config="$(cat /home/mother/cloud-init-scripts/network-config-infra-container.yaml | sed -e 's/IP_ADDR_10/10.10.10.80/g;s/IP_ADDR_99/10.10.99.80/g;s/IP_GATEWAY/10.10.10.1/g')"
root@juju-client:~# juju status --relations
Model    Controller       Cloud/Region  Version  SLA          Timestamp
testing  juju-maas-3-0-3  maas/default  3.0.3.1  unsupported  17:36:14Z

App                   Version  Status  Scale  Charm                 Channel      Rev  Exposed  Message
mysql-innodb-cluster  8.0.31   active      3  mysql-innodb-cluster  latest/edge   37  no       Unit is ready: Mode: R/W, Cluster is ONLINE and can tolerate up to ONE failure.

Unit                     Workload  Agent  Machine  Public address  Ports  Message
mysql-innodb-cluster/4*  active    idle   19       10.10.99.141           Unit is ready: Mode: R/W, Cluster is ONLINE and can tolerate up to ONE failure.
mysql-innodb-cluster/5   active    idle   20       10.10.99.142           Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
mysql-innodb-cluster/6   active    idle   21       10.10.99.143           Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.

Machine  State    Address       Inst id              Base          AZ       Message
5        started  10.10.10.171  juju-k8s-1           ubuntu@22.04  default  Deployed
19       started  10.10.99.141  manual:10.10.99.141  ubuntu@22.04           Manually provisioned machine
20       started  10.10.99.142  manual:10.10.99.142  ubuntu@22.04           Manually provisioned machine
21       started  10.10.99.143  manual:10.10.99.143  ubuntu@22.04           Manually provisioned machine

Relation provider                 Requirer                          Interface             Type  Message
mysql-innodb-cluster:cluster      mysql-innodb-cluster:cluster      mysql-innodb-cluster  peer
mysql-innodb-cluster:coordinator  mysql-innodb-cluster:coordinator  coordinator           peer
root@juju-client:~# juju add-machine ssh:ubuntu@juju-test
The authenticity of host 'juju-test (240.11.13.171)' can't be established.
ECDSA key fingerprint is SHA256:cVZY0P27zdpJ6veqlyK2q4bvwFAKRiuknai6iE0c9ak.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
The authenticity of host 'juju-test (240.11.13.171)' can't be established.
ECDSA key fingerprint is SHA256:cVZY0P27zdpJ6veqlyK2q4bvwFAKRiuknai6iE0c9ak.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
The authenticity of host 'juju-test (240.11.13.171)' can't be established.
ECDSA key fingerprint is SHA256:cVZY0P27zdpJ6veqlyK2q4bvwFAKRiuknai6iE0c9ak.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
The authenticity of host 'juju-test (240.11.13.171)' can't be established.
ECDSA key fingerprint is SHA256:cVZY0P27zdpJ6veqlyK2q4bvwFAKRiuknai6iE0c9ak.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Failed to add the host to the list of known hosts (/root/.ssh/known_hosts).
created machine 22
root@juju-client:~# juju status --relations
Model    Controller       Cloud/Region  Version  SLA          Timestamp
testing  juju-maas-3-0-3  maas/default  3.0.3.1  unsupported  17:41:56Z

App                   Version  Status  Scale  Charm                 Channel      Rev  Exposed  Message
mysql-innodb-cluster  8.0.31   active      3  mysql-innodb-cluster  latest/edge   37  no       Unit is ready: Mode: R/W, Cluster is ONLINE and can tolerate up to ONE failure.

Unit                     Workload  Agent  Machine  Public address  Ports  Message
mysql-innodb-cluster/4*  active    idle   19       10.10.99.141           Unit is ready: Mode: R/W, Cluster is ONLINE and can tolerate up to ONE failure.
mysql-innodb-cluster/5   active    idle   20       10.10.99.142           Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
mysql-innodb-cluster/6   active    idle   21       10.10.99.143           Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.

Machine  State    Address       Inst id              Base          AZ       Message
5        started  10.10.10.171  juju-k8s-1           ubuntu@22.04  default  Deployed
19       started  10.10.99.141  manual:10.10.99.141  ubuntu@22.04           Manually provisioned machine
20       started  10.10.99.142  manual:10.10.99.142  ubuntu@22.04           Manually provisioned machine
21       started  10.10.99.143  manual:10.10.99.143  ubuntu@22.04           Manually provisioned machine
22       started  juju-test     manual:juju-test     ubuntu@22.04           Manually provisioned machine

Relation provider                 Requirer                          Interface             Type  Message
mysql-innodb-cluster:cluster      mysql-innodb-cluster:cluster      mysql-innodb-cluster  peer
mysql-innodb-cluster:coordinator  mysql-innodb-cluster:coordinator  coordinator           peer
root@juju-client:~#

An attempt to deploy mysql to that machine ends with following error in debug


Traceback (most recent call last):
File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/mysql/v0/mysql.py", line 497, in configure_instance
self._run_mysqlsh_script("\n".join(configure_instance_command))
File "/var/lib/juju/agents/unit-mysql-0/charm/src/mysql_vm_helpers.py", line 184, in _run_mysqlsh_script
raise MySQLClientError(e.stderr)
charms.mysql.v0.mysql.MySQLClientError: b'Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\nConfiguring local MySQL instance listening at port 3306 for use in an InnoDB cluster...\n\nThis instance reports its own addressas \x1b[1mjuju-test:3306\x1b[0m\nClients and other cluster members will communicate with it through this address by default. If this is not correct, the report_host MySQL system variable should be changed.\n\x1b[31mERROR: \x1b[0mCannot use host \'juju-test\' for instance \'juju-test:3306\' because it resolves to an IP address (127.0.1.1) that does not match a real network interface, thus it is not supported. Change your system settings and/or set the MySQL server \'report_host\' variable to a hostname that resolves to a supported IP address.\nTraceback (most recent call last):\n  File "<string>", line 1, in <module>\nRuntimeError: Dba.configure_instance: Invalid host/IP \'juju-test\' resolves to \'127.0.1.1\' which is not supported.\n\n'

What does juju show-machine <x> give you for one of these machines?

root@juju-client:~# juju show-machine 34
model: testing
machines:
  "34":
    juju-status:
      current: started
      since: 16 Jan 2023 22:39:37Z
      version: 3.0.3.1
    hostname: juju-cassandra-3
    dns-name: 10.10.99.143
    ip-addresses:
    - 10.10.99.143
    - 10.10.10.143
    - 240.11.13.54
    instance-id: manual:10.10.99.143
    machine-status:
      current: running
      message: Manually provisioned machine
      since: 16 Jan 2023 22:39:14Z
    modification-status:
      current: idle
      since: 16 Jan 2023 22:39:13Z
    base:
      name: ubuntu
      channel: "22.04"
    network-interfaces:
      eth0:
        ip-addresses:
        - 10.10.99.143
        mac-address: 00:16:3e:68:73:81
        space: lub-net
        is-up: true
      eth1:
        ip-addresses:
        - 240.11.13.54
        mac-address: 00:16:3e:8a:42:e7
        space: lub-net
        is-up: true
      eth2:
        ip-addresses:
        - 10.10.10.143
        mac-address: 00:16:3e:52:1e:df
        gateway: 10.10.10.1
        space: lub-net
        is-up: true
    hardware: arch=amd64 cores=8 mem=3814M

Interesting, right now even when I add miachine via IP innodb cluster charm shows followin:

unit-mysql-innodb-cluster-16: 09:12:25 ERROR juju.worker.uniter.operation hook "install" (via explicit, bespoke hook script) failed: exit status 1
unit-mysql-innodb-cluster-16: 09:17:49 ERROR unit.mysql-innodb-cluster/16.juju-log Failed to connect to database due to '(1045, "Access denied for user 'root'@'localhost' (using password: YES)")'
unit-mysql-innodb-cluster-16: 09:17:49 ERROR unit.mysql-innodb-cluster/16.juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-16/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-16/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-16/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-16/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-16/charm/reactive/mysql_innodb_cluster_handlers.py", line 71, in create_local_cluster_user
    if not instance.create_cluster_user(
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-16/charm/lib/charm/openstack/mysql_innodb_cluster.py", line 539, in create_cluster_user
    m_helper.connect(password=self.mysql_password)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-16/.venv/lib/python3.10/site-packages/charmhelpers/contrib/database/mysql.py", line 107, in connect
    self.connection = MySQLdb.connect(**_connection_info)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-16/.venv/lib/python3.10/site-packages/MySQLdb/__init__.py", line 123, in Connect
    return Connection(*args, **kwargs)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-16/.venv/lib/python3.10/site-packages/MySQLdb/connections.py", line 185, in __init__
    super().__init__(*args, **kwargs2)
MySQLdb._exceptions.OperationalError: (1045, "Access denied for user 'root'@'localhost' (using password: YES)")

unit-mysql-innodb-cluster-16: 09:17:49 ERROR juju.worker.uniter.operation hook "install" (via explicit, bespoke hook script) failed: exit status 1

instance-id: manual:10.10.99.143 would indicate that this was added by its IP. Is that correct?

I was asking after one added by hostname.