Debug data.
This run was better than I had seen on any previous run because a machine was found with the --storage option specified but the unit and application got stuck allocating and no volume was created or mounted. Debug log could not be attached (system only allowed image files).
I defined a raw software RAID volume on the machine and gave it the tag ‘raid’ in MAAS. The tag I had previously used (where no machine match was found was ‘md0’).
The commands I used to create storage and deploy were:
$ juju storage-pools
Name Provider Attrs
loop loop
maas maas
rootfs rootfs
tmpfs tmpfs
$ juju create-storage-pool raid maas tags=raid
$ juju storage-pools
Name Provider Attrs
loop loop
maas maas
raid maas tags=raid
rootfs rootfs
tmpfs tmpfs
$ juju deploy --constraints tags=Elasticsearch-debug --storage data=raid ~/charms/elasticsearch
Deploying charm "local:bionic/elasticsearch-3".
$ juju status
Model Controller Cloud/Region Version SLA Timestamp
default maas-controller maas-region 2.6.9 unsupported 22:42:42Z
App Version Status Scale Charm Store Rev OS Notes
elasticsearch waiting 0/1 elasticsearch local 3 ubuntu
Unit Workload Agent Machine Public address Ports Message
elasticsearch/3 waiting allocating 3 waiting for machine
Machine State DNS Inst id Series AZ Message
3 pending pending bionic failed to start machine 3 in zone "default", retrying in 10s with new availability zone: failed to acquire node: No available machine matches constraints: [('agent_name', ['f9ce5fcc-1afb-4ada-8b97-3e23422cf4ce']), ('storage', ['root:0,0:1(raid)']), ('tags', ['Elasticsearch-debug']), ('zone', ['default'])] (resolved to "storage=root:0,0:1(raid) tags=Elasticsearch-debug zone=default")
$ juju status
Model Controller Cloud/Region Version SLA Timestamp
default maas-controller maas-region 2.6.9 unsupported 22:43:11Z
App Version Status Scale Charm Store Rev OS Notes
elasticsearch waiting 0/1 elasticsearch local 3 ubuntu
Unit Workload Agent Machine Public address Ports Message
elasticsearch/3 waiting allocating 3 10.XXX.XXX.218 waiting for machine
Machine State DNS Inst id Series AZ Message
3 pending 10.XXX.XXX.218 machine-3 bionic rack_14 Deploying
Since I am a new user, the system is telling me that I am at my maximum reply count for this topic. Is there another way to upload the debug log? Last bit of the log (up to where it got stuck):
machine-3: 22:58:56 DEBUG juju.worker.fanconfigurer Fan not enabled
machine-3: 22:58:56 DEBUG juju.worker.dependency "fan-configurer" manifold worker started at 2019-10-18 22:58:56.800789384 +0000 UTC
machine-3: 22:58:56 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=INFO;juju.provider.maas=TRACE;unit=DEBUG"
machine-3: 22:58:56 ERROR juju.worker.dependency "broker-tracker" manifold worker returned unexpected error: no container types determined
machine-3: 22:58:56 INFO juju.worker.machiner setting addresses for "machine-3" to [local-machine:127.0.0.1 local-cloud:10.XXX.XXX.218 local-cloud:192.168.100.251 local-machine:::1]
machine-3: 22:58:56 INFO juju.worker.upgradeseries no series upgrade lock present
machine-3: 22:58:56 INFO juju.worker.authenticationworker "machine-3" key updater worker started
machine-3: 22:58:56 INFO juju.worker.machiner "machine-3" started
machine-3: 22:58:57 INFO juju.api.common no addresses observed on interface "enp4s0f0"
machine-3: 22:58:57 INFO juju.api.common no addresses observed on interface "enp4s0f1"
machine-3: 22:58:57 INFO juju.api.common no addresses observed on interface "enp133s0f0"
machine-3: 22:58:57 INFO juju.api.common no addresses observed on interface "enp133s0f1"
machine-3: 22:58:57 INFO juju.api.common no addresses observed on interface "enp135s0f0"
machine-3: 22:58:57 INFO juju.api.common no addresses observed on interface "enp135s0f1"
machine-3: 22:58:57 INFO juju.worker.deployer checking unit "elasticsearch/3"
machine-3: 22:58:57 INFO juju.worker.deployer deploying unit "elasticsearch/3"
machine-3: 22:58:57 INFO juju.api.common no addresses observed on interface "enp4s0f0"
machine-3: 22:58:57 INFO juju.api.common no addresses observed on interface "enp4s0f1"
machine-3: 22:58:57 INFO juju.api.common no addresses observed on interface "enp133s0f0"
machine-3: 22:58:57 INFO juju.api.common no addresses observed on interface "enp133s0f1"
machine-3: 22:58:57 INFO juju.api.common no addresses observed on interface "enp135s0f0"
machine-3: 22:58:57 INFO juju.api.common no addresses observed on interface "enp135s0f1"
machine-3: 22:58:58 INFO juju.service Installing and starting service &{Service:{Name:jujud-unit-elasticsearch-3 Conf:{Desc:juju unit agent for elasticsearch/3 Transient:false AfterStopped: Env:map[JUJU_CONTAINER_TYPE:] Limit:map[] Timeout:300 ExecStart:/lib/systemd/system/jujud-unit-elasticsearch-3/exec-start.sh ExecStopPost: Logfile:/var/log/juju/unit-elasticsearch-3.log ExtraScript: ServiceBinary:/var/lib/juju/tools/unit-elasticsearch-3/jujud ServiceArgs:[unit --data-dir /var/lib/juju --unit-name elasticsearch/3 --debug]}} ConfName:jujud-unit-elasticsearch-3.service UnitName:jujud-unit-elasticsearch-3.service DirName:/lib/systemd/system/jujud-unit-elasticsearch-3 FallBackDirName:/var/lib/juju/init Script:[35 33 47 117 115 114 47 98 105 110 47 101 110 118 32 98 97 115 104 10 10 35 32 83 101 116 32 117 112 32 108 111 103 103 105 110 103 46 10 116 111 117 99 104 32 39 47 118 97 114 47 108 111 103 47 106 117 106 117 47 117 110 105 116 45 101 108 97 115 116 105 99 115 101 97 114 99 104 45 51 46 108 111 103 39 10 99 104 111 119 110 32 115 121 115 108 111 103 58 115 121 115 108 111 103 32 39 47 118 97 114 47 108 111 103 47 106 117 106 117 47 117 110 105 116 45 101 108 97 115 116 105 99 115 101 97 114 99 104 45 51 46 108 111 103 39 10 99 104 109 111 100 32 48 54 48 48 32 39 47 118 97 114 47 108 111 103 47 106 117 106 117 47 117 110 105 116 45 101 108 97 115 116 105 99 115 101 97 114 99 104 45 51 46 108 111 103 39 10 101 120 101 99 32 62 62 32 39 47 118 97 114 47 108 111 103 47 106 117 106 117 47 117 110 105 116 45 101 108 97 115 116 105 99 115 101 97 114 99 104 45 51 46 108 111 103 39 10 101 120 101 99 32 50 62 38 49 10 10 35 32 82 117 110 32 116 104 101 32 115 99 114 105 112 116 46 10 39 47 118 97 114 47 108 105 98 47 106 117 106 117 47 116 111 111 108 115 47 117 110 105 116 45 101 108 97 115 116 105 99 115 101 97 114 99 104 45 51 47 106 117 106 117 100 39 32 117 110 105 116 32 45 45 100 97 116 97 45 100 105 114 32 39 47 118 97 114 47 108 105 98 47 106 117 106 117 39 32 45 45 117 110 105 116 45 110 97 109 101 32 101 108 97 115 116 105 99 115 101 97 114 99 104 47 51 32 45 45 100 101 98 117 103] newDBus:0xb57120}
unit-elasticsearch-3: 22:58:59 INFO juju.cmd running jujud [2.6.9 gc go1.11.13]
unit-elasticsearch-3: 22:58:59 DEBUG juju.cmd args: []string{"/var/lib/juju/tools/unit-elasticsearch-3/jujud", "unit", "--data-dir", "/var/lib/juju", "--unit-name", "elasticsearch/3", "--debug"}
unit-elasticsearch-3: 22:58:59 DEBUG juju.agent read agent config, format "2.0"
unit-elasticsearch-3: 22:58:59 INFO juju.worker.upgradesteps upgrade steps for 2.6.9 have already been run.
unit-elasticsearch-3: 22:58:59 DEBUG juju.worker.dependency "agent" manifold worker started at 2019-10-18 22:58:59.646885396 +0000 UTC
unit-elasticsearch-3: 22:58:59 DEBUG juju.worker.dependency "upgrade-check-gate" manifold worker started at 2019-10-18 22:58:59.647097174 +0000 UTC
unit-elasticsearch-3: 22:58:59 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2019-10-18 22:58:59.647358681 +0000 UTC
unit-elasticsearch-3: 22:58:59 DEBUG juju.worker.apicaller connecting with old password
unit-elasticsearch-3: 22:58:59 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2019-10-18 22:58:59.647799832 +0000 UTC
unit-elasticsearch-3: 22:58:59 DEBUG juju.worker.introspection introspection worker listening on "@jujud-unit-elasticsearch-3"
unit-elasticsearch-3: 22:58:59 DEBUG juju.worker.introspection stats worker now serving
unit-elasticsearch-3: 22:58:59 DEBUG juju.worker.dependency "upgrade-check-flag" manifold worker started at 2019-10-18 22:58:59.656365678 +0000 UTC
unit-elasticsearch-3: 22:58:59 DEBUG juju.api successfully dialed "wss://10.XXX.XXX.226:17070/model/f9ce5fcc-1afb-4ada-8b97-3e23422cf4ce/api"
unit-elasticsearch-3: 22:58:59 INFO juju.api connection established to "wss://10.XXX.XXX.226:17070/model/f9ce5fcc-1afb-4ada-8b97-3e23422cf4ce/api"
unit-elasticsearch-3: 22:58:59 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2019-10-18 22:58:59.659018794 +0000 UTC
unit-elasticsearch-3: 22:59:00 INFO juju.worker.apicaller [f9ce5f] "unit-elasticsearch-3" successfully connected to "10.XXX.XXX.226:17070"
unit-elasticsearch-3: 22:59:00 DEBUG juju.worker.apicaller changing password...
unit-elasticsearch-3: 22:59:00 INFO juju.worker.apicaller [f9ce5f] password changed for "unit-elasticsearch-3"
unit-elasticsearch-3: 22:59:00 DEBUG juju.api RPC connection died
unit-elasticsearch-3: 22:59:00 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: restart immediately
unit-elasticsearch-3: 22:59:00 DEBUG juju.worker.apicaller connecting with current password
unit-elasticsearch-3: 22:59:00 DEBUG juju.api successfully dialed "wss://10.XXX.XXX.226:17070/model/f9ce5fcc-1afb-4ada-8b97-3e23422cf4ce/api"
unit-elasticsearch-3: 22:59:00 INFO juju.api connection established to "wss://10.XXX.XXX.226:17070/model/f9ce5fcc-1afb-4ada-8b97-3e23422cf4ce/api"
unit-elasticsearch-3: 22:59:01 INFO juju.worker.apicaller [f9ce5f] "unit-elasticsearch-3" successfully connected to "10.XXX.XXX.226:17070"
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2019-10-18 22:59:01.041181613 +0000 UTC
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2019-10-18 22:59:01.051612576 +0000 UTC
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2019-10-18 22:59:01.051683442 +0000 UTC
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2019-10-18 22:59:01.053458345 +0000 UTC
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2019-10-18 22:59:01.053959333 +0000 UTC
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
unit-elasticsearch-3: 22:59:01 INFO juju.worker.upgrader abort check blocked until version event received
unit-elasticsearch-3: 22:59:01 INFO juju.worker.upgrader unblocking abort check
unit-elasticsearch-3: 22:59:01 INFO juju.worker.upgrader desired agent binary version: 2.6.9
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "upgrade-check-flag" manifold worker stopped: gate unlocked
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "upgrade-check-flag" manifold worker started at 2019-10-18 22:59:01.098426818 +0000 UTC
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2019-10-18 22:59:01.108687667 +0000 UTC
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2019-10-18 22:59:01.119214019 +0000 UTC
unit-elasticsearch-3: 22:59:01 INFO juju.worker.migrationminion migration phase is now: NONE
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2019-10-18 22:59:01.131532043 +0000 UTC
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2019-10-18 22:59:01.131606144 +0000 UTC
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.leadership elasticsearch/3 making initial claim for elasticsearch leadership
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
unit-elasticsearch-3: 22:59:01 INFO juju.worker.logger logger worker started
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "leadership-tracker" manifold worker started at 2019-10-18 22:59:01.131706924 +0000 UTC
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "meter-status" manifold worker started at 2019-10-18 22:59:01.131892911 +0000 UTC
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "metric-spool" manifold worker started at 2019-10-18 22:59:01.131926901 +0000 UTC
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2019-10-18 22:59:01.132134214 +0000 UTC
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2019-10-18 22:59:01.132169028 +0000 UTC
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2019-10-18 22:59:01.134157118 +0000 UTC
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.XXX.XXX.226,127.0.0.1,192.168.100.243,::1,localhost", AutoNoProxy:""}
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.proxyupdater saving new legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.XXX.XXX.226,127.0.0.1,192.168.100.243,::1,localhost", AutoNoProxy:""}
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.proxyupdater new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "metric-sender" manifold worker started at 2019-10-18 22:59:01.143663895 +0000 UTC
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.dependency "uniter" manifold worker started at 2019-10-18 22:59:01.144553492 +0000 UTC
unit-elasticsearch-3: 22:59:01 INFO juju.worker.leadership elasticsearch/3 promoted to leadership of elasticsearch
unit-elasticsearch-3: 22:59:01 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=INFO;juju.provider.maas=TRACE;unit=DEBUG"
unit-elasticsearch-3: 22:59:01 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-elasticsearch-3
unit-elasticsearch-3: 22:59:01 INFO juju.agent.tools was a symlink, now looking at /var/lib/juju/tools/2.6.9-bionic-amd64
unit-elasticsearch-3: 22:59:01 INFO juju.worker.meterstatus skipped "meter-status-changed" hook (missing)
unit-elasticsearch-3: 22:59:01 INFO juju.worker.uniter unit "elasticsearch/3" started
unit-elasticsearch-3: 22:59:01 INFO juju.worker.uniter resuming charm install
unit-elasticsearch-3: 22:59:01 INFO juju.worker.uniter.charm downloading local:bionic/elasticsearch-3 from API server
unit-elasticsearch-3: 22:59:01 INFO juju.downloader downloading from local:bionic/elasticsearch-3
unit-elasticsearch-3: 22:59:01 INFO juju.downloader download complete ("local:bionic/elasticsearch-3")
unit-elasticsearch-3: 22:59:01 INFO juju.downloader download verified ("local:bionic/elasticsearch-3")
unit-elasticsearch-3: 22:59:02 INFO juju.worker.uniter hooks are retried true