Looks like we started trusting Juju upgrades a bit too much and an upgrade to 2.7.0 broke vSphere support.
Due to a recent change, Juju now again contacts the vSphere hosts instead of the API to upload images. These hosts are on their own separate network not accessible to Juju. All communication should happen using the API. So now that we try to create a new VM, we get an error.
failed to start machine 47 (creating template VM: streaming http://cloud-images.ubuntu.com/releases/server/releases/xenial/release-20200108/ubuntu-16.04-server-cloudimg-amd64.ova to https://vnode1.test/nfc/521dec15-9a16-7856-f55f-8c92a328e84a/disk-0.vmdk: Post https://vnode1.test/nfc/521dec15-9a16-7856-f55f-8c92a328e84a/disk-0.vmdk: dial tcp 172.16.254.1:443: connect: no route to host), retrying in 10s (10 more attempts)
Another bug report from 2017 mentions the same issue saying that Juju should use the API instead of contacting the vSphere hosts. This was then fixed, however, a recent change somehow brought this behavior back. A quick look through the source code suggests the issue was brought back in this PR: https://github.com/juju/juju/pull/10461
@timClicks am I correct that that PR is the cause of our issue? Is there anything we can do to work around this issue?