VMware vSphere and Juju

List of supported clouds > VMware vSphere

This document describes details specific to using your existing VMware vSphere cloud with Juju.

See more: VMware vSphere

When using the VMware vSphere cloud with Juju, it is important to keep in mind that it is a (1) machine cloud and (2) not some other cloud.

See more: Cloud differences in Juju

As the differences related to (1) are already documented generically in our Tutorial, How-to guides, and Reference docs, here we record just those that follow from (2).

Juju points of variation Notes for the VMware vSphere cloud
setup (chronological order):
CLOUD
supported versions:
requirements: In order to add a vSphere cloud you will need to have an existing vSphere installation which supports, or has access to, the following:

- VMware Hardware Version 8 (or greater)

- ESXi 5.0 (or greater)

- Internet access

- DNS and DHCP

Juju supports both high-availability vSAN deployments as well and standard deployments.

definition:
- name: user-defined
- type: vsphere
- authentication types: [userpass]
- regions: [TO BE ADDED]
- cloud-specific model configuration keys: datastore
The datastore in which to create VMs. If this is not specified, the process will abort unless there is only one datastore available.

disk-provisioning-type
This dictates how template VM disks should be cloned when creating a new machine.
Valid values:

- thin - Sparse provisioning, only written blocks will take up disk space on the datastore

- thick - The entire size of the virtual disk will be deducted from the datastore, but unwritten blocks will not be zeroed out. This adds 2 potential pitfalls. See comments in provider/vsphere/internal/vsphereclient/client.go regarding DiskProvisioningType.

- thickEagerZero (default) - The entire size of the virtual disk is deducted from the datastore, and unwritten blocks are zeroed out. Improves latency when committing to disk, as no extra step needs to be taken before writing data.

external-network
An external network that VMs will be connected to. The resulting IP address for a VM will be used as its public address. An external network provides the interface to the internet for virtual machines connected to external organization vDC networks.

force-vm-hardware-version (integer)
Adds a new model level flag that allows operators to set a newer compatibility version for the instances that get spawned by juju. E.g., juju bootstrap vsphere --config force-vm-hardware-version=17

primary-network
The primary network that VMs will be connected to. If this is not specified, Juju will look for a network named VM Network.

CREDENTIAL
definition: auth-type: userpass. You will have to provide your username, password and, optionally, the vmfolder.

:warning: If your credential stops working: Credentials for the vSphere cloud have been reported to occasionally stop working over time. If this happens, try juju update-credential (passing as an argument the same credential) or juju add-credential (passing as an argument a new credential) + juju default-credential.

CONTROLLER
notes on bootstrap: Recommended: Bootstrap with the following cloud-specific model-configuration keys: datastore and primary-network. See more below.

Pro tip: When creating a controller with vSphere, a cloud image is downloaded to the client and then uploaded to the ESX host. This depends on your network connection and can take a while. Using templates can speed up bootstrap and machine deployment.

other (alphabetical order:)
CONSTRAINT
conflicting:
supported?
- allocate-public-ip :x:
- arch :white_check_mark:
Valid values: [amd64, i386].
- container :white_check_mark:
- cores :white_check_mark:
- cpu-power :white_check_mark:
- image-id :x:
- instance-role :x:
- instance-type :white_check_mark:
- mem :white_check_mark:
- root-disk :white_check_mark:
- root-disk-source :white_check_mark:
root-disk-source is the datastore for the root disk
- spaces :x:
- tags :x:
- virt-type :x:
- zones :white_check_mark:
PLACEMENT DIRECTIVE
<machine> TBA
subnet=... :x:
system-id=... :x:
zone=... :white_check_mark:
MACHINE
RESOURCE (cloud)

Consistent naming, tagging, and the ability to add user-controlled tags to created instances.

:x:

Other notes

Using templates

To speed up bootstrap and deploy, you can use VM templates, already created in your vSphere. Templates can be created by hand on your vSphere, or created from an existing VM.

Examples assume that the templates are in directory $DATA_STORE/templates.

Via simplestreams:

mkdir -p $HOME/simplestreams
juju-metadata generate-image -d $HOME/simplestreams/ -i "templates/juju-focal-template" -s focal -r $DATA_STORE -u $CLOUD_ENDPOINT
juju-metadata generate-image -d $HOME/simplestreams/ -i "templates/juju-bionic-template" -s bionic -r $DATA_STORE -u $CLOUD_ENDPOINT
juju bootstrap --metadata-source $HOME/image-streams vsphere

Bootstrap juju with the controller on a VM running focal:

juju bootstrap vsphere --bootstrap-image="templates/focal-test-template"  --bootstrap-series=focal --bootstrap-constraints "arch=amd64"

Using add-image:

juju metadata add-image templates/bionic-test-template --series bionic

Levels of placement

When creating a controller, there are three levels of placement that Juju understands: cloud, region, and availability zone. In vSphere, these are mapped in two different ways depending on your topology:

  • cloud (vSphere endpoint), region (Datacenter), availability zone (host)
  • cloud (vSphere endpoint), region (Datacenter), availability zone (cluster)

If your topology has a cluster without a host, Juju will see this as an Availability Zone and may fail silently. To solve this, either make sure the host is within the cluster, or use a placement directive:

juju bootstrap vsphere/<datacenter> <controllername> --to zone=<cluster|host>

To create a controller using Datacenter ‘dc1’ you would enter the following:

juju bootstrap vsp-cloud/dc1 vsp-controller

Specify a datastore when deploying an application

There is a constraint called ‘root-disk-source’ that can stipulate the name of a vSphere datastore to house the root disk:

juju deploy myapp --constraints root-disk-source=mydatastore

Deploy applications to a specific host or cluster

Resource pools within a host or cluster can be specified with the ‘zones’ constraint:

juju deploy myapp --constraints zones=myhost
juju deploy myapp --constraints zones=myfolder/myhost
juju deploy myapp --constraints zones=mycluster/mypool
juju deploy myapp --constraints zones=mycluster/myparent/mypool
1 Like

Question: "How do I add a “centos7” image to vsphere?

Question: “How do I make use of additional datastores apart from the one configured in the controller or model?”

Question: “Can juju handle NFS type datastores which exists in vsphere?” Very useful for providing NAS to juju units.

Question: “How do I make use of networks in vsphere?” - Is spaces available yet for vsphere?

What is a correct example for

juju deploy myapp --constraints zones=mycluster/mygroup
juju deploy myapp --constraints zones=mycluster/myparent/mygroup

Because I cannot get it to work

We have a mapping as:

(vsphere) datacenter = (juju) region

According to docs for vsphere, this mapping can be different depending on your vsphere configuration.

Hope it helps.

It would be nice to explain the minimum required privileges need in vSphere to bootstrap/deploy a juju environment to a vSphere cloud. Based on this commit [1] we know at least System.Read is required on the Datacenter entity. Clearly you need more than just read.

Thanks,
Nick

[1] https://github.com/juju/juju/commit/c9eeea11702d1172e5f68deff2c298b9199e8299#diff-4f35754aeeeeb70bd4f4e59b409336ab

Hey Nick

We are running juju with vsphere and have some experience.

The latest release of juju came with alot of changes to improve the situation with vsphere but I agree that it’s unclear just exactly how to setup any cloud for a proper juju situation. Lxd might be the best documented cloud in my opinion.

We have experimented our way forward and are not super happy about that.

Perhaps we missed something, but, yeah.

@rick_h

Erik,
When giving juju an admin account everything is fine, but in the spirit of security and least privileged access it would be good to know exactly what that least privilege is. We also have customers that have this requirement and would like to see this documentation.

Nick

1 Like

@nniehoff I feel with you.

I am not an expert in VMware, so I rely on some people at work to deal with these things which in our case also relates to Active directory I think.

But, the main issue for me at the moment is to be able to properly separate users views on provisioned vms etc. This is what seems to be partially solved/addressed/mitigated with later versions of juju.

The previous situation was that every user had a view of all vms for everyone since user access was enabled for the whole datacenter in vsphere.

There certainly is missing information for best practice. I particularly hate that we ask people to give Juju full admin privileges on AWS: https://discourse.jujucharms.com/t/using-amazon-aws-with-juju/1084

To address some of these concerns, I hope to expand the post below into a full guide. It sounds like guide should be supplemented for guides for each cloud.

1 Like

That IP address is real. Can you please change it to something else? e.g., 192.0.2.1 from RFC 5737.

Good call – updated.

The description assumes that datacenters are the top level resources. However, we saw some cases where a datacenter was under a folder, for example in LP: #1884490.

It would be nice if the doc covers such a scenario and the usecase is validated. I’m talking about the case govc returns the following.

$ govc ls
/MyFolder/MyDC/vm
/MyFolder/MyDC/network
/MyFolder/MyDC/host
/MyFolder/MyDC/datastore

and Juju config syntax would look like the following:

  regions:
   dc0: {}  # these empty maps
   dc1: {}  # are necessary
   MyFolder/MyDC: {} #  datacenter under a folder
juju bootstrap vsphere/<folder>/<datacenter> <controllername> --to zone=<cluster|host>

What would the “primary-network” typically look like from a IP point of view? 172.X.X.X or would it be something else?

Will juju automatically create VM:s with 2 NICs connected to these networks (primary-network + external-network) if they are available?

[Update #1] Yes, I see from experimentation that juju seem to spawn nodes with a second NIC if I set "juju config external-network=NETWORK_NAME_2" and attaches that to the running VM:s. So, I guess that the assignment if their IP:s now needs a second DHCP server also running on that network?

I didn’t know there was such a tool “govc” is this a supported tool or what can you tell me about it? It looks very interesting to use in conjunction with juju for vsphere cloud.

@jamesbeedy @hallback @heitor

I cant get this to work. I did the following:

DATA_STORE="datastore1"
CLOUD_ENDPOINT=10.230.65.170
mkdir -p $HOME/simplestreams
/home/ubuntu/internal_git/juju/_build/linux_amd64/bin/juju-metadata generate-image -d $HOME/simplestreams/ -i "cloud-images/ubuntu-20.04-server-cloudimg-amd64.ova" -s focal -r $DATA_STORE -u $CLOUD_ENDPOINT

/home/ubuntu/internal_git/juju/_build/linux_amd64/bin/juju --debug bootstrap vsphere --metadata-source $HOME/simplestreams --bootstrap-image="cloud-images/ubuntu-20.04-server-cloudimg-amd64.ova"  --bootstrap-series=focal --bootstrap-constraints "arch=amd64" --config datastore=$DATA_STORE

I’m getting the following error:

20:27:43 DEBUG juju.environs.simplestreams simplestreams.go:491 skipping index "file:///home/ubuntu/simplestreams/images/streams/v1/index.json" because of missing information: "image-downloads" data not found

It seems that Juju is still searching for ‘image-url’ in order to download the data, when it should not do any downloads. In the end, the bootstrap works but because it falls back to download from the internet.

[1] Full logs: https://bit.ly/3urzDe9

@sombrafam,

How did you create the template?

Juju does inject some things into the general templates when one is not specified. I’ve had the best experience copying a template created by juju to a folder which can be shared within the datastore.

Also double check the location specified in the $HOME/simplestreams/images/streams/v1/com.ubuntu.cloud-released-imagemetadata.json file. Each image has an id like "id": "juju-ci-root/templates/focal-test-template", that you should be able to find in the datastore. juju-ci-root is a directory in the datastore. I also use it as the vmfolder specified in my vsphere credentials.

A second thought is there is a conflict by specifying both --metadata-source $HOME/simplestreams --bootstrap-series=focal and --bootstrap-image="cloud-images/ubuntu-20.04-server-cloudimg-amd64.ova" at bootstrap. You’re giving juju seemingly 2 sources of truth. Please try again, leaving out the --bootstrap-image flag.