Custom OS images when using juju with VMware

Hello expert juju community!

I’m looking for a way to deploy custom OS images with VMware’s vSphere virtualization platform. We already have juju and VMware’s vSphere nicely talking to each other. The juju controller runs version 2.8.1 (my juju client is v2.8.5). juju spawns new virtualized machines using juju’s add-machine. We can even specify a different Ubuntu flavor using --series <ubuntu flavor name>, such as --series bionic. But how can we use our own cloud-based images? Ideally, we would we would like to deploy the image as an ISO or tar.gz file.

I’m looking for any advice into how we can find a solution. This is what I have tried, and it does not work unfortunately:

Setup a server hosting a web server to be used as a “juju stream” for the images (let’s say its IP is “ip1”). Here I just use nginx on port 80 with the extra config “autoindex on;” for “/” so that directory listening can be performed.

I then create the following directories:

mkdir -p /var/www/html/releases/streams/v1
mkdir -p /var/www/html/releases/releases/bionic/release-20201014

In “/var/www/html/releases/streams/v1” I have the following two files: index.json and com.ubuntu.cloud:released:download.json.

The content of index.json is the following:

{
    "index": {
        "com.ubuntu.cloud:released:download": {
            "datatype": "image-downloads", 
            "path": "streams/v1/com.ubuntu.cloud:released:download.json", 
            "updated": "Fri, 23 Oct 2020 03:06:29 +0000", 
            "products": [
                "com.ubuntu.cloud:server:18.04:amd64"
            ], 
            "format": "products:1.0"
        }
    },
    "updated": "Fri, 23 Oct 2020 11:08:44 +0000", 
    "format": "index:1.0"
}

The content of com.ubuntu.cloud:released:download.json is the following:

{                                  
    "content_id": "com.ubuntu.cloud:released:download",
    "datatype": "image-downloads",                                                                                                      
    "format": "products:1.0",                                                                                                           
    "license": "http://www.canonical.com/intellectual-property-policy",                                  
    "products": {                            
        "com.ubuntu.cloud:server:18.04:amd64": {
            "aliases": "ubuntu_test",
            "arch": "amd64",
            "os": "ubuntu",
            "release": "ubuntu_test",
            "release_codename": "Bionic Beaver",
            "release_title": "18.04 LTS",
            "support_eol": "2023-04-26",
            "supported": true,
            "version": "18.04",
            "versions": {
                "20180426.2": {
                    "items": {
                        "manifest": {
                            "ftype": "manifest",
                            "md5": "e85b6d6c3318fc06a27933c897164b4d",
                            "path": "releases/bionic/release-20201014/ubuntu-18.04-server-cloudimg-amd64.manifest",
                            "sha256": "c30ad8ba2d494f6c7ac22e136a453bec0154998b1ef6bb38ab2612ab233e449d",
                            "size": 15603
                        },
                        "disk1.img": {
                            "ftype": "disk1.img",
                            "md5": "9aa011b2b79b1fe42a7c306555923b1b",
                            "path": "releases/bionic/release-20201014/ubuntu-18.04-server-cloudimg-amd64.img",
                            "sha256": "9fdd8fa3091b8a40ea3f571d3461b246fe4e75fbd329b217076f804c9dda06a3",
                            "size": 359923712
                        },
                        "ova": {
                            "ftype": "ova",
                            "md5": "8f1a3ff03867d7db982798a3100ba342",
                            "path": "releases/bionic/release-20201014/ubuntu-18.04-server-cloudimg-amd64.ova",
                            "sha256": "0c9b26e9a98161f801fd15be8b6327f6300d95372fa01ed0a3307f923fee6343",
                            "size": 343183360
                        },
                        "root.manifest": {
                            "ftype": "root.manifest",
                            "md5": "9c1685e41e53a7c8cc8e63fc94595ba9",
                            "path": "releases/bionic/release-20201014/ubuntu-18.04-server-cloudimg-amd64-root.manifest",
                            "sha256": "12b58b09e60848b924d54079c58c5254f99e079b1e068a17b3ea60190cddd583",
                            "size": 14750
                        },
                        "root.tar.xz": {
                            "ftype": "root.tar.xz",
                            "md5": "60a18cd930796582a8b66e74725f84af",
                            "path": "releases/bionic/release-20201014/ubuntu-18.04-server-cloudimg-amd64-root.tar.xz",
                            "sha256": "ecfd3e8e12c0e675d0b9d8ac312d799ea2ce3b402bbb0b94e19dc4ff6ccf19f0",
                            "size": 158014756
                        },
                        "squashfs": {
                            "ftype": "squashfs",
                            "md5": "363a60f6cbfac12fb1fd3e7c25075d1d",
                            "path": "releases/bionic/release-20201014/ubuntu-18.04-server-cloudimg-amd64.squashfs",
                            "sha256": "7adaa11077ce8601ee049ea7364ecddbdd2bd3d66436ead4006b7877995b1460",
                            "size": 198164480
                        },
                        "squashfs.manifest": {
                            "ftype": "squashfs.manifest",
                            "md5": "9c1685e41e53a7c8cc8e63fc94595ba9",
                            "path": "releases/bionic/release-20201014/ubuntu-18.04-server-cloudimg-amd64.squashfs.manifest",
                            "sha256": "12b58b09e60848b924d54079c58c5254f99e079b1e068a17b3ea60190cddd583",
                            "size": 14750
                        },
                    "label": "release",                                                                                               
                    "pubname": "ubuntu-bionic-18.04-amd64-server-20180426.2"
                }                                                                                                                     
            }                                                                                                                         
        }                                                                                                                             
    },                                                                                                                                
    "updated": "Fri, 23 Oct 2020 03:06:29 +0000"                                                                                      
}

The above JSON is basically just copied from one of the release files from http://cloud-images.ubuntu.com/ with the difference that I have changed aliases and release keys to the value ubuntu_test. The rest of the metadata corresponds to Ubuntu bionic.

In /var/www/html/releases/releases/bionic/release-20201014 I have the following files downloaded from cloud-images.ubuntu.com:

total 1.3G
 16K ubuntu-18.04-server-cloudimg-amd64-root.manifest
151M ubuntu-18.04-server-cloudimg-amd64-root.tar.xz
344M ubuntu-18.04-server-cloudimg-amd64.img
 16K ubuntu-18.04-server-cloudimg-amd64.manifest
328M ubuntu-18.04-server-cloudimg-amd64.ova
189M ubuntu-18.04-server-cloudimg-amd64.squashfs
 16K ubuntu-18.04-server-cloudimg-amd64.squashfs.manifest
305M ubuntu-18.04-server-cloudimg-amd64.tar.gz

Then I configure the model (which uses the virtualized cloud), so that it should use the above stream server:

juju model-config image-metadata-url=http://ip1/releases

image-stream has the default value released.

I then test the setup:

juju add-machine --series ubuntu_test

When I type juju status juju has tried to add the new machine, but it says “down” and the message says ‘unknown OS for series: “ubuntu_test”’.

When I check the nginx log file (/var/log/nginx/access.log) on the stream server (ip1) I can see that the juju client has retrieved the index.json successfully:

IP_of_the_juju_controller - - [23/Oct/2020:17:26:09 +0000] "GET /releases/streams/v1/index.json HTTP/1.1" 200 402 "-" "Go-http-client/1.1"

This is basically where I’m not really sure how to proceed and I would really appreciate input from the juju masters out there.

Thanks in advance.

Oscar

1 Like

Very good writeup @oscarf - I hope we can get help understanding more on how to get the equation: “vsphere + juju + centos” to work.

Hi all,

Thank you for the detailed writeup on the subject. The Juju Core team has some work scheduled this coming cycle specifically to address custom VMDKs in vSphere. Progress tracked here: https://bugs.launchpad.net/juju/+bug/1867154

I’ve linked this discourse post to the bug, and welcome discussion here about interim workarounds, as well as more discussion about what Juju operators need from the feature.

2 Likes

Wohoo! @jamesbeedy this is a missing piece which would make life better for many of us stuck with vsphere.

1 Like

In the meantime while we wait for faster and more automatic ways to generate the metadata this can be helpful:

1 Like