First steps with Charmed OSM

Thanks @davigar15

Your reply confirms my deduction of the issue. Thanks for suggestions, I will give them a try tomorrow. If you do find a way to alter the microstack dashboard port that would be appreciated, if I figure it out I’ll post back.

This is the first time I have used multipass. I had hoped the tutorial would provide a simple way to get all the way to a running OSM installation, including accessing the web gui. :slight_smile:

The edge release of microstack (or the use of the local_setting.d directory overrides) have gotten me past the ALLOWED_HOSTS issue for the moment.

Hello @aahamlin,

I have spoken with the main developer of microstack, and we sort out a list of steps that should solve the issue you’re having. When installing microstack, do this:

sudo snap install microstack --edge --classic
sudo snap set microstack config.network.ports=8080
sudo snap set microstack config.network.dashboard-allowed-hosts=*
sudo microstack.init --auto

Then try to access the openstack dashboard at http://<VM_IP>:8080
That should also unblock you from not being able to use the microk8s ingress, since the openstack dashboard will be running in another port.

PS: I will test those steps, but if you’re able to test it before me, please make sure you ping me how did it go.

Thanks!
Ping me if you need anything.

2 Likes

Hello!
I don’t see the setup_lxd.sh file. Can anyone please help to point me where I can find it?
Thanks!

Like Alex righty asked I can’t find the following files in the git repo:

setup_lxd.sh
install_nfs.sh
vca_config.sh

Also how can deploy this using a generic k8s cluster (i.e. if I don’t want to use microk8s)?

Those of you who ran into issues might want to check out the current set of instructions which have been updated by @davigar15, thanks David!

2 Likes

Hello,
I have been trying to follow this tutorial steps (OSM Development Stack). When I run: osmclient.overlay, I get this error: osmclient.overlay: command not found. So, I’ve decided to follow the official docs installation for OSM charmed (https://osm.etsi.org/docs/user-guide/03-installing-osm.html#charmed-installation). When I run: ./install_osm.sh --charmed, I get this error that I can’t solve:permanently dropping privs did not work: File exists
permanently dropping privs did not work: File exists
permanently dropping privs did not work: File exists
permanently dropping privs did not work: File exists
permanently dropping privs did not work: File exists
permanently dropping privs did not work: File exists
permanently dropping privs did not work: File exists
permanently dropping privs did not work: File exists
permanently dropping privs did not work: File exists
permanently dropping privs did not work: File exists
permanently dropping privs did not work: File exists
permanently dropping privs did not work: File exists
permanently dropping privs did not work: File exists
permanently dropping privs did not work: File exists
permanently dropping privs did not work: File exists
permanently dropping privs did not work: File exists
permanently dropping privs did not work: File exists
Microk8s storage failed to enable
permanently dropping privs did not work: File exists
I can’t find anything that works and I am in a dead end.
Can you please help me?

Thank you for your time

Hello @zalos56,

First of all, sorry for the late response. We’re now in the OSM Hackfest 10 and that got us distracted these days. Apologies, and I will help you out.

Regarding the osmclient.overlay command not found error, that is because that command no longer exists, but the installer script you executed is tested everyday, so let’s find out what’s going on :slight_smile:

Could you please let me know the specs of the VM/Machine in which you’re installing OSM? Could you also share the output of the microk8s.inspect command? (And the tarball generated)

1 Like

Hi @davigar15,
I had the exact same issue “Microk8s storage failed to enable
permanently dropping privs did not work: File exists”

I create a new VM with Ubuntu VERSION=“18.04.5 LTS (Bionic Beaver)” with the recommended resources setup, and executed the ‘./install_osm.sh --charmed’

Here goes the output and the inspect tar

      Service snap.microk8s.daemon-cluster-agent is running
      Service snap.microk8s.daemon-containerd is running
      Service snap.microk8s.daemon-apiserver is running
      Service snap.microk8s.daemon-apiserver-kicker is running
      Service snap.microk8s.daemon-control-plane-kicker is running
      Service snap.microk8s.daemon-proxy is running
      Service snap.microk8s.daemon-kubelet is running
      Service snap.microk8s.daemon-scheduler is running
      Service snap.microk8s.daemon-controller-manager is running
      Copy service arguments to the final report tarball
    Inspecting AppArmor configuration
    Gathering system information
      Copy processes list to the final report tarball
      Copy snap list to the final report tarball
      Copy VM name (or none) to the final report tarball
      Copy disk usage information to the final report tarball
      Copy memory usage information to the final report tarball
      Copy server uptime to the final report tarball
      Copy current linux distribution to the final report tarball
      Copy openSSL information to the final report tarball
      Copy network configuration to the final report tarball
    Inspecting kubernetes cluster
      Inspect kubernetes cluster
    Inspecting juju
      Inspect Juju
    Inspecting kubeflow
      Inspect Kubeflow

inspection-report-20210111_161018.tar.gz

Thanks!

Hello @joaocoucelo! I’ve been looking at tar you attached, and I cannot see what’s the issue. You have enough memory and disk, and I’m not seeing anything obvious.

@kos.tsakalozos Could you please take a look at the inspection report? I heard more than once the issue about “Microk8s storage failed to enable dropping privs did not work: File exists”; but cannot find the root cause of it.

@joaocoucelo the cluster looks healthy to me. It has pods scheduled and running. What happens if you run microk8s enable dns storage?

@davigar15 where can I find the source code that sets up MicroK8s?

Hi @kos.tsakalozos and @davigar15,

I was only able to repeat the installation now, and I no longer have that issue. Don’t know if I did something wrong during the last install, or if there was some change in the meanwhile. Anyway, I can see now the microk8s cluster running and the ‘juju status’ showing all all Apps and Units active, exposing the expected ports.

I did the suggesting config in Juju | The simplest way to deploy and maintain applications in the cloud, and I’m able to use osmclient.osm CLI locally.
However I’m not able to access any of the exposed services externally.

I can see the following ports exposed by ingress

$ microk8s kubectl describe ingress
grafana.192.168.55.2.xip.io
prometheus.192.168.55.2.xip.io
nbi.192.168.55.2.xip.io
ui.192.168.55.2.xip.io
osm.192.168.55.2.xip.io

I can reach these links from outside, but I get “404 Not Found” for all of them.

For instance, for the OSM Web GUI (ng-ui), all appears to be up and running, but the service is not replying.

$ microk8s kubectl describe ingress
Name: ng-ui
Namespace: osm
Address:
Default backend: default-http-backend:80 (<error: endpoints “default-http-backend” not found>)
Rules:
Host Path Backends


ui.192.168.55.2.xip.io
/ ng-ui:80 (10.1.120.246:80)
Annotations: ingress.kubernetes.io/rewrite-target:
ingress.kubernetes.io/ssl-passthrough: false
ingress.kubernetes.io/ssl-redirect: false
kubernetes.io/ingress.allow-http: false
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/proxy-body-size: 0
Events:

$ juju status | grep ng-ui
ng-ui active 1 ng-ui jujucharms 9 kubernetes 10.152.183.119 exposed
ng-ui/0* active idle 10.1.120.246 80/TCP ready

$ microk8s kubectl get all | grep ng-ui
pod/ng-ui-operator-0 1/1 Running 0 4h37m
pod/ng-ui-6d9c8ddd9c-sf6zm 1/1 Running 0 4h32m
service/ng-ui-operator ClusterIP 10.152.183.125 30666/TCP 4h37m
service/ng-ui ClusterIP 10.152.183.119 80/TCP 4h32m
deployment.apps/ng-ui 1/1 1 1 4h32m
replicaset.apps/ng-ui-6d9c8ddd9c 1 1 1 4h32m
statefulset.apps/ng-ui-operator 1/1 4h37m

$ netstat -an | grep 10.1.120.246
tcp 0 0 192.168.55.2:51494 10.1.120.246:80 TIME_WAIT
tcp 0 0 192.168.55.2:51204 10.1.120.246:80 TIME_WAIT
tcp 0 0 192.168.55.2:51200 10.1.120.246:80 TIME_WAIT

Any idea what I may be missing, or how can I troubleshoot further this issue?

2 Likes

Hello @davigar15,

This^ has been most helpful! :slight_smile:


I am setting up a charmed distribution of OSM along with MicroStack, i.e., ./install_osm.sh --charmed --microstack and although the whole process completed successfully, MicroStack has not been added as a VIM.

My system is running Ubuntu 18.04.

I grepped for errors and could only find:
Error while executing command: ConflictException: 409, Security group rule already exists. Rule id is 65339638-dfd1-4ca5-96a9-74c06acc2342.

Another problem that would lead to MicroStack not being added as a VIM is the OSM_HOSTNAME not working with the xip.io DNS service (which I believe is down).

Thus, due OSM_HOSTNAME being non resolvable, any osm-client command will have failed, as we can see here:

curl https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img | microstack.openstack image create --public --container-format=bare --disk-format=qcow2 ubuntu20.04
(6, 'Could not resolve host: nbi.VM_IP.xip.io')                                                                                                                            
Maybe "--hostname" option or OSM_HOSTNAME environment variable needs to be specified                                                                                                   
Your installation is now complete, follow these steps for configuring the osmclient:                                                                                                   
                                                                                                                                                                                       
1. Create the OSM_HOSTNAME environment variable with the NBI IP                                                                                                                        
                                                                                                                                                                                       
export OSM_HOSTNAME=nbi.VM_IP.xip.io:443                                                                                                                                        
                                                                                                                                                                                       
2. Add the previous command to your .bashrc for other Shell sessions

echo "export OSM_HOSTNAME=nbi.VM_IP.xip.io:443" >> ~/.bashrc

Questions

  1. Is this a known issue with xip.io?
  2. I searched for what the installation script does for a charmed + microstack installation and I can see from the osm-devops repository, here that it doesn’t use xip.io.
    Am I missing something with the installation script?
    I downloaded it via wget https://osm-download.etsi.org/ftp/osm-9.0-nine/install_osm.sh.

Thank you in advance! :slightly_smiling_face:

Hello @chris.asl!

Thanks for your message! Really glad to know that you are using Microstack :slight_smile:

Unfortunately, the xip.io Wildcard DNS is no longer available, and we moved to nip.io.

You can use the latest installer following these instructions:

wget http://osm-download.etsi.org/ftp/osm-9.0-nine/install_osm.sh
chmod +x install_osm.sh
./install_osm.sh -R testing-daily -r testing --charmed --microstack

I hope this message will unblock you. If you face any more issues, let me know :slight_smile:

Regards,
David

Thank you for the prompt reply @davigar15!

I’ve tried the command you’ve send me but got an error regarding MicroStack and the --auto flag.

juju status shows all services active, but sudo snap services microstack shows many of the core services, e.g. neutron-api disabled.

Here’s the error output from the installation script

Trying to install osmclient from channel 9.0/stable
osmclient (9.0/stable) v9.1.1-dirty from OSM Support (osmsupport) installed
osmclient snap installed
39002d9a-a987-434d-950d-cd66365df912
28b43380-8e1f-4f60-982c-561be6bfb461
Warning: flag --classic ignored for strictly confined snap microstack

microstack (beta) ussuri from Canonical✓ installed
Traceback (most recent call last):
  File "/snap/microstack/233/bin/microstack_init", line 11, in <module>
    load_entry_point('microstack-init==0.0.1', 'console_scripts', 'microstack_init')()
  File "/snap/microstack/233/lib/python3.8/site-packages/init/main.py", line 60, in wrapper
    return func(*args, **kwargs)
  File "/snap/microstack/233/lib/python3.8/site-packages/init/main.py", line 169, in init
    auto = process_init_args(args)
  File "/snap/microstack/233/lib/python3.8/site-packages/init/main.py", line 126, in process_init_args
    raise ValueError('A role (--compute or --control) must be specified '
ValueError: A role (--compute or --control) must be specified  when using --auto

Due to the output above, I believe microstack --init wasn’t run.


Additionally, I presume here are the commands the installation script runs to setup MicroStack networking and add it as a VIM.

I presume we can use these commands to setup an externally instantiated instance of MicroStack :slightly_smiling_face:

@davigar15 sorry for bugging you again, but I just figured out from the error above that the osm-devops package installed does not contain the updated version of the charmed_install.sh script :frowning_face:

It downloads this package
https://osm-download.etsi.org/repository/osm/debian/testing-daily/pool/devops/

which doesn’t contain the latest 9.0 version.

I’ve modified the release to use ReleaseNINE-daily which installs 9.1.1 that contains the updated charmed_install.sh script.

Am I missing something or it is simply a matter of releasing a new 9.0.0rc1 package which contains the latest code changes?

Thank you in advance :slightly_smiling_face:

I am also getting this error while triggering installation script with --charmed flag

cmd: ./install_osm.sh --charmed

error: permanently dropping privs did not work: File exists
permanently dropping privs did not work: File exists
Microk8s storage failed to enable
permanently dropping privs did not work: File exists

Please help !!!

I am stuck

Hello @rachit24,

Where are you installing OSM? Could you tell me please the OS you are using, the memory, disk, and cpus that you are allocating to the machine?

I have updated the instructions to deploy the latest release of OSM (Release TEN)

Please provide me the link where can i find the latest instructions.

Thanks for replying.

Please find the below details.

OS - Ubuntu 20.4 and 18.04
Memory- 32GB
disk- 150 GB
cpu-16