Continuous 40-70 Mb/s traffic from controller to vcenter

merlijn-sebrechts · 19 September 2018 09:49

Hi all

We’re running a VMware vsphere cloud setup with about 200 virtual machines spread over about 20 models with 20 Juju accounts and 3 VMWare credentials. We see a constant traffic of 40-70Mb/s from our controller to vcenter (vmware cluster admin service). The traffic seems to be a constant stream of login > authorizemanager > retrievecontent > logout actions for every credential on the controller.

Any idea what might be going wrong here and how we can solve this? This is putting an incredible strain on the controller and VCenter.

Below are some of the logs. Let me know if I can help you with more information.

Logged traffic on VCenter

Juju controller logs excerpt

2018-09-19 09:47:36 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:36 ERROR juju.worker.dependency engine.go:587 "charm-revision-updater" manifold worker returned unexpected error: setting the resource failed
2018-09-19 09:47:37 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:37 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:37 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:37 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:38 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:39 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:39 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:40 ERROR juju.worker.dependency engine.go:587 "charm-revision-updater" manifold worker returned unexpected error: setting the resource failed
2018-09-19 09:47:40 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:41 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:42 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:42 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:42 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:43 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:43 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:43 ERROR juju.worker.dependency engine.go:587 "charm-revision-updater" manifold worker returned unexpected error: setting the resource failed
2018-09-19 09:47:44 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:45 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:45 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:46 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:46 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:46 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:47 ERROR juju.worker.dependency engine.go:587 "charm-revision-updater" manifold worker returned unexpected error: setting the resource failed
2018-09-19 09:47:47 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:47 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:48 ERROR juju.worker.remoterelations remoteapplicationworker.go:73 error in remote application worker for deve: watching status for offer: application offer "4a57fa88-bfb9-4fd7-893a-219cc8e34899" not found
2018-09-19 09:47:48 ERROR juju.worker.remoterelations remoteapplicationworker.go:73 error in remote application worker for hyperion-depl: watching status for offer: application offer "06426a90-3ca4-4dfd-81bc-6d97152778c5" not found
2018-09-19 09:47:48 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:48 ERROR juju.worker.remoterelations remoteapplicationworker.go:73 error in remote application worker for devdeploy: watching status for offer: application offer "a30e50a1-1b53-4f05-8794-1e62b3e95ab1" not found
2018-09-19 09:47:49 ERROR juju.worker.remoterelations remoteapplicationworker.go:73 error in remote application worker for nginx-api-gateway: watching status for offer: application offer "27be625f-cbb7-4b29-8b64-93b57f311f1b" not found
2018-09-19 09:47:49 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:49 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:50 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:50 ERROR juju.worker.dependency engine.go:587 "charm-revision-updater" manifold worker returned unexpected error: setting the resource failed
2018-09-19 09:47:51 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:52 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:52 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:53 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:53 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:53 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:54 ERROR juju.worker.dependency engine.go:587 "charm-revision-updater" manifold worker returned unexpected error: setting the resource failed
2018-09-19 09:47:54 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports: 
2018-09-19 09:47:54 ERROR juju.worker.dependency engine.go:587 "firewaller" manifold worker returned unexpected error: failed to list open ports:

Although there are a lot of firewaller lines, there seems to be much more vsphere calls/s than firewall errors/s, so I don’t think this is the same issue.

jameinel · 19 September 2018 14:35

This definitely feels like something that should just be filed as a bug and attach logs, etc, there.

Its entirely possible that there are several steps being executed for every firewaller error, and that is causing us to create far too many requests.

Note that VSphere doesn’t use a central firewall anyway. We do firewalling using iptables on each instance, so we shouldn’t be trying to talk to vsphere.
Now what might be happening is that we are thinking about SSHing into a given instance, and we are querying VSphere to look for an IP address for a machine, etc, and then we fail at some earlier time.