AHA! - Finally I found what was missing from a pure Juju install vs a conjure-up install.
The br_netfilter
kernel module.
root@my-shell-5dbb49b954-rj8nd:/# dig google.com
;; reply from unexpected source: 10.1.20.10#53, expected 10.152.183.219#53
;; reply from unexpected source: 10.1.20.10#53, expected 10.152.183.219#53
lxc profile create netfilter
$ lxc profile set netfilter linux.kernel_modules br_netfilter
$ lxc profile show netfilter
name: netfilter
config:
linux.kernel_modules: br_netfilter
description: ""
devices: {}
lxc profile apply juju-91f454-3 default,juju-default,juju-default-kubernetes-master-754,netfilter
lxc profile apply juju-91f454-4 default,juju-default,juju-default-kubernetes-worker-590,netfilter
lxc restart juju-91f454-4
lxc restart juju-91f454-5
root@my-shell-5dbb49b954-mkdj5:/# dig google.com
; <<>> DiG 9.10.3-P4-Ubuntu <<>> google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56774
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;google.com. IN A
;; ANSWER SECTION:
google.com. 30 IN A 172.217.14.238
;; Query time: 18 msec
;; SERVER: 10.152.183.219#53(10.152.183.219)
;; WHEN: Tue Nov 12 23:32:45 UTC 2019
;; MSG SIZE rcvd: 65
And now DNS works. This is what was missing.