So, since yesterday I have this odd issue with the JAAS controller on AWS. I can deploy charms from charmhub, but I can not deploy local charms. I also can not use some commands like juju debug-log.
When I try to deploy local charms (this one is a subordinate), I get the following output with the --debug flag:
12:57:59 INFO juju.cmd supercommand.go:56 running juju [2.9.42 7b871e782195bdac9c90f8a8f01723cc3e08ab92 gc go1.18.10]
12:57:59 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/22345/bin/juju", "deploy", "./grafana-agent_ubuntu-22.04-amd64_ubuntu-20.04-amd64.charm", "--debug"}
12:57:59 INFO juju.juju api.go:86 connecting to API addresses: [jimm.jujucharms.com:443]
12:57:59 DEBUG juju.api apiclient.go:805 looked up jimm.jujucharms.com -> [162.213.33.88 162.213.33.244]
12:57:59 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://jimm.jujucharms.com:443/api"
12:57:59 INFO juju.api apiclient.go:687 connection established to "wss://jimm.jujucharms.com:443/api"
12:57:59 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://jimm.jujucharms.com:443/api"
12:57:59 INFO juju.juju api.go:86 connecting to API addresses: [jimm.jujucharms.com:443]
12:57:59 DEBUG juju.api apiclient.go:805 looked up jimm.jujucharms.com -> [162.213.33.88 162.213.33.244]
12:57:59 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://jimm.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/api"
12:57:59 INFO juju.api apiclient.go:687 connection established to "wss://jimm.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/api"
12:57:59 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://jimm.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/api"
12:58:00 DEBUG juju.api apiclient.go:805 looked up 1.eu-west-1.aws.jaas.jujucharms.com -> [54.194.14.188 52.212.33.191]
12:58:00 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://1.eu-west-1.aws.jaas.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/api"
12:58:00 INFO juju.api apiclient.go:687 connection established to "wss://1.eu-west-1.aws.jaas.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/api"
12:58:00 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://1.eu-west-1.aws.jaas.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/api"
12:58:00 DEBUG juju.core.charm computedseries.go:27 series "jammy, focal" for charm "grafana-agent" with format 1, Kubernetes false
12:58:00 DEBUG juju.core.charm computedseries.go:27 series "jammy, focal" for charm "grafana-agent" with format 1, Kubernetes false
12:58:00 INFO cmd charm.go:384 Preparing to deploy local charm: "grafana-agent_ubuntu-22.04-amd64_ubuntu-20.04-amd64.charm"
12:58:01 DEBUG juju.api monitor.go:35 RPC connection died
12:58:01 DEBUG juju.api monitor.go:35 RPC connection died
ERROR Post "https://1.eu-west-1.aws.jaas.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/charms?revision=0&schema=local&series=jammy": write tcp 10.0.144.195:54996->52.212.33.191:443: use of closed network connection
12:58:01 DEBUG cmd supercommand.go:537 error stack:
Post "https://1.eu-west-1.aws.jaas.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/charms?revision=0&schema=local&series=jammy": write tcp 10.0.144.195:54996->52.212.33.191:443: use of closed network connection
/build/snapcraft-juju-c599cfcc83a3daebe09e9bdb6bee70ec/parts/juju/build/vendor/github.com/go-macaroon-bakery/macaroon-bakery/v3/httpbakery/client.go:299:
/build/snapcraft-juju-c599cfcc83a3daebe09e9bdb6bee70ec/parts/juju/build/vendor/github.com/go-macaroon-bakery/macaroon-bakery/v3/httpbakery/client.go:284:
/build/snapcraft-juju-c599cfcc83a3daebe09e9bdb6bee70ec/parts/juju/build/vendor/gopkg.in/httprequest.v1/client.go:147:
github.com/juju/juju/api/client/charms.(*Client).httpPost:315:
github.com/juju/juju/api/client/charms.(*Client).uploadCharm:291:
github.com/juju/juju/api/client/charms.(*Client).AddLocalCharm:275:
github.com/juju/juju/cmd/juju/application/deployer.(*localCharm).PrepareAndDeploy:394:
Alternating with the following:
12:57:49 INFO juju.cmd supercommand.go:56 running juju [2.9.42 7b871e782195bdac9c90f8a8f01723cc3e08ab92 gc go1.18.10]
12:57:49 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/22345/bin/juju", "deploy", "./grafana-agent_ubuntu-22.04-amd64_ubuntu-20.04-amd64.charm", "--debug"}
12:57:49 INFO juju.juju api.go:86 connecting to API addresses: [jimm.jujucharms.com:443]
12:57:49 DEBUG juju.api apiclient.go:805 looked up jimm.jujucharms.com -> [162.213.33.88 162.213.33.244]
12:57:49 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://jimm.jujucharms.com:443/api"
12:57:49 INFO juju.api apiclient.go:687 connection established to "wss://jimm.jujucharms.com:443/api"
12:57:49 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://jimm.jujucharms.com:443/api"
12:57:49 INFO juju.juju api.go:86 connecting to API addresses: [jimm.jujucharms.com:443]
12:57:49 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://jimm.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/api"
12:57:49 INFO juju.api apiclient.go:1054 cannot resolve "jimm.jujucharms.com": lookup jimm.jujucharms.com: operation was canceled
12:57:49 INFO juju.api apiclient.go:687 connection established to "wss://jimm.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/api"
12:57:50 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://jimm.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/api"
12:57:50 DEBUG juju.api apiclient.go:805 looked up 1.eu-west-1.aws.jaas.jujucharms.com -> [52.212.33.191 54.194.14.188]
12:57:50 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://1.eu-west-1.aws.jaas.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/api"
12:57:50 INFO juju.api apiclient.go:687 connection established to "wss://1.eu-west-1.aws.jaas.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/api"
12:57:50 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://1.eu-west-1.aws.jaas.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/api"
12:57:50 DEBUG juju.core.charm computedseries.go:27 series "jammy, focal" for charm "grafana-agent" with format 1, Kubernetes false
12:57:50 DEBUG juju.core.charm computedseries.go:27 series "jammy, focal" for charm "grafana-agent" with format 1, Kubernetes false
12:57:50 INFO cmd charm.go:384 Preparing to deploy local charm: "grafana-agent_ubuntu-22.04-amd64_ubuntu-20.04-amd64.charm"
12:57:51 DEBUG juju.api monitor.go:35 RPC connection died
12:57:51 DEBUG juju.api monitor.go:35 RPC connection died
ERROR Post https://1.eu-west-1.aws.jaas.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/charms?revision=0&schema=local&series=jammy: unexpected content type text/html; want application/json; content: 400 Bad request; Your browser sent an invalid request
12:57:51 DEBUG cmd supercommand.go:537 error stack:
unexpected content type text/html; want application/json; content: 400 Bad request; Your browser sent an invalid request
github.com/juju/juju/api.unmarshalHTTPErrorResponse:139:
/build/snapcraft-juju-c599cfcc83a3daebe09e9bdb6bee70ec/parts/juju/build/vendor/gopkg.in/httprequest.v1/client.go:307: Post https://1.eu-west-1.aws.jaas.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/charms?revision=0&schema=local&series=jammy
/build/snapcraft-juju-c599cfcc83a3daebe09e9bdb6bee70ec/parts/juju/build/vendor/gopkg.in/httprequest.v1/client.go:185:
github.com/juju/juju/api/client/charms.(*Client).httpPost:315:
github.com/juju/juju/api/client/charms.(*Client).uploadCharm:291:
github.com/juju/juju/api/client/charms.(*Client).AddLocalCharm:275:
github.com/juju/juju/cmd/juju/application/deployer.(*localCharm).PrepareAndDeploy:394:
Debug-log gives the following:
13:26:12 INFO juju.cmd supercommand.go:56 running juju [2.9.42 7b871e782195bdac9c90f8a8f01723cc3e08ab92 gc go1.18.10]
13:26:12 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/22345/bin/juju", "debug-log", "--debug"}
13:26:12 INFO juju.juju api.go:86 connecting to API addresses: [jimm.jujucharms.com:443]
13:26:12 DEBUG juju.api apiclient.go:805 looked up jimm.jujucharms.com -> [162.213.33.88 162.213.33.244]
13:26:12 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://jimm.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/api"
13:26:12 INFO juju.api apiclient.go:687 connection established to "wss://jimm.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/api"
13:26:12 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://jimm.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/api"
13:26:13 DEBUG juju.api apiclient.go:805 looked up 1.eu-west-1.aws.jaas.jujucharms.com -> [54.194.14.188 52.212.33.191]
13:26:13 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://1.eu-west-1.aws.jaas.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/api"
13:26:13 INFO juju.api apiclient.go:687 connection established to "wss://1.eu-west-1.aws.jaas.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/api"
13:26:13 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://1.eu-west-1.aws.jaas.jujucharms.com:443/model/14dd8d16-c898-496c-8872-c5678d5c7e73/api"
13:26:13 DEBUG juju.api monitor.go:35 RPC connection died
ERROR <html><body><h1>400 Bad request</h1>
Your browser sent an invalid request.
</body></html> (Bad Request)
13:26:13 DEBUG cmd supercommand.go:537 error stack:
github.com/juju/juju/api.WebsocketDialWithErrors:65: <html><body><h1>400 Bad request</h1>
Your browser sent an invalid request.
</body></html> (Bad Request)
github.com/juju/juju/api.(*state).connectStreamWithRetry:411:
github.com/juju/juju/api.(*state).ConnectStream:372:
github.com/juju/juju/api/common.StreamDebugLog:114:
Understandably, I am a little confused. Exactly what is going on here?