Failing create controller backup

juju
, ,
#1

Since a few days ago our backups started to fail for one of our controllers.

$ juju create-backup --model=dwellir-sodertalje:admin/controller --filename=./test-backup.tar.gz
ERROR while creating backup archive: while dumping juju state database: rename /tmp/snap.juju-db/tmp/jujuBackup-1553460136/juju-backup/dump /tmp/jujuBackup-1553460136/juju-backup/dump: no such file or directory

Any idea what this is about?

#2

running with --debug --verbose --show-log

09:32:20 INFO  juju.cmd supercommand.go:56 running juju [3.0.0 35c560704ee254219ae0c4a37810bde5278e99bb gc go1.18.5]
09:32:20 DEBUG juju.cmd supercommand.go:57   args: []string{"/snap/juju/21122/bin/juju", "create-backup", "--model=dwellir-sodertalje:admin/controller", "--filename=./test-backup.tar.gz", "--debug", "--verbose", "--show-log"}
09:32:20 INFO  juju.juju api.go:86 connecting to API addresses: [192.168.111.64:17070 10.108.1.1:17070]
09:32:20 DEBUG juju.api apiclient.go:1168 successfully dialed "wss://192.168.111.64:17070/model/11ca14fd-f66a-43aa-8644-06f7f70de9a0/api"
09:32:20 INFO  juju.api apiclient.go:703 connection established to "wss://192.168.111.64:17070/model/11ca14fd-f66a-43aa-8644-06f7f70de9a0/api"
09:32:20 INFO  juju.juju api.go:340 API endpoints changed from [10.108.1.1:17070 192.168.111.64:17070] to [192.168.111.64:17070 10.108.1.1:17070]
09:32:54 DEBUG juju.api monitor.go:35 RPC connection died
ERROR while creating backup archive: while dumping juju state database: rename /tmp/snap.juju-db/tmp/jujuBackup-157646157/juju-backup/dump /tmp/jujuBackup-157646157/juju-backup/dump: no such file or directory
09:32:54 DEBUG cmd supercommand.go:537 error stack: 
while creating backup archive: while dumping juju state database: rename /tmp/snap.juju-db/tmp/jujuBackup-157646157/juju-backup/dump /tmp/jujuBackup-157646157/juju-backup/dump: no such file or directory
github.com/juju/juju/rpc.(*Conn).Call:178: 
github.com/juju/juju/api.(*state).APICall:1267: 
github.com/juju/juju/api/client/backups.(*Client).Create:23: 
github.com/juju/juju/cmd/juju/backups.(*createCommand).create:173: 
github.com/juju/juju/cmd/juju/backups.(*createCommand).Run:122:
#3

@hmlanigan do you have a clue whats going on here?

#4

@wallyworld We have this error above which seems similar to Bug #1998989 “create-backup not working with default backup-dir” : Bugs : juju

However, we don’t get the same error from another controller on the same version of the juju-db snap on version:

juju-db 4.0.9 29 4.0/stable juju-qa -

E.g. one controller makes it through the backup, the other wont.

We have been running without backup for weeks now and we feel worried about the situation. What can we do to get our backups going again?

@joakimnyman

#5

I’ve been on EOY holidays. Been too long since I was in the backup code to have a quick answer.

1 Like
#6

I think there was some known CVE/bug which possibly is fixed in the next release. But its strange since it still works on another controller on the same version.

#7

Quick answer as I’m on leave for another couple of weeks and in and out of phone range - the error looks the same as what the linked bug fixes. As per the bug description, you can work around it by setting the backup-dir config. Or you can wait till juju 2.9.38 is released which is where the bug fix is targetted to.

1 Like