RPC connection issues with pylibjuju

dgft · 14 September 2020 12:35

Hi all,

There a juju python script (launching backup action) that works until last night.
machine from where it is used is not a juju node.
it is a bionic, with pylibjuju version 2.8.3 installed.

The beginning of the script is:
controller = Controller() await controller.connect(endpoint=endpoint, username=username, password=password, cacert=cert) model = await controller.get_model(model_name)

That breaks on the get_model()

When launching the script manually the output (beside facades warning) is:

RPC: Connection closed, reconnecting
RPC: Connection closed, reconnecting
Receiver: Connection closed, reconnecting
RPC: Automatic reconnect failed

And script hangs there.

On controller side, in machine-0.log file we have:

2020-09-14 12:31:31 ERROR juju.rpc server.go:600 error writing response: *net.OpError write tcp 172.31.29.252:17070->172.31.46.101:59870: write: connection reset by peer
2020-09-14 12:31:31 ERROR juju.apiserver apiserver.go:939 error serving RPCs: codec.ReadHeader error: error receiving message: websocket: close 1009 (message too big)

Not trusting the actual vm, I created a new one, based on focal, pylibjuju version is 2.8.0 like juju controller version.
And applying the same script, result the same output error.

Another last point is I tried to adjust the max_frame_size in the controller.connect() with no much luck.

Any help to debug this appreciated.

Regards,
Dave

dgft · 15 September 2020 20:47

Hi all,

For people who would get the same problem.
It looks like our model is “big”.
I find that MAX_FRAME_SIZE has a default value of about 4Mb, define here https://github.com/juju/python-libjuju/blob/9556d3cec1938f199687640b336252b0d2d85c49/juju/client/connection.py#L213

Doubling that value, when connecting to the controller, make RPC connections happy again.

The only issue now would be, why our model is that big? and to clean it up to reduce the size?

Regards,
Dave

pengale · 15 September 2020 21:00

I wonder if this has something to do w/ bundling a bunch of additional facades in for the JavaScript client.

@wallyworld this shows up in 2.8.2, and didn’t show up before. Are we pushing a bunch of facades over the RPC connection in 2.8.2? Could that be the underlying issue?

dgft · 16 September 2020 08:46

This morning the RPC issue came back.
I need to add 49696 to the MAX_FRAME_SIZE to remove the RPC issue (at 8:44am utc), will try to see how much that is growing.
What would make the “model” growing?

@pengale to give you more version number, the pylibjuju and controller here are both v2.8.0.

Output of script starts with

unknown facade CAASAdmission
unexpected facade CAASAdmission found, unable to decipher version to use
unknown facade CAASModelOperator
unexpected facade CAASModelOperator found, unable to decipher version to use
unknown facade ImageMetadataManager
unexpected facade ImageMetadataManager found, unable to decipher version to use

Even if that’s known and harmless.

simonrichardson · 16 September 2020 11:09

I wonder if this has something to do w/ bundling a bunch of additional facades in for the JavaScript client.

The new changes to expose the facades for the JavaScript client where already exposed. The difference really was that we allow a type definition for those facades. Essentially we provided documentation and types to the functions/types. That way it always works with any revision of Juju, not just the latest ones.

The schema we produce is never passed over the wire, just used to generate libraries to interact with the API server.

Are we pushing a bunch of facades over the RPC connection in 2.8.2? Could that be the underlying issue?

When pylibjuju connects to a model, it does two things (it does other things, but we can safely ignore them) …

We watch for any changes and wait for the first “packet” of information. Which in turn can be all the data.
We get the model information.

So until I can reproduce this I’ve got a feeling it’s either one of those, but I’ll hedge a bet on number 1, but I’m unsure why atm.

dgft · 16 September 2020 11:17

Hi @simonrichardson,

You points (reading the code) sounds like a very potential options. As I already know that some “old”, “uncleaned” data, might be carried then back through that RPC call.
Any idea where should I debug what is returned? To look into it, I think crossing with log information I should be able to see if some data “should not be there”…

Thanks

simonrichardson · 16 September 2020 12:54

There are two things I would do; first work out how many deltas are received (additionally to that I’d probably work out how long between the watch and the wait call takes).

Secondly use the debug log to find out what ModelInfo is returned.

dgft · 16 September 2020 14:15

@simonrichardson

210 items (deltas)
0.5302503108978271s between watch and watch_received
dumping the deltas is about 16mb of toString() list item.
At list the last one is the output of running action, like last the percona percona script running backup action… there is a copy of the file via aws cli tool, which output a lot of similar file progress info. Which I will turn off…

After having a quick look into the deltas, the main content is text output of run-action or apt updates or so… the oldest seems to be from 2019-09, so I suppose that those deltas are “limited” overtime, it might be an option no to reduce even more?
After depending on “actions” I might be able to reduce verbosity, but mainly that sounds kind of normal…
What should be reduced then…

Could those items being clean up? or limited?

wallyworld · 16 September 2020 21:59

We need to figure out what’s bloating the model. One possible piece of the puzzle could be obtained by dumping the model as YAML

export JUJU_DEV_FEATURE_FLAGS=developer-mode
juju dump-model

There’s also juju dump-db which might highlight what’s blowing things up.

Don’t paste the raw output here as it may contain sensitive data, but you could provide some santised details or a summary of entity counts etc.

simonrichardson · 17 September 2020 08:59

@dgft any chance we can get a summary of the deltas?

If we could do something like this (see diff below), and if you could omit the sensitive data, before sending to us. If you do notice anything large, could you identify that if possible. That way we can see what’s going on…

diff --git a/juju/delta.py b/juju/delta.py
index 9849f9a..547f4c1 100644
--- a/juju/delta.py
+++ b/juju/delta.py
@@ -2,6 +2,7 @@ from .client import client


 def get_entity_delta(d):
+    print(d.entity, d.deltas)
     return _delta_types[d.entity](d.deltas)

dgft · 17 September 2020 09:04

@simonrichardson sure, did something similar yesterday.
I had 17Mb of dump.
I need to llok into those data.
As I said there is a lot of command output.
I will do my best (meaning in the shortest time) to upload some content.
Give me the day…
Btw the way, today the amount did not “grow”. But one script did not run, which has some output previously. So it sounds that the deltas are definitely containing some extra datas.

Be back as quick as possible.

Thanks

dgft · 17 September 2020 20:06

Hi,

A bit buzy today, but I might have some more useful data.
I dumped a file by delta.
So here the list of 210 deltas

 total 8928
drwxrwxr-x  2 ubuntu ubuntu   16384 Sep 17 19:56 ./
drwxr-xr-x 13 ubuntu ubuntu    4096 Sep 17 19:50 ../
-rw-rw-r--  1 ubuntu ubuntu    3159 Sep 17 19:56 output-delta-001-model.txt
-rw-rw-r--  1 ubuntu ubuntu    1014 Sep 17 19:56 output-delta-002-machine.txt
-rw-rw-r--  1 ubuntu ubuntu    1014 Sep 17 19:56 output-delta-003-machine.txt
-rw-rw-r--  1 ubuntu ubuntu    1014 Sep 17 19:56 output-delta-004-machine.txt
-rw-rw-r--  1 ubuntu ubuntu    1014 Sep 17 19:56 output-delta-005-machine.txt
-rw-rw-r--  1 ubuntu ubuntu    1013 Sep 17 19:56 output-delta-006-machine.txt
-rw-rw-r--  1 ubuntu ubuntu    1013 Sep 17 19:56 output-delta-007-machine.txt
-rw-rw-r--  1 ubuntu ubuntu    1013 Sep 17 19:56 output-delta-008-machine.txt
-rw-rw-r--  1 ubuntu ubuntu    1011 Sep 17 19:56 output-delta-009-machine.txt
-rw-rw-r--  1 ubuntu ubuntu    1011 Sep 17 19:56 output-delta-010-machine.txt
-rw-rw-r--  1 ubuntu ubuntu    1011 Sep 17 19:56 output-delta-011-machine.txt
-rw-rw-r--  1 ubuntu ubuntu    1011 Sep 17 19:56 output-delta-012-machine.txt
-rw-rw-r--  1 ubuntu ubuntu     386 Sep 17 19:56 output-delta-013-application.txt
-rw-rw-r--  1 ubuntu ubuntu     387 Sep 17 19:56 output-delta-014-application.txt
-rw-rw-r--  1 ubuntu ubuntu     676 Sep 17 19:56 output-delta-015-unit.txt
-rw-rw-r--  1 ubuntu ubuntu     610 Sep 17 19:56 output-delta-016-unit.txt
-rw-rw-r--  1 ubuntu ubuntu     598 Sep 17 19:56 output-delta-017-action.txt
-rw-rw-r--  1 ubuntu ubuntu     598 Sep 17 19:56 output-delta-018-action.txt
-rw-rw-r--  1 ubuntu ubuntu     598 Sep 17 19:56 output-delta-019-action.txt
-rw-rw-r--  1 ubuntu ubuntu     847 Sep 17 19:56 output-delta-020-action.txt
-rw-rw-r--  1 ubuntu ubuntu     847 Sep 17 19:56 output-delta-021-action.txt
-rw-rw-r--  1 ubuntu ubuntu     847 Sep 17 19:56 output-delta-022-action.txt
-rw-rw-r--  1 ubuntu ubuntu     604 Sep 17 19:56 output-delta-023-action.txt
-rw-rw-r--  1 ubuntu ubuntu     849 Sep 17 19:56 output-delta-024-action.txt
-rw-rw-r--  1 ubuntu ubuntu     458 Sep 17 19:56 output-delta-025-action.txt
-rw-rw-r--  1 ubuntu ubuntu     314 Sep 17 19:56 output-delta-026-action.txt
-rw-rw-r--  1 ubuntu ubuntu     314 Sep 17 19:56 output-delta-027-action.txt
-rw-rw-r--  1 ubuntu ubuntu     598 Sep 17 19:56 output-delta-028-action.txt
-rw-rw-r--  1 ubuntu ubuntu     573 Sep 17 19:56 output-delta-029-action.txt
-rw-rw-r--  1 ubuntu ubuntu     598 Sep 17 19:56 output-delta-030-action.txt
-rw-rw-r--  1 ubuntu ubuntu     573 Sep 17 19:56 output-delta-031-action.txt
-rw-rw-r--  1 ubuntu ubuntu     463 Sep 17 19:56 output-delta-032-action.txt
-rw-rw-r--  1 ubuntu ubuntu     441 Sep 17 19:56 output-delta-033-action.txt
-rw-rw-r--  1 ubuntu ubuntu     527 Sep 17 19:56 output-delta-034-action.txt
-rw-rw-r--  1 ubuntu ubuntu     463 Sep 17 19:56 output-delta-035-action.txt
-rw-rw-r--  1 ubuntu ubuntu    6341 Sep 17 19:56 output-delta-036-action.txt
-rw-rw-r--  1 ubuntu ubuntu     730 Sep 17 19:56 output-delta-037-action.txt
-rw-rw-r--  1 ubuntu ubuntu     730 Sep 17 19:56 output-delta-038-action.txt
-rw-rw-r--  1 ubuntu ubuntu     580 Sep 17 19:56 output-delta-039-action.txt
-rw-rw-r--  1 ubuntu ubuntu     580 Sep 17 19:56 output-delta-040-action.txt
-rw-rw-r--  1 ubuntu ubuntu    1927 Sep 17 19:56 output-delta-041-action.txt
-rw-rw-r--  1 ubuntu ubuntu    8238 Sep 17 19:56 output-delta-042-action.txt
-rw-rw-r--  1 ubuntu ubuntu    6098 Sep 17 19:56 output-delta-043-action.txt
-rw-rw-r--  1 ubuntu ubuntu    8235 Sep 17 19:56 output-delta-044-action.txt
-rw-rw-r--  1 ubuntu ubuntu    8235 Sep 17 19:56 output-delta-045-action.txt
-rw-rw-r--  1 ubuntu ubuntu    8235 Sep 17 19:56 output-delta-046-action.txt
-rw-rw-r--  1 ubuntu ubuntu    1927 Sep 17 19:56 output-delta-047-action.txt
-rw-rw-r--  1 ubuntu ubuntu    1923 Sep 17 19:56 output-delta-048-action.txt
-rw-rw-r--  1 ubuntu ubuntu    1924 Sep 17 19:56 output-delta-049-action.txt
-rw-rw-r--  1 ubuntu ubuntu    1924 Sep 17 19:56 output-delta-050-action.txt
-rw-rw-r--  1 ubuntu ubuntu    1924 Sep 17 19:56 output-delta-051-action.txt
-rw-rw-r--  1 ubuntu ubuntu     463 Sep 17 19:56 output-delta-052-action.txt
-rw-rw-r--  1 ubuntu ubuntu     402 Sep 17 19:56 output-delta-053-action.txt
-rw-rw-r--  1 ubuntu ubuntu   17952 Sep 17 19:56 output-delta-054-action.txt
-rw-rw-r--  1 ubuntu ubuntu     467 Sep 17 19:56 output-delta-055-action.txt
-rw-rw-r--  1 ubuntu ubuntu     557 Sep 17 19:56 output-delta-056-action.txt
-rw-rw-r--  1 ubuntu ubuntu     402 Sep 17 19:56 output-delta-057-action.txt
-rw-rw-r--  1 ubuntu ubuntu   17952 Sep 17 19:56 output-delta-058-action.txt
-rw-rw-r--  1 ubuntu ubuntu     467 Sep 17 19:56 output-delta-059-action.txt
-rw-rw-r--  1 ubuntu ubuntu     557 Sep 17 19:56 output-delta-060-action.txt
-rw-rw-r--  1 ubuntu ubuntu     402 Sep 17 19:56 output-delta-061-action.txt
-rw-rw-r--  1 ubuntu ubuntu   18207 Sep 17 19:56 output-delta-062-action.txt
-rw-rw-r--  1 ubuntu ubuntu     467 Sep 17 19:56 output-delta-063-action.txt
-rw-rw-r--  1 ubuntu ubuntu     557 Sep 17 19:56 output-delta-064-action.txt
-rw-rw-r--  1 ubuntu ubuntu     402 Sep 17 19:56 output-delta-065-action.txt
-rw-rw-r--  1 ubuntu ubuntu   18207 Sep 17 19:56 output-delta-066-action.txt
-rw-rw-r--  1 ubuntu ubuntu     467 Sep 17 19:56 output-delta-067-action.txt
-rw-rw-r--  1 ubuntu ubuntu     557 Sep 17 19:56 output-delta-068-action.txt
-rw-rw-r--  1 ubuntu ubuntu     402 Sep 17 19:56 output-delta-069-action.txt
-rw-rw-r--  1 ubuntu ubuntu   18207 Sep 17 19:56 output-delta-070-action.txt
-rw-rw-r--  1 ubuntu ubuntu     467 Sep 17 19:56 output-delta-071-action.txt
-rw-rw-r--  1 ubuntu ubuntu     557 Sep 17 19:56 output-delta-072-action.txt
-rw-rw-r--  1 ubuntu ubuntu     402 Sep 17 19:56 output-delta-073-action.txt
-rw-rw-r--  1 ubuntu ubuntu   18207 Sep 17 19:56 output-delta-074-action.txt
-rw-rw-r--  1 ubuntu ubuntu     467 Sep 17 19:56 output-delta-075-action.txt
-rw-rw-r--  1 ubuntu ubuntu     557 Sep 17 19:56 output-delta-076-action.txt
-rw-rw-r--  1 ubuntu ubuntu     402 Sep 17 19:56 output-delta-077-action.txt
-rw-rw-r--  1 ubuntu ubuntu   18207 Sep 17 19:56 output-delta-078-action.txt
-rw-rw-r--  1 ubuntu ubuntu     467 Sep 17 19:56 output-delta-079-action.txt
-rw-rw-r--  1 ubuntu ubuntu     690 Sep 17 19:56 output-delta-080-action.txt
-rw-rw-r--  1 ubuntu ubuntu     402 Sep 17 19:56 output-delta-081-action.txt
-rw-rw-r--  1 ubuntu ubuntu   18207 Sep 17 19:56 output-delta-082-action.txt
-rw-rw-r--  1 ubuntu ubuntu     467 Sep 17 19:56 output-delta-083-action.txt
-rw-rw-r--  1 ubuntu ubuntu     557 Sep 17 19:56 output-delta-084-action.txt
-rw-rw-r--  1 ubuntu ubuntu     402 Sep 17 19:56 output-delta-085-action.txt
-rw-rw-r--  1 ubuntu ubuntu   18207 Sep 17 19:56 output-delta-086-action.txt
-rw-rw-r--  1 ubuntu ubuntu     467 Sep 17 19:56 output-delta-087-action.txt
-rw-rw-r--  1 ubuntu ubuntu     557 Sep 17 19:56 output-delta-088-action.txt
-rw-rw-r--  1 ubuntu ubuntu 1385120 Sep 17 19:56 output-delta-089-action.txt
-rw-rw-r--  1 ubuntu ubuntu     428 Sep 17 19:56 output-delta-090-action.txt
-rw-rw-r--  1 ubuntu ubuntu     420 Sep 17 19:56 output-delta-091-action.txt
-rw-rw-r--  1 ubuntu ubuntu     402 Sep 17 19:56 output-delta-092-action.txt
-rw-rw-r--  1 ubuntu ubuntu   18207 Sep 17 19:56 output-delta-093-action.txt
-rw-rw-r--  1 ubuntu ubuntu     467 Sep 17 19:56 output-delta-094-action.txt
-rw-rw-r--  1 ubuntu ubuntu     557 Sep 17 19:56 output-delta-095-action.txt
-rw-rw-r--  1 ubuntu ubuntu 1401479 Sep 17 19:56 output-delta-096-action.txt
-rw-rw-r--  1 ubuntu ubuntu     428 Sep 17 19:56 output-delta-097-action.txt
-rw-rw-r--  1 ubuntu ubuntu     420 Sep 17 19:56 output-delta-098-action.txt
-rw-rw-r--  1 ubuntu ubuntu     598 Sep 17 19:56 output-delta-099-action.txt
-rw-rw-r--  1 ubuntu ubuntu     598 Sep 17 19:56 output-delta-100-action.txt
-rw-rw-r--  1 ubuntu ubuntu     573 Sep 17 19:56 output-delta-101-action.txt
-rw-rw-r--  1 ubuntu ubuntu     573 Sep 17 19:56 output-delta-102-action.txt
-rw-rw-r--  1 ubuntu ubuntu     598 Sep 17 19:56 output-delta-103-action.txt
-rw-rw-r--  1 ubuntu ubuntu     573 Sep 17 19:56 output-delta-104-action.txt
-rw-rw-r--  1 ubuntu ubuntu     462 Sep 17 19:56 output-delta-105-action.txt
-rw-rw-r--  1 ubuntu ubuntu    1787 Sep 17 19:56 output-delta-106-action.txt
-rw-rw-r--  1 ubuntu ubuntu     402 Sep 17 19:56 output-delta-107-action.txt
-rw-rw-r--  1 ubuntu ubuntu    1984 Sep 17 19:56 output-delta-108-action.txt
-rw-rw-r--  1 ubuntu ubuntu     467 Sep 17 19:56 output-delta-109-action.txt
-rw-rw-r--  1 ubuntu ubuntu     557 Sep 17 19:56 output-delta-110-action.txt
-rw-rw-r--  1 ubuntu ubuntu     706 Sep 17 19:56 output-delta-111-action.txt
-rw-rw-r--  1 ubuntu ubuntu     428 Sep 17 19:56 output-delta-112-action.txt
-rw-rw-r--  1 ubuntu ubuntu     420 Sep 17 19:56 output-delta-113-action.txt
-rw-rw-r--  1 ubuntu ubuntu    1984 Sep 17 19:56 output-delta-114-action.txt
-rw-rw-r--  1 ubuntu ubuntu   18083 Sep 17 19:56 output-delta-115-action.txt
-rw-rw-r--  1 ubuntu ubuntu     598 Sep 17 19:56 output-delta-116-action.txt
-rw-rw-r--  1 ubuntu ubuntu     573 Sep 17 19:56 output-delta-117-action.txt
-rw-rw-r--  1 ubuntu ubuntu     527 Sep 17 19:56 output-delta-118-action.txt
-rw-rw-r--  1 ubuntu ubuntu     462 Sep 17 19:56 output-delta-119-action.txt
-rw-rw-r--  1 ubuntu ubuntu    2150 Sep 17 19:56 output-delta-120-action.txt
-rw-rw-r--  1 ubuntu ubuntu    2150 Sep 17 19:56 output-delta-121-action.txt
-rw-rw-r--  1 ubuntu ubuntu     497 Sep 17 19:56 output-delta-122-action.txt
-rw-rw-r--  1 ubuntu ubuntu     462 Sep 17 19:56 output-delta-123-action.txt
-rw-rw-r--  1 ubuntu ubuntu     402 Sep 17 19:56 output-delta-124-action.txt
-rw-rw-r--  1 ubuntu ubuntu   16523 Sep 17 19:56 output-delta-125-action.txt
-rw-rw-r--  1 ubuntu ubuntu     467 Sep 17 19:56 output-delta-126-action.txt
-rw-rw-r--  1 ubuntu ubuntu     557 Sep 17 19:56 output-delta-127-action.txt
-rw-rw-r--  1 ubuntu ubuntu     432 Sep 17 19:56 output-delta-128-action.txt
-rw-rw-r--  1 ubuntu ubuntu 1213261 Sep 17 19:56 output-delta-129-action.txt
-rw-rw-r--  1 ubuntu ubuntu     428 Sep 17 19:56 output-delta-130-action.txt
-rw-rw-r--  1 ubuntu ubuntu     420 Sep 17 19:56 output-delta-131-action.txt
-rw-rw-r--  1 ubuntu ubuntu     617 Sep 17 19:56 output-delta-132-charm.txt
-rw-rw-r--  1 ubuntu ubuntu     288 Sep 17 19:56 output-delta-133-charm.txt
-rw-rw-r--  1 ubuntu ubuntu     217 Sep 17 19:56 output-delta-134-charm.txt
-rw-rw-r--  1 ubuntu ubuntu    1198 Sep 17 19:56 output-delta-135-charm.txt
-rw-rw-r--  1 ubuntu ubuntu    4704 Sep 17 19:56 output-delta-136-charm.txt
-rw-rw-r--  1 ubuntu ubuntu    4704 Sep 17 19:56 output-delta-137-charm.txt
-rw-rw-r--  1 ubuntu ubuntu    4704 Sep 17 19:56 output-delta-138-charm.txt
-rw-rw-r--  1 ubuntu ubuntu    4749 Sep 17 19:56 output-delta-139-charm.txt
-rw-rw-r--  1 ubuntu ubuntu    4749 Sep 17 19:56 output-delta-140-charm.txt
-rw-rw-r--  1 ubuntu ubuntu    4749 Sep 17 19:56 output-delta-141-charm.txt
-rw-rw-r--  1 ubuntu ubuntu    4749 Sep 17 19:56 output-delta-142-charm.txt
-rw-rw-r--  1 ubuntu ubuntu    4744 Sep 17 19:56 output-delta-143-charm.txt
-rw-rw-r--  1 ubuntu ubuntu     289 Sep 17 19:56 output-delta-144-relation.txt
-rw-rw-r--  1 ubuntu ubuntu     447 Sep 17 19:56 output-delta-145-relation.txt
-rw-rw-r--  1 ubuntu ubuntu     285 Sep 17 19:56 output-delta-146-relation.txt
-rw-rw-r--  1 ubuntu ubuntu     289 Sep 17 19:56 output-delta-147-relation.txt
-rw-rw-r--  1 ubuntu ubuntu     466 Sep 17 19:56 output-delta-148-relation.txt
-rw-rw-r--  1 ubuntu ubuntu     466 Sep 17 19:56 output-delta-149-relation.txt
-rw-rw-r--  1 ubuntu ubuntu     459 Sep 17 19:56 output-delta-150-relation.txt
-rw-rw-r--  1 ubuntu ubuntu     449 Sep 17 19:56 output-delta-151-relation.txt
-rw-rw-r--  1 ubuntu ubuntu     449 Sep 17 19:56 output-delta-152-relation.txt
-rw-rw-r--  1 ubuntu ubuntu     459 Sep 17 19:56 output-delta-153-relation.txt
-rw-rw-r--  1 ubuntu ubuntu     435 Sep 17 19:56 output-delta-154-application.txt
-rw-rw-r--  1 ubuntu ubuntu     669 Sep 17 19:56 output-delta-155-application.txt
-rw-rw-r--  1 ubuntu ubuntu     679 Sep 17 19:56 output-delta-156-unit.txt
-rw-rw-r--  1 ubuntu ubuntu     679 Sep 17 19:56 output-delta-157-unit.txt
-rw-rw-r--  1 ubuntu ubuntu     478 Sep 17 19:56 output-delta-158-application.txt
-rw-rw-r--  1 ubuntu ubuntu     700 Sep 17 19:56 output-delta-159-unit.txt
-rw-rw-r--  1 ubuntu ubuntu    2150 Sep 17 19:56 output-delta-160-action.txt
-rw-rw-r--  1 ubuntu ubuntu     490 Sep 17 19:56 output-delta-161-action.txt
-rw-rw-r--  1 ubuntu ubuntu    2150 Sep 17 19:56 output-delta-162-action.txt
-rw-rw-r--  1 ubuntu ubuntu    2150 Sep 17 19:56 output-delta-163-action.txt
-rw-rw-r--  1 ubuntu ubuntu    2150 Sep 17 19:56 output-delta-164-action.txt
-rw-rw-r--  1 ubuntu ubuntu    2150 Sep 17 19:56 output-delta-165-action.txt
-rw-rw-r--  1 ubuntu ubuntu    2150 Sep 17 19:56 output-delta-166-action.txt
-rw-rw-r--  1 ubuntu ubuntu    2150 Sep 17 19:56 output-delta-167-action.txt
-rw-rw-r--  1 ubuntu ubuntu    2150 Sep 17 19:56 output-delta-168-action.txt
-rw-rw-r--  1 ubuntu ubuntu     416 Sep 17 19:56 output-delta-169-action.txt
-rw-rw-r--  1 ubuntu ubuntu     413 Sep 17 19:56 output-delta-170-action.txt
-rw-rw-r--  1 ubuntu ubuntu     500 Sep 17 19:56 output-delta-171-action.txt
-rw-rw-r--  1 ubuntu ubuntu   18047 Sep 17 19:56 output-delta-172-action.txt
-rw-rw-r--  1 ubuntu ubuntu     478 Sep 17 19:56 output-delta-173-action.txt
-rw-rw-r--  1 ubuntu ubuntu     561 Sep 17 19:56 output-delta-174-action.txt
-rw-rw-r--  1 ubuntu ubuntu     591 Sep 17 19:56 output-delta-175-action.txt
-rw-rw-r--  1 ubuntu ubuntu     416 Sep 17 19:56 output-delta-176-action.txt
-rw-rw-r--  1 ubuntu ubuntu     413 Sep 17 19:56 output-delta-177-action.txt
-rw-rw-r--  1 ubuntu ubuntu     413 Sep 17 19:56 output-delta-178-action.txt
-rw-rw-r--  1 ubuntu ubuntu   18151 Sep 17 19:56 output-delta-179-action.txt
-rw-rw-r--  1 ubuntu ubuntu     478 Sep 17 19:56 output-delta-180-action.txt
-rw-rw-r--  1 ubuntu ubuntu     561 Sep 17 19:56 output-delta-181-action.txt
-rw-rw-r--  1 ubuntu ubuntu 1316125 Sep 17 19:56 output-delta-182-action.txt
-rw-rw-r--  1 ubuntu ubuntu     431 Sep 17 19:56 output-delta-183-action.txt
-rw-rw-r--  1 ubuntu ubuntu     431 Sep 17 19:56 output-delta-184-action.txt
-rw-rw-r--  1 ubuntu ubuntu     416 Sep 17 19:56 output-delta-185-action.txt
-rw-rw-r--  1 ubuntu ubuntu     413 Sep 17 19:56 output-delta-186-action.txt
-rw-rw-r--  1 ubuntu ubuntu     500 Sep 17 19:56 output-delta-187-action.txt
-rw-rw-r--  1 ubuntu ubuntu   18099 Sep 17 19:56 output-delta-188-action.txt
-rw-rw-r--  1 ubuntu ubuntu     478 Sep 17 19:56 output-delta-189-action.txt
-rw-rw-r--  1 ubuntu ubuntu     561 Sep 17 19:56 output-delta-190-action.txt
-rw-rw-r--  1 ubuntu ubuntu 1319429 Sep 17 19:56 output-delta-191-action.txt
-rw-rw-r--  1 ubuntu ubuntu     431 Sep 17 19:56 output-delta-192-action.txt
-rw-rw-r--  1 ubuntu ubuntu     431 Sep 17 19:56 output-delta-193-action.txt
-rw-rw-r--  1 ubuntu ubuntu     685 Sep 17 19:56 output-delta-194-unit.txt
-rw-rw-r--  1 ubuntu ubuntu     679 Sep 17 19:56 output-delta-195-unit.txt
-rw-rw-r--  1 ubuntu ubuntu     685 Sep 17 19:56 output-delta-196-unit.txt
-rw-rw-r--  1 ubuntu ubuntu     402 Sep 17 19:56 output-delta-197-action.txt
-rw-rw-r--  1 ubuntu ubuntu   17719 Sep 17 19:56 output-delta-198-action.txt
-rw-rw-r--  1 ubuntu ubuntu     467 Sep 17 19:56 output-delta-199-action.txt
-rw-rw-r--  1 ubuntu ubuntu     557 Sep 17 19:56 output-delta-200-action.txt
-rw-rw-r--  1 ubuntu ubuntu 1294460 Sep 17 19:56 output-delta-201-action.txt
-rw-rw-r--  1 ubuntu ubuntu     428 Sep 17 19:56 output-delta-202-action.txt
-rw-rw-r--  1 ubuntu ubuntu     420 Sep 17 19:56 output-delta-203-action.txt
-rw-rw-r--  1 ubuntu ubuntu     685 Sep 17 19:56 output-delta-204-unit.txt
-rw-rw-r--  1 ubuntu ubuntu     597 Sep 17 19:56 output-delta-205-unit.txt
-rw-rw-r--  1 ubuntu ubuntu    3246 Sep 17 19:56 output-delta-206-application.txt
-rw-rw-r--  1 ubuntu ubuntu     596 Sep 17 19:56 output-delta-207-unit.txt
-rw-rw-r--  1 ubuntu ubuntu     597 Sep 17 19:56 output-delta-208-unit.txt
-rw-rw-r--  1 ubuntu ubuntu    3880 Sep 17 19:56 output-delta-209-application.txt
-rw-rw-r--  1 ubuntu ubuntu     598 Sep 17 19:56 output-delta-210-unit.txt

As you can see some are very big
(129, 201,182,191,089,096)
Those are the result of a run-action backup + a copy to a aws s3 (cli)
eg inside you would have that kind of command output
emaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted

option --quiet has been apply to the source command.

All in all that sound related to that Excessive growth of delta
At least same symptom.

So I suppose those deltas would disappear overtime no?

A last thought, is that, it seems not a good idea to use a script (pylibjuju) to perform a recurrent task (such as backup), if that cumulate some data overtime that would reach some limit anyway at some point?

dgft · 18 September 2020 07:36

Hi @wallyworld,

Looking at the dump this morning, it is mainly the output of actions
Is there a way to clean those actions from the model?

I think run-action should not be used as recurrent task (cron) obviously.

simonrichardson · 18 September 2020 09:11

As we’ve diagnosed the juju run-action and juju run deltas will be streamed via the AllWatcher to the pylibjuju client. This is by design, the pylibjuju client uses the deltas to work out when an action has been performed.

The underlying issue is the size of the payloads and the fact that they’re exceeding the max_size of a websocket frame. Upping the frame size should allow you to work around the problem short term (RFC-6455 states that a single WebSocket frame can be 2^63 bytes, so you’ve got a lot to play with).

The long term fix for this is to get your charm to save the action output of a given item to a location (s3, local file system) and perform a juju scp (if using local file system) to get that data.

Advanced

If you’re really sure you don’t want that data from a given action and I would strongly advise you don’t run this on a production server.

For all controller machines, SSH into them and run:

systemctl stop jujud-machine-*

Get access to mongo:

Then run the following, it will leave the actions in mongo, but clean out the results.

db.actions.updateMany({}, {$set: {"results": {} }})

Once you have run the database updates, you can then do:

systemctl start jujud-machine-*

matuskosut · 18 September 2020 09:22

@dgft beside setting pretty high frame size, did you try tweaking model-config options?

max-action-results-age (default 336h)
max-action-results-size (default 5G)

dgft · 18 September 2020 09:32

Many thanks for this script (updateMany) , is it possible to do for some actions only?
like db.actions.updateMany({id: XXX}, ($set: {"results": {} }}) ?

probably just use updateOne ?

simonrichardson · 18 September 2020 09:50

I think this is the best solution.

dgft · 18 September 2020 09:52

@matuskosut many thanks for those.
I did not and I will look into those two.

simonrichardson · 18 September 2020 09:52

Correct, the best way is probably to dump out all your actions and audit each one to ensure you’re not breaking anything unexpected.

dgft · 18 September 2020 09:53

@simonrichardson

Just found that the following seems to work.

db.actions.updateOne({"_id": "<model_uuid>:<action_id>"}, ($set: {"results": {} }})