Hi @simonrichardson,
You points (reading the code) sounds like a very potential options. As I already know that some “old”, “uncleaned” data, might be carried then back through that RPC call.
Any idea where should I debug what is returned? To look into it, I think crossing with log information I should be able to see if some data “should not be there”…
Thanks
There are two things I would do; first work out how many deltas are received (additionally to that I’d probably work out how long between the watch and the wait call takes).
Secondly use the debug log to find out what ModelInfo is returned.
210 items (deltas)
0.5302503108978271s between watch and watch_received
dumping the deltas is about 16mb of toString() list item.
At list the last one is the output of running action, like last the percona percona script running backup action… there is a copy of the file via aws cli tool, which output a lot of similar file progress info. Which I will turn off…
After having a quick look into the deltas, the main content is text output of run-action or apt updates or so… the oldest seems to be from 2019-09, so I suppose that those deltas are “limited” overtime, it might be an option no to reduce even more?
After depending on “actions” I might be able to reduce verbosity, but mainly that sounds kind of normal…
What should be reduced then…
Could those items being clean up? or limited?
We need to figure out what’s bloating the model. One possible piece of the puzzle could be obtained by dumping the model as YAML
export JUJU_DEV_FEATURE_FLAGS=developer-mode
juju dump-model
There’s also juju dump-db which might highlight what’s blowing things up.
Don’t paste the raw output here as it may contain sensitive data, but you could provide some santised details or a summary of entity counts etc.
@dgft any chance we can get a summary of the deltas?
If we could do something like this (see diff below), and if you could omit the sensitive data, before sending to us. If you do notice anything large, could you identify that if possible. That way we can see what’s going on…
diff --git a/juju/delta.py b/juju/delta.py
index 9849f9a..547f4c1 100644
--- a/juju/delta.py
+++ b/juju/delta.py
@@ -2,6 +2,7 @@ from .client import client
def get_entity_delta(d):
+ print(d.entity, d.deltas)
return _delta_types[d.entity](d.deltas)
@simonrichardson sure, did something similar yesterday.
I had 17Mb of dump.
I need to llok into those data.
As I said there is a lot of command output.
I will do my best (meaning in the shortest time) to upload some content.
Give me the day…
Btw the way, today the amount did not “grow”. But one script did not run, which has some output previously. So it sounds that the deltas are definitely containing some extra datas.
Be back as quick as possible.
Thanks
Hi,
A bit buzy today, but I might have some more useful data.
I dumped a file by delta.
So here the list of 210 deltas
total 8928
drwxrwxr-x 2 ubuntu ubuntu 16384 Sep 17 19:56 ./
drwxr-xr-x 13 ubuntu ubuntu 4096 Sep 17 19:50 ../
-rw-rw-r-- 1 ubuntu ubuntu 3159 Sep 17 19:56 output-delta-001-model.txt
-rw-rw-r-- 1 ubuntu ubuntu 1014 Sep 17 19:56 output-delta-002-machine.txt
-rw-rw-r-- 1 ubuntu ubuntu 1014 Sep 17 19:56 output-delta-003-machine.txt
-rw-rw-r-- 1 ubuntu ubuntu 1014 Sep 17 19:56 output-delta-004-machine.txt
-rw-rw-r-- 1 ubuntu ubuntu 1014 Sep 17 19:56 output-delta-005-machine.txt
-rw-rw-r-- 1 ubuntu ubuntu 1013 Sep 17 19:56 output-delta-006-machine.txt
-rw-rw-r-- 1 ubuntu ubuntu 1013 Sep 17 19:56 output-delta-007-machine.txt
-rw-rw-r-- 1 ubuntu ubuntu 1013 Sep 17 19:56 output-delta-008-machine.txt
-rw-rw-r-- 1 ubuntu ubuntu 1011 Sep 17 19:56 output-delta-009-machine.txt
-rw-rw-r-- 1 ubuntu ubuntu 1011 Sep 17 19:56 output-delta-010-machine.txt
-rw-rw-r-- 1 ubuntu ubuntu 1011 Sep 17 19:56 output-delta-011-machine.txt
-rw-rw-r-- 1 ubuntu ubuntu 1011 Sep 17 19:56 output-delta-012-machine.txt
-rw-rw-r-- 1 ubuntu ubuntu 386 Sep 17 19:56 output-delta-013-application.txt
-rw-rw-r-- 1 ubuntu ubuntu 387 Sep 17 19:56 output-delta-014-application.txt
-rw-rw-r-- 1 ubuntu ubuntu 676 Sep 17 19:56 output-delta-015-unit.txt
-rw-rw-r-- 1 ubuntu ubuntu 610 Sep 17 19:56 output-delta-016-unit.txt
-rw-rw-r-- 1 ubuntu ubuntu 598 Sep 17 19:56 output-delta-017-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 598 Sep 17 19:56 output-delta-018-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 598 Sep 17 19:56 output-delta-019-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 847 Sep 17 19:56 output-delta-020-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 847 Sep 17 19:56 output-delta-021-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 847 Sep 17 19:56 output-delta-022-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 604 Sep 17 19:56 output-delta-023-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 849 Sep 17 19:56 output-delta-024-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 458 Sep 17 19:56 output-delta-025-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 314 Sep 17 19:56 output-delta-026-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 314 Sep 17 19:56 output-delta-027-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 598 Sep 17 19:56 output-delta-028-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 573 Sep 17 19:56 output-delta-029-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 598 Sep 17 19:56 output-delta-030-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 573 Sep 17 19:56 output-delta-031-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 463 Sep 17 19:56 output-delta-032-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 441 Sep 17 19:56 output-delta-033-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 527 Sep 17 19:56 output-delta-034-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 463 Sep 17 19:56 output-delta-035-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 6341 Sep 17 19:56 output-delta-036-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 730 Sep 17 19:56 output-delta-037-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 730 Sep 17 19:56 output-delta-038-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 580 Sep 17 19:56 output-delta-039-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 580 Sep 17 19:56 output-delta-040-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 1927 Sep 17 19:56 output-delta-041-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 8238 Sep 17 19:56 output-delta-042-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 6098 Sep 17 19:56 output-delta-043-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 8235 Sep 17 19:56 output-delta-044-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 8235 Sep 17 19:56 output-delta-045-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 8235 Sep 17 19:56 output-delta-046-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 1927 Sep 17 19:56 output-delta-047-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 1923 Sep 17 19:56 output-delta-048-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 1924 Sep 17 19:56 output-delta-049-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 1924 Sep 17 19:56 output-delta-050-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 1924 Sep 17 19:56 output-delta-051-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 463 Sep 17 19:56 output-delta-052-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 402 Sep 17 19:56 output-delta-053-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 17952 Sep 17 19:56 output-delta-054-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 467 Sep 17 19:56 output-delta-055-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 557 Sep 17 19:56 output-delta-056-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 402 Sep 17 19:56 output-delta-057-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 17952 Sep 17 19:56 output-delta-058-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 467 Sep 17 19:56 output-delta-059-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 557 Sep 17 19:56 output-delta-060-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 402 Sep 17 19:56 output-delta-061-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 18207 Sep 17 19:56 output-delta-062-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 467 Sep 17 19:56 output-delta-063-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 557 Sep 17 19:56 output-delta-064-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 402 Sep 17 19:56 output-delta-065-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 18207 Sep 17 19:56 output-delta-066-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 467 Sep 17 19:56 output-delta-067-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 557 Sep 17 19:56 output-delta-068-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 402 Sep 17 19:56 output-delta-069-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 18207 Sep 17 19:56 output-delta-070-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 467 Sep 17 19:56 output-delta-071-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 557 Sep 17 19:56 output-delta-072-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 402 Sep 17 19:56 output-delta-073-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 18207 Sep 17 19:56 output-delta-074-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 467 Sep 17 19:56 output-delta-075-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 557 Sep 17 19:56 output-delta-076-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 402 Sep 17 19:56 output-delta-077-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 18207 Sep 17 19:56 output-delta-078-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 467 Sep 17 19:56 output-delta-079-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 690 Sep 17 19:56 output-delta-080-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 402 Sep 17 19:56 output-delta-081-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 18207 Sep 17 19:56 output-delta-082-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 467 Sep 17 19:56 output-delta-083-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 557 Sep 17 19:56 output-delta-084-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 402 Sep 17 19:56 output-delta-085-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 18207 Sep 17 19:56 output-delta-086-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 467 Sep 17 19:56 output-delta-087-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 557 Sep 17 19:56 output-delta-088-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 1385120 Sep 17 19:56 output-delta-089-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 428 Sep 17 19:56 output-delta-090-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 420 Sep 17 19:56 output-delta-091-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 402 Sep 17 19:56 output-delta-092-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 18207 Sep 17 19:56 output-delta-093-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 467 Sep 17 19:56 output-delta-094-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 557 Sep 17 19:56 output-delta-095-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 1401479 Sep 17 19:56 output-delta-096-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 428 Sep 17 19:56 output-delta-097-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 420 Sep 17 19:56 output-delta-098-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 598 Sep 17 19:56 output-delta-099-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 598 Sep 17 19:56 output-delta-100-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 573 Sep 17 19:56 output-delta-101-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 573 Sep 17 19:56 output-delta-102-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 598 Sep 17 19:56 output-delta-103-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 573 Sep 17 19:56 output-delta-104-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 462 Sep 17 19:56 output-delta-105-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 1787 Sep 17 19:56 output-delta-106-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 402 Sep 17 19:56 output-delta-107-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 1984 Sep 17 19:56 output-delta-108-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 467 Sep 17 19:56 output-delta-109-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 557 Sep 17 19:56 output-delta-110-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 706 Sep 17 19:56 output-delta-111-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 428 Sep 17 19:56 output-delta-112-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 420 Sep 17 19:56 output-delta-113-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 1984 Sep 17 19:56 output-delta-114-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 18083 Sep 17 19:56 output-delta-115-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 598 Sep 17 19:56 output-delta-116-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 573 Sep 17 19:56 output-delta-117-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 527 Sep 17 19:56 output-delta-118-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 462 Sep 17 19:56 output-delta-119-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 2150 Sep 17 19:56 output-delta-120-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 2150 Sep 17 19:56 output-delta-121-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 497 Sep 17 19:56 output-delta-122-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 462 Sep 17 19:56 output-delta-123-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 402 Sep 17 19:56 output-delta-124-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 16523 Sep 17 19:56 output-delta-125-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 467 Sep 17 19:56 output-delta-126-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 557 Sep 17 19:56 output-delta-127-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 432 Sep 17 19:56 output-delta-128-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 1213261 Sep 17 19:56 output-delta-129-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 428 Sep 17 19:56 output-delta-130-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 420 Sep 17 19:56 output-delta-131-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 617 Sep 17 19:56 output-delta-132-charm.txt
-rw-rw-r-- 1 ubuntu ubuntu 288 Sep 17 19:56 output-delta-133-charm.txt
-rw-rw-r-- 1 ubuntu ubuntu 217 Sep 17 19:56 output-delta-134-charm.txt
-rw-rw-r-- 1 ubuntu ubuntu 1198 Sep 17 19:56 output-delta-135-charm.txt
-rw-rw-r-- 1 ubuntu ubuntu 4704 Sep 17 19:56 output-delta-136-charm.txt
-rw-rw-r-- 1 ubuntu ubuntu 4704 Sep 17 19:56 output-delta-137-charm.txt
-rw-rw-r-- 1 ubuntu ubuntu 4704 Sep 17 19:56 output-delta-138-charm.txt
-rw-rw-r-- 1 ubuntu ubuntu 4749 Sep 17 19:56 output-delta-139-charm.txt
-rw-rw-r-- 1 ubuntu ubuntu 4749 Sep 17 19:56 output-delta-140-charm.txt
-rw-rw-r-- 1 ubuntu ubuntu 4749 Sep 17 19:56 output-delta-141-charm.txt
-rw-rw-r-- 1 ubuntu ubuntu 4749 Sep 17 19:56 output-delta-142-charm.txt
-rw-rw-r-- 1 ubuntu ubuntu 4744 Sep 17 19:56 output-delta-143-charm.txt
-rw-rw-r-- 1 ubuntu ubuntu 289 Sep 17 19:56 output-delta-144-relation.txt
-rw-rw-r-- 1 ubuntu ubuntu 447 Sep 17 19:56 output-delta-145-relation.txt
-rw-rw-r-- 1 ubuntu ubuntu 285 Sep 17 19:56 output-delta-146-relation.txt
-rw-rw-r-- 1 ubuntu ubuntu 289 Sep 17 19:56 output-delta-147-relation.txt
-rw-rw-r-- 1 ubuntu ubuntu 466 Sep 17 19:56 output-delta-148-relation.txt
-rw-rw-r-- 1 ubuntu ubuntu 466 Sep 17 19:56 output-delta-149-relation.txt
-rw-rw-r-- 1 ubuntu ubuntu 459 Sep 17 19:56 output-delta-150-relation.txt
-rw-rw-r-- 1 ubuntu ubuntu 449 Sep 17 19:56 output-delta-151-relation.txt
-rw-rw-r-- 1 ubuntu ubuntu 449 Sep 17 19:56 output-delta-152-relation.txt
-rw-rw-r-- 1 ubuntu ubuntu 459 Sep 17 19:56 output-delta-153-relation.txt
-rw-rw-r-- 1 ubuntu ubuntu 435 Sep 17 19:56 output-delta-154-application.txt
-rw-rw-r-- 1 ubuntu ubuntu 669 Sep 17 19:56 output-delta-155-application.txt
-rw-rw-r-- 1 ubuntu ubuntu 679 Sep 17 19:56 output-delta-156-unit.txt
-rw-rw-r-- 1 ubuntu ubuntu 679 Sep 17 19:56 output-delta-157-unit.txt
-rw-rw-r-- 1 ubuntu ubuntu 478 Sep 17 19:56 output-delta-158-application.txt
-rw-rw-r-- 1 ubuntu ubuntu 700 Sep 17 19:56 output-delta-159-unit.txt
-rw-rw-r-- 1 ubuntu ubuntu 2150 Sep 17 19:56 output-delta-160-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 490 Sep 17 19:56 output-delta-161-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 2150 Sep 17 19:56 output-delta-162-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 2150 Sep 17 19:56 output-delta-163-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 2150 Sep 17 19:56 output-delta-164-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 2150 Sep 17 19:56 output-delta-165-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 2150 Sep 17 19:56 output-delta-166-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 2150 Sep 17 19:56 output-delta-167-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 2150 Sep 17 19:56 output-delta-168-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 416 Sep 17 19:56 output-delta-169-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 413 Sep 17 19:56 output-delta-170-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 500 Sep 17 19:56 output-delta-171-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 18047 Sep 17 19:56 output-delta-172-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 478 Sep 17 19:56 output-delta-173-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 561 Sep 17 19:56 output-delta-174-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 591 Sep 17 19:56 output-delta-175-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 416 Sep 17 19:56 output-delta-176-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 413 Sep 17 19:56 output-delta-177-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 413 Sep 17 19:56 output-delta-178-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 18151 Sep 17 19:56 output-delta-179-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 478 Sep 17 19:56 output-delta-180-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 561 Sep 17 19:56 output-delta-181-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 1316125 Sep 17 19:56 output-delta-182-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 431 Sep 17 19:56 output-delta-183-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 431 Sep 17 19:56 output-delta-184-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 416 Sep 17 19:56 output-delta-185-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 413 Sep 17 19:56 output-delta-186-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 500 Sep 17 19:56 output-delta-187-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 18099 Sep 17 19:56 output-delta-188-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 478 Sep 17 19:56 output-delta-189-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 561 Sep 17 19:56 output-delta-190-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 1319429 Sep 17 19:56 output-delta-191-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 431 Sep 17 19:56 output-delta-192-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 431 Sep 17 19:56 output-delta-193-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 685 Sep 17 19:56 output-delta-194-unit.txt
-rw-rw-r-- 1 ubuntu ubuntu 679 Sep 17 19:56 output-delta-195-unit.txt
-rw-rw-r-- 1 ubuntu ubuntu 685 Sep 17 19:56 output-delta-196-unit.txt
-rw-rw-r-- 1 ubuntu ubuntu 402 Sep 17 19:56 output-delta-197-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 17719 Sep 17 19:56 output-delta-198-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 467 Sep 17 19:56 output-delta-199-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 557 Sep 17 19:56 output-delta-200-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 1294460 Sep 17 19:56 output-delta-201-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 428 Sep 17 19:56 output-delta-202-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 420 Sep 17 19:56 output-delta-203-action.txt
-rw-rw-r-- 1 ubuntu ubuntu 685 Sep 17 19:56 output-delta-204-unit.txt
-rw-rw-r-- 1 ubuntu ubuntu 597 Sep 17 19:56 output-delta-205-unit.txt
-rw-rw-r-- 1 ubuntu ubuntu 3246 Sep 17 19:56 output-delta-206-application.txt
-rw-rw-r-- 1 ubuntu ubuntu 596 Sep 17 19:56 output-delta-207-unit.txt
-rw-rw-r-- 1 ubuntu ubuntu 597 Sep 17 19:56 output-delta-208-unit.txt
-rw-rw-r-- 1 ubuntu ubuntu 3880 Sep 17 19:56 output-delta-209-application.txt
-rw-rw-r-- 1 ubuntu ubuntu 598 Sep 17 19:56 output-delta-210-unit.txt
As you can see some are very big
(129, 201,182,191,089,096)
Those are the result of a run-action backup + a copy to a aws s3 (cli)
eg inside you would have that kind of command output
emaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (119.9 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted 5.0 GiB/5.0 GiB (120.0 MiB/s) with 1 file(s) remaining \rCompleted
option --quiet has been apply to the source command.
All in all that sound related to that Excessive growth of delta
At least same symptom.
So I suppose those deltas would disappear overtime no?
A last thought, is that, it seems not a good idea to use a script (pylibjuju) to perform a recurrent task (such as backup), if that cumulate some data overtime that would reach some limit anyway at some point?
Hi @wallyworld,
Looking at the dump this morning, it is mainly the output of actions
Is there a way to clean those actions from the model?
I think run-action should not be used as recurrent task (cron) obviously.
As we’ve diagnosed the juju run-action and juju run deltas will be streamed via the AllWatcher to the pylibjuju client. This is by design, the pylibjuju client uses the deltas to work out when an action has been performed.
The underlying issue is the size of the payloads and the fact that they’re exceeding the max_size of a websocket frame. Upping the frame size should allow you to work around the problem short term (RFC-6455 states that a single WebSocket frame can be 2^63 bytes, so you’ve got a lot to play with).
The long term fix for this is to get your charm to save the action output of a given item to a location (s3, local file system) and perform a juju scp (if using local file system) to get that data.
If you’re really sure you don’t want that data from a given action and I would strongly advise you don’t run this on a production server.
For all controller machines, SSH into them and run:
systemctl stop jujud-machine-*
Get access to mongo:
Then run the following, it will leave the actions in mongo, but clean out the results.
db.actions.updateMany({}, {$set: {"results": {} }})
Once you have run the database updates, you can then do:
systemctl start jujud-machine-*
@dgft beside setting pretty high frame size, did you try tweaking model-config options?
Many thanks for this script (updateMany) , is it possible to do for some actions only?
like db.actions.updateMany({id: XXX}, ($set: {"results": {} }}) ?
probably just use updateOne ?
I think this is the best solution.
Correct, the best way is probably to dump out all your actions and audit each one to ensure you’re not breaking anything unexpected.
Just found that the following seems to work.
db.actions.updateOne({"_id": "<model_uuid>:<action_id>"}, ($set: {"results": {} }})
Hi @simonrichardson,
I did remove a few results from db.actions using this syntax:
db.actions.updateOne({"_id": "<model_uuid>:<action_id>"}, ($set: {"results": {} }})
Doing a find("_id": "<model_uuid>:<action_id>"}) shows an empty results entry.
Now when running a script (pylibjuju) the deltas still contains the previous data, as emptying results is not committed.
You need to bounce the agent (service), as we hold the information in a local cache. I’ve updated the instructions above.
@simonrichardson, I may do something wrong,
juju dump-model show empty resultsafter systemctl restart jujud-machine-0.service (controller)
juju dump-model still smaller (passed from 9477406 to 349957, that was doing the db.actions.updateOne())Ok, might be some kind of delay, fine by now
Hi all,
To let you know about the size of the model this morning.
Between the 18th around 11am, size=349717
This morning 21th around 05am, size=349964
So no much than 200 bytes fairly good.
Two thoughts.
The script is reduced to two “run” calls.
action = await unit_instance.run_action('backup') <= which obviously is the backup action, which should generate a first “delta”action = await unit_instance.run(cmd.format()) <= and this, which will launch a cmd on the remote host, generating a second “delta”.Thus, the minimum of deltas are created.
the second cmd is a string a of shell commands, to be performed after the dump from the backup run_action, where the backup folder is packed, move to storage, then things are cleaned up.
Each shell command is redirected to a local log file, minimizing the delta results entries.
Many thanks to all,
Know a bit more of juju internals.
Regards,
And Have a good day.
Honestly, we shouldn’t be including the content of the action results in the all watcher delta. We could include the fact that an action has been run/status changed/etc, but the output of the action should not be in a Watcher stream.