I think I ended up sorting out the primary issue. there is a fringe issue with x540-dt2 intel dual 10gb nic chipsets. they will work fine and communicate traffic however if they are added to a bond they will work but have intermittent communications failure by spamming traffic to themselves? i am not 100% sure the exact issue as it too me a lot of time to sort out. i switched all of my equipment to x520-dt2 network cards and the issue never happened again. There is apparently a kernel upgrade driver to fix the issue but I was worried about overwriting during updates so I just swapped to the other chipset. There was a lot of chasing ghosts as it was connected to the network. it could ssh and all the software ran. it was just something to do with communication between systems that had issues. there weren’t even issues with communication to the switches so… crazy edge case.