Debugger and Juju

Hey folks,

I am Guillaume, I joined Juju six months ago, as a developer.

The Story

I had always been concerned by how quickly most developers give up on the idea of running a debugger on their software as it becomes increasingly complex, especially in distributed systems. I’ve stopped to count the number of projects I’ve been involved in where I’ve asked, during a peer programming session, “Let’s run a debugger and see what happens!”… And get the same answer from some jaded developer: “A debugger ? Never needed one.” Then spend an eternity adding logs, rebuilding, getting bored, and finally finding the bug in pain.

But I agree, even if it’s not difficult, debugging distributed systems can be a pain.

In Juju for instance, be able to have debug capabilities from end to end (from a command to controller), you need to:

  • Build Juju in debug (to export symbols alongside the binary)
  • Bootstrap a controller with those debug binaries
  • SSH to the controller
  • Install delve with the correct version (compatible with the golang version used to compile Juju)
  • Find jujud’s PID
  • Run delve with a cryptic command to attach it to Juju and listen to an exposed port
  • On your laptop, find out the IP of your controller
  • Finally, in your favorite IDE, run a remote delve client connected to this IP & port
  • Then you can run a Juju command in debug mode and follow step by step from your laptop to your controller (provided you put some breakpoint on the facade) Oh, then you also want to debug another controller ? Add another machine and run the debugger on it ? You are screwed, you will need to do that for every machine.

That’s boring.

However, it is probably better than adding log and rebuild, in my opinion. But not enough to engage an argument with my fellow jaded developer.

My wish and its fulfillment

I do love using a debugger. It is why, with the help of @tlm, we’ve set up a simple way to run a debugger on Juju. It comes in two part:

  • Embed Juju in a delve server whenever the binaries are compiled in DEBUG
  • Provide a script to auto discover the main use case (“I want to debug a controller”), and redirect the delve Port from the controller to a local port on your machine.

This makes it possible to debug a simple Juju configuration (a controller) with very few commands. The script works directly for controller’s machines, which can be accessed through Juju client. For added machines, you may need to set up an SSH key pair to make it works.

It stabilize the local configuration for remote debugging (on my IDE I defined several configuration each linked to a specific port, which help me to debug quite easily migration use case, with several controller for instance)

It’s work in progress. For now it works only on Machine Cloud, and has been mostly used on LXD. However, it should works on any Machine Cloud.

Curious ?

If you are curious, you can check the way we have done it:

It may inspire you if you want to debug distributed systems in golang. Notable ideas were:

  • Use a unix socket on the remote server to be able to run several delve server if there is several binaries, without worrying too much about ports ranges
  • Use of a ssh tunnel to bind the unix socket to local port, in order to be able to locally stabilize the debug configuration.
  • Use build tags to not ship debug dependencies in production binaries
2 Likes