API & APIServer re-work

I noticed the other day that we’re making users of the api package bring in apiserver and also potentially bring in state. This isn’t ideal and I’d like to propose a change for both the api and apiserver packages. Unfortunately none of these are a simple task, but I still believe we should strive for it.

The problem is that our api client (/api/ package) imports from the apiserver package. This is out of necessity for the transport parameters, causing us to bring in the api server when ever you use the api client. The side affect for this is dire as we then unfortunately bring in the state package. If we reduced the scope of our packages and the imports they bring in, we could quicken up compiling tests[1] and reduce binary sizes that just depend on the api clients.

What I would like to see is the following:

  1. Move apiserver/params to core/transport/params or transport/params
  2. Drop api packages from apiserver.

The problem with both of these steps is that we send core juju concepts over the wire (constraints, placements) without marshalling them to an intermediate step (see params/applications.go).
We would have to refactor this to ensure that we deal with structures within the params package. Moving the helper functions to common for now. Although common marshalling functions should also be moved closer to where they’re used and not in a dumping ground package.

  1. Interestingly I noticed that whilst reading the go help test package that we could actually disable the vetting of your tests packages to improve the speed using go test -vet=off.

I think we need to move farther towards splitting them apart completely.
Specifically, you have a ‘client side’ understanding of the api, vs the ‘server’ understanding of the API that it is serving. And the client is meant to support multiple versions of the server (and vice versa). Sharing the types means that changing one immediately changes the other, which is nice for effort, but means that you could easily have broken API compatibility without much safety checks making it clear.
I believe we’ve already done most of the work to change the state types as being separated from the apiserver types. This feels like another layer of separation that we need to do. (And it would go very far in us having a clean ‘libjuju-golang’ that is just the client apis.)

To enforce it, we can just lint that nothing in ‘api’ imports anything from ‘apiserver’.

The dream :sunny:

We can probably avoid having totally separate api and apiserver types and just extract params, which we’ve wanted to do since forever. The reason I say that is because we already are careful to avoid changing existing params structs in a breaking way. We will typically introduce a FooV2 struct if we need to incompatibly change Foo struct. We don’t gain much IMO by introducing a pretty large overhead of maintaining and marshalling between 2 sets of (a very large number of) structs for over the wire transport of data. The current approach has served us ok for the past many years, over 17 or more facade version bumps etc etc, without any real consequences for libjuju etc.

IMO, let’s make the easy step of moving params to a separate package and get things aligned there first, and get a pretty good payback for minimal effort. There’s nothing stopping us from considering next steps after that, but we would have a made a huge improvement to start with.