Python libjuju

[simonrichardson]

1. Syncing with juju
2. Releasing
3. Definitions

When developing Juju we sometimes have to change various API server interfaces, most if not all of the time these are or should be compatible with previous versions of Juju clients. Pythonlibjuju (pylibjuju) can be considered one of these clients. Going forward the CI infrastructure will start to test if your Juju PR will cause any issues with the pylibjuju client, if changes have broken the client, additional work will be required to fix the python code before attempting to merge your changes. These could include new endpoints, changes to parameters or deprecation of old existing endpoints.

Syncing with Juju

If you’re going to introduce new methods that you want to expose in pylibjuju you’re going to have to update the schema file found within pylibjuju. Inside Juju is a schema file that is sync’d for every pull request and release of Juju. This schema file contains the current schema (think layout) of the apiserver. In practice we want to copy that schema file from Juju to pylibjuju and rebuild the client from that.

Ensure you have the correct Juju and that you have everything installed according to the Juju documentation. (see below for script as a hole)

1. Firstly remove the previous juju latest schema symlink by doing: rm ~/python-libjuju/juju/client/schemas-juju-latest.json
2. Copy the file ensuring to set $JUJU_VERSION in the following command to something sensible like 2.6.4-dev:
   cp "$GOPATH/src/github.com/juju/juju/apiserver/facades/schema.json" "$HOME/python-libjuju/juju/client/schemas-juju-$JUJU_VERSION.json"
3. Create a symbolic link to the schema you just created
   cd ~/python-libjuju/juju/client
ln -s schemas-juju-$JUJU_VERSION.json schemas-juju-latest.json

Then move back to the root of pylibjuju

1. Run make client - You should expect to see updates to the juju/client/_definitions.py file, as well as one or more of the juju/client/_clientX.py files, depending on which facades were touched.

Additional work

Once syncing with Juju is complete you may need to integrate any new or changed API calls into the object layer, to provide a clean, Pythonic way to interact with the model. This may be as simple as adding an optional parameter to an existing model method, tweaking what manipulations, if any the model method does to the data before it is sent to the API, or it may require adding an entirely new model method to capture the new functionality.

In general, the approach should be to make the interactions with the model layer use the same patterns as when you use the CLI, just with Python idioms and OO approaches.

See: Syncing Upstream Updates — juju 2.6.3 documentation

Script

Ensure you replace $JUJU_VERSION with a value.

rm ~/python-libjuju/juju/client/schemas-juju-latest.json
cp "$GOPATH/src/github.com/juju/juju/apiserver/facades/schema.json" "$HOME/python-libjuju/juju/client/schemas-juju-$JUJU_VERSION.json"
cd ~/python-libjuju/juju/client
ln -s schemas-juju-$JUJU_VERSION.json schemas-juju-latest.json
cd ~/python-libjuju
make client

Releasing

Pylibjuju is relatively painless to release as opposed to Juju release process. Below is a series of changes required for releasing pylibjuju.

Firstly you’ll need an account for https://pypi.org/ (long term we could use a generic juju account, short term create your own - don’t forget to turn on 2FA). Secondly you’ll need to be made an contributor/maintainer of the pylibjuju project - speak to @pengale .

1. Update the changelog.rst inside the docs folder with the new version and relative information. Might be useful to mention if any facades where changed along with any new python features added.
2. Update VERSION inside the root of the project.
3. Create a new branch and get it reviewed before merging with the target branch (2.6 etc)

You are ready to release!

Ensure you’ve got everything set and ready.

1. make release

[simonrichardson]

Definitions

One of the good things about the architecture of pylibjuju library is the way it can handle multiple versions of Juju using different schemas. Using the different schemas allows it to generate the different clients version for interacting with different versions of the Juju API server. Keeping everyone isn’t really required and Juju itself does a good job at keep backwards compatible with itself, so in the future the number of schemas found within pylibjuju will drop to the latest and the potentially the previous major release (I’m really thinking about the alphas and betas that we have in the repo atm).

One of the downsides to having multiple versions of the schemas is how do we know which parameters/arguments/return results (we’ll call them definitions from now on) to use with which schema version. Currently it would locate the first occurrence of the definition and be done with it. It would then see any new version; let’s say 2.5.0 params.DestroyRelation vs 2.6.5 params.DestroyRelation as the same and write the 2.5.0 out into the client. The issue here is that a new field was added to the parameter (Force) and that wouldn’t be available to the client. The original would shadow the new.

I had known about this when I implemented trusted in the client, but originally I pushed anything that wasn’t known about in the kwargs (in the library they’re called unknown_fields). This was great for getting field values out, but you couldn’t set the value. Other users where also hitting the problem, so a maintainable solution was required.

Going forward the definition being picked will always be the latest one. It’s the only sane way to pass new parameters to the api server. I’ve spent sometime cleaning up how we generate the definitions and ensuring that we get the full richness of the new changes in Juju, but there comes a downside.

The downside is that when you change a definition, especially around deployment, units, machines, you’ll end up breaking the library and it’ll be you’re responsibility to fix where it breaks. Most of the time you’ll need to add an argument to a method that was missing or ensure that a default value should be used and then passed to the API server. It’s relatively simple, but needs to be done.

This is an new and ongoing process, but the consequences of changing Juju means that we should ensure that we keep other clients up to date.

[jameinel (John A Meinel)]

The flip side of this is that it is another place that highlights why even just adding fields is not a safe thing to do without updating the facade versions. Clients can’t tell if they are passing the new field and it is being ignored.
Having to rev the structs is a bit annoying, and it is tempting to use “,omitempty” to say that it is ok, because it just won’t be sent if you’re being compatible with the old version that wasn’t setting it. But things like libjuju can’t really tell that V4 didn’t have the field and V5 does. (especially bad if in V5 it is considered mandatory).