That seems quite surprising, my first thought is that the test to see if it exists is somehow incorrect, thus it isn’t properly causing it to install. My quick bash test seems to say it should be working, though.
$ if ! type -a blababa; then echo "nope" ; fi
bash: type: blababa: not found
nope
Could it be a case of something like env caching the paths? I know for ‘bash’ if you install something you often need to call hash -r so it forgets the existing lookups that it has done. Would there be something equivalent here?
You could also do a more direct approach:
if [ "$JUJU_DISPATCH_PATH" == "hooks/install" ] ; then
...
fi
The main problem with that one is that storage attached hooks fire before install (so that you can know where your storage is by the time you go to install software). You could also do something like:
if [ ! -e ".installed" ] ; then
yum install python
touch .installed
fi
this also looked like it worked
$ if ! /usr/bin/env pppp ; then echo 'nope'; fi
/usr/bin/env: ‘pppp’: No such file or directory
nope
And has the advantage that it is the same executable doing the lookup as the one that will be execed in src/charm.py
@jameinel are there other options we can look at here? For example, can we expose some type of charm pre-exec hook that could be separate from and ran before any operator code or actual charm hooks? I know people have previously used a convention where they would just have the install hook do any pre-install work, then have the install hook call install.real and subsequently handle install.real in the python charm code.
The openstack charms are a good example of where this convention is currently used:
Possibly the operator framework can expose something to support this functionality?
The main hurdle here is making sure python3 is on the system prior to any operator code being ran. Maybe it is better to solve that problem individually rather then in the context of the operator framework.
One idea I have is to send python3 in the venv with the operator charm. This seems like a reasonable solution, though I’ve yet to give it a try.
I ended up making some modifications that add python3 to the charm at build time. I’m not sure if this is something that works for everyone, but I thought it was worth exploring.
I think to get this working, glibc also needs to exist at a specific version on the system.
When I try to deploy a centos7 charm and use the supplied python3 from my example, I see that the python3 I supply requires a higher version of glibc then exists on the system.
$ sudo yum install glibc
Package glibc-2.17-307.el7.1.x86_64 already installed and latest version
The charm’s log shows
./venv/bin/python3: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by ./venv/bin/python3)
From the python packaging tutorial ,
Linux C-runtime compatibility is determined by the version of **glibc** used for the build.
The glibc library shared by the system is forwards compatible but not backwards compatible. That is, a package built on an older system *will* work on a newer system, while a package built on a newer system will not work on an older system.
Which is probably what is going on underneath here.
I also created this issue for further discussion on the charmcraft side of things.
After some reading up, I have found the manylinux project. From what I can tell, manylinux builds python in their docker images with an older toolchain, and older version of glibc.
This leads me to believe that it might be possible to generate and package the venv and python3.8 binary in the charm by using the python3.8 in the manylinux docker image (which is built with the older glibc).
The version of glibc used in the manylinux image is 2.17.
$ docker run -it --rm quay.io/pypa/manylinux2014_x86_64 bash
[root@0b68add7d530 /]# /opt/python/cp38-cp38/bin/python
Python 3.8.5 (default, Jul 27 2020, 16:20:29)
[GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.confstr('CS_GNU_LIBC_VERSION')
'glibc 2.17'
By mounting the requirements.txt and the build dir, we are able to generate the venv and python3.8 to be packaged into the charm with the manylinux python built with glibc 2.17.
#!/bin/bash
set -eux
docker run -it --rm \
-v `pwd`/requirements.txt:/srv/requirements.txt \
-v `pwd`/build:/srv/build \
-v `pwd`/out:/srv/out \
quay.io/pypa/manylinux2014_x86_64 \
/bin/sh -c \
'/opt/python/cp38-cp38/bin/python -m venv --copies --clear /srv/build/venv && \
/opt/python/cp38-cp38/bin/pip install -r /srv/requirements.txt -t /srv/build/venv/lib/python3.8/site-packages && \
cp -r /usr/local/lib/libcrypt.* /srv/build/venv/lib/ && \
cp -r /opt/_internal/cpython-3.8.5/lib/python3.8 /srv/build/venv/lib/ && \
cp -r /opt/_internal/cpython-3.8.5/include/* /srv/build/venv/include/ && \
chown -R 1000:1000 /srv/build && \
/opt/python/cp38-cp38/bin/python /srv/build/scripts/create_zip.py && \
chown -R 1000:1000 /srv/out'
By doing this we were able to build a .charm that we can deploy on centos and ubuntu.
This leaves me wondering if a) is charmcraft the right place to look at doing this sort of thing?, b) if yes, can we make charmcraft support a docker based build component like this?
To finish out the day, I was able to create a few scripts that build the operator charm with the venv generated from the manylinux python.
This looks cool, but would this swap dependencies to docker then? Its perhaps not ideal since it then likely adds a whole new dependency perhaps even bigger than python itself?
Then again, I might not fully grasp the manylinux construct from my initial reading.
It uses docker to generate python3.8 at build time , packages python3.8 with the charm and uses python3.8 to execute the charm code at run time. The charm unaware of docker in this case, only used at build time to get a specific python into the charm.
I’ve iterated on this a bit further and have come up with a working prototype of something that generates the venv once, and then seeds it into multiple charms as it builds them
There is no specific reason why you shouldn’t be able to do that with dispatch. It is just a bash script, just like they would be doing with install and install.real. dispatch should be able to do exactly what you want, and then invoke src/charm.py.
I’d rather just figure out why test-and-install isn’t allowing src/charm.py to be executed. There are several options, like doing:
exec $PYTHON3 src/charm.py
rather than just ‘exec src/charm.py’
That leaves bash and dispatch to find the PYTHON to use, rather than ‘/usr/bin/env’ in case there is some sort of problem there.
Bundling the python executable itself is possible, though we’ve looked into it a bit, and you run into problems with dynamic libraries. (Even if you make python itself statically compiled, you almost always still depend on libc and RedHat’s libc is different from Ubuntu, vs different versions of Ubuntu, etc.)
Though as I understand it, you can statically link against libc (AIUI the Juju binaries do).
It is possible to bundle, but realize that it does add its own overhead (larger charm binaries, larger attack surface, more to update when, eg, and SSL vulnerability is found, etc.) Distro’s already solve most of the security issues of keeping packages up to date, so leveraging them for that is usually wise.
@jameinel I agree with all of that. The problem though, is that if the dispatch file exists at the time when the install event happens then the dispatch file will run and call src/charm.py without even noticing my install hook, hooks/install. Which, is why I initially did the hack of not writing out dispatch until I am in the install hook, after python3 gets installed.
I think there are two things to bring to light; 1) I can’t use dispatch without using either the hack of writing out the dispatch file in the install hook or sending python with the charm, 2) I’m blocked from using charmcraft because I have to use one of those two methods to get dispatch working.
and
Oh, this is it. Thank you. My apologies for not groking this a few days back.
Using this, I don’t need to write my own hooks/install, but rather my own dispatch file.
Trying things out, it looks like I can just specify my own dispatch file in the charm root and charmcraft build will bring that dispatch file forward into the charm instead of generating one.
This allows my charm to be built with charmcraft and additionally/optionally allows me to add my own custom dependencies before the charm code executes by creating my own dispatch file that is kept/built with the charm.
This is exactly what I was looking for and allows for my use case of getting deps onto the system before the charm code runs.
Thank you for entertaining the conversation!