I have a multiuser system on centos. Many hundred users.
Users home is on NFS.
We have resorted to having use the binary for the juju client on centos since no packages exists.
Further. We can’t build charms anymore since the buildprocess (snapcraft) requires lxd, which is not possible on centos+snap and multiuser system.
We are in a bad spot with juju atm.
So snaps do not work with home dir mounted on NFS? That seems unfortunate 
Have you tried asking on the snapcraft forums to see if the snap folks have plans to fix that?
With centos, we have never packaged rpms. Ideally the snap with NFS home issue would be fixed as that seems to be the root of the problem. I didn’t release snapcraft required lxd now. That sorta sucks if you can’t run lxd
This thread seems to have an answer?
snapcraft spawns an lxd. I have not managed to build a charm since long in this context and is about to give up on this.
We have tried to use juju with nfs homes in the past and gave up on it. I know @hallback has major issues with this aswell on a userbase on LDAP. But he might have to fill in here as I don’t know details.
Further, its extremely unfortunate that other communities running other distros will be alienated from juju altogether. I mean, snap, lxd, juju…
We have a large user-base on centos and then we have suse etc. Juju is not working well or at all in these user contexts atm.
It would be great to enumerate the exact issues and we can engage the various teams to see what can be done. eg it seems like an answer to the NFS home issue might unblock other things like lxd and therefore snapcraft.
Given the above snapcraft forum thread with a supposed solution, maybe you can see if that works and if not, share some more details about your setup and we can follow up.
We certainly want juju to be able to be used on platforms besides Ubuntu, so let’s work on understanding what the gaps are and we can go from there.
I’ll give it a try:
Our multiuser environment with NFS, ActiveDirectory integrations (LDAP/kerberos) makes it impossible to build charms with charmcraft. This is true on Ubuntu + Centos (likele all other distros too). This is a client side issue.
The same issue is true since juju also installs the “ubuntu” user via the machine agent. It breaks all distros that does not have the ubuntu user natively (and frankly irritates the f*ck out of other distro-ambasadors. Juju is offending to centos-users (and all other distros). This is a server side issue.
If I read correctly, if we address 2 root cause issues:
then snaps will work, therefore lxd works, therefore charmcraft works etc. And juju agents will stop breaking.
We’ve talked about fixing the “ubuntu” user issue in the past. We can revisit that. And there might be a snap config that makes #1 possible, given the issue was first raised in 2017 and I think folks have looked into it. The suggested solution in the snapcraft discourse needs to be validated.
A better solution IMO is to package Juju/charmcraft properly for Ubuntu and CentOS. Snaps are recreating all sorts of problems that were solved decades ago on Linux.
Having an rpm/deb for Juju would automatically solve @erik-lonroth’s problems 1 and 2.
Problem 3 could be solved by renaming the ubuntu user to juju and making it a system user, not a regular one.
Problem 4 is a bit tricky: charmcraft has explicit dependencies on specific versions of python wrappers around APT. Making it work on distros not based on Debian might require significant work. It also contains a --destructive flag to not use LXD for packing the charms, but that might break the packing if the host OS is not the same as the one specified on the charmcraft.yaml file. On the other hand, if charmcraft were able to use systemd-nspawn, all those problems would go away.
Those two issues are indeed root problems to a number of issues.
Also, experience from running snap on centos and other distros is that for example mechanisms such as “apparmor” or “selinux” hits hard to get to play well with snap:s. I’ve tried this path myself a number of times with the same outcome every time.
I would say that providing a “non snap” packaging of juju is needed, in such a way that it will be good for some of the distributions that are explicitly supported by juju already.
If there was packages for at least the client for distros lika suse, centos, debian, arch, etc. chances would be that a community could grow. That alone is a strong argument for supplying classic packages along the snap.
Then there is the charmcraft issue. It would be also a “root” issue itself since it is a hard requirement to run a lxd host to be able to build charms. This is adding to the complexity for people to be able to build charms… In our multiuser environment, the lxd snap would likely not work for example.
But yes, the 2 root causes you suggest would be a significant improvement.
Hi Erik, can you post a question to the snapcraft forum describing your issues running snaps, explaining how you have your home dirs set up etc. It would be good for the snap guys to be able to give the latest information on this, and we can have a conversation about how best to solve it with the folks that are in a position to help. In my view, if we claim that snaps are a reasonable cross distro software packaging solution, then issues which arise which impact that need to be looked at.
In terms of packaging juju, we do publish tarballs with each release of the relevant binaries. We don’t plan on doing debs or rpms or anything else at this stage.
With charmcraft, if snaps work, then it should just be transparent that lxd is used to pack the charm. Yes, there’s complexity behind the scenes, but that’s for charmcraft to manage. I think Facundo the charmcraft guy has asked for input in another forum thread, so anything that can be shared there would be good as I know he’s keen to help get this solved also.
With the “ubuntu” home dir issue, that’s on Juju to solve and we’ll take another look at it.
Could this be a thing for the community to develop then as a contribution to the project if Canonical won’t? With some help it should be doable.
Hi Erik,
Canonical as a company is focused on the snap as a delivery mechanism, while making the standalone Go binary available for places where the snap doesn’t make sense.
Juju is an open source project, though, so folks are welcome to package and deliver it however they please, as long as they honor the license. 
Love that.
But the dynamics of this would then be still that the upstream project would accept this contribution and ideally also at least provide some infrastructure to maintain it as part of releases etc. For example to be able to release both snap:s and deb:s as part of a release schedule for juju. I’m not experienced in how all that works, but for sure Canonical as the project lead would have to be partially involved.
I’m sure you (Canonical) have done these things before…
Is there a point in trying to engage with some independent Debian developers to produce deb? @heitor
I know @hallback is good at rpm.
@erik-lonroth it would be great if you could post a query to the snapscraft forum about your setup regarding NFS mounted home dir etc; I’d be interested to see what they respond with, and we can get a feel for what the gaps are etc.
Thanx for coming back to this.
I have posted a number of texts, bugs, discourse posts and expressed my concerns on this in both juju and snap discourses over the years on this.
Its painful and time consuming. It is also difficult for me to do the deepdive into the complexity related to the combination of many technologies. Not only snap, but also centos, nfs, selinux, etc.
I think that the easier path is to invest in getting a RPM built and possibly a DEB.
I don’t think snapd will be included in centos in a very long time, so getting a stable solution meanwhile is critical.
I appreciate the snap for juju in some contexts, but it has also caused alot of very serious problems that keeps coming back.
Oh wow, that’s a lot of posts. I’ll ask some people, but it does seem like it’s not a solved issue . It would be good to understand why.
My understanding is that snaps are supported on centos, so if the NFS home issue were solved, they could be a viable solution for the juju client.
@wallyworld I apprechiate the effort. But I don’t want to have the snap when working on systems which has NFS or/and centos.
I’m trying as we speak to build juju on centos7 and running into problem since the Makefile depends on snap aswell.
## install-mongo-dependencies: Install Mongo and its dependencies
@echo Installing ${JUJU_DB_CHANNEL} juju-db snap for mongodb
@sudo snap refresh juju-db --channel=${JUJU_DB_CHANNEL} 2> /dev/null; sudo snap install juju-db --channel=${JUJU_DB_CHANNEL} 2> /dev/null
@$(WAIT_FOR_DPKG)
@sudo apt-get --yes install $(strip $(DEPENDENCIES))
I’m looking for help to build juju so that I can produce a .RPM as a PoC.
How can I get mongodb rependencies in from sources?
I can’t find what is needed from here: https://github.com/juju/juju-db-snap
I’m currently at this point…
[root@c7 juju]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
[root@c7 juju]# make install-dependencies
Using installed go-1.17
Installing 4.4/stable juju-db snap for mongodb
make: *** [install-mongo-dependencies] Error 1
[root@c7 juju]#
I’d like to know if anyone would be able to assist me in building the code on centos and then I might figure out how to get help building the RPM from experts like @hallback =)