QA of charms in charmstore?

framework
1

Is there presently some quality assuring process in charmstore in place or discussed? @rick_h

2

No, there’s not a global promise around the charms. It depends on the charm and the community around the charm to handle their development and QA practices as they build charms and go through the edge to stable channels.

In a way it’s like github or other hosting sites. In the end you end up learning who you trust I guess.

3

Right. I fear that a complete lack of QA places alot of charms in the low end and affect the “general feel” of juju itself.

I think that it would make some sense to have perhaps a “optional” test-suite automated that indicates if a charm passes some minimal QA checks.

I envision, for example, charms getting “rated” according to some badge system. That would encourages charmers to add an extra touch of quality without for that sake enforcing a QA.

The most pro charmers will end up having nice looking “badges” for their charms in the store based on the quality score or tests they reach/pass.

2 Likes
4

One thing that Github provides–as flawed as it is–is a count of stars/forks. That tends to provide some signal about the package’s quality. It’s also pretty easy to detect how alive the project is because the code is so close to the UI. Additionally, many repositories embed stickers from CI tools into their README.

It would certainly be nice to emulate some of that functionality. At the moment, it’s very difficult to detect what level of quality there is in each of the charms.

3 Likes
5

This is definitely something we can look at as we look at ways to build a common experience from the snap store through the Juju Charmstore.

1 Like
6

I agree with all this. When I look at a Github page, I look at general activity as well as frequency of updates, how recently it was updated, stars, forks, watchers, issues, commits, freshness of PRs, then the quality of the readme, with specific focus on build/install/deploy/integrate instructions. It’s a reasonable initial gut check. Agreed it’s flawed, but there’s enough metrics for me to get a good feeling for what I’m about to bite off.

On the charm store it’s really hard to see how recent something was updated (almost all outside of the promoted charms that I look at are very stale and broken, but that takes considerable effort to discover), forking a charm to upgrade series or something else simple is often challenged by unresolvable interface deps. Generally any charm with a series <Xenial is simply broken and unusable, if it appears in the ‘recommended’ section, in my experience (admittedly my window of use case is very blinkered, but some stats revealing the contrary would be welcome). In the community section a series-upgrade or charm pull and some simple tweaks often will suffice so long as the charm complexity is low.

4 Likes
7

Weeks later, I pick up on this topic again now that bionic and even disco will begin to enter the store.

Many, (most?) charms are getting either “out-dated” on the series, they often contain small but annoying errors and the lack of anything at all to be able to assess the quality of the community/recommended charms really needs some kind of plan.

I’ve hit a number of problems with charms I would just expect to work flawlessly but since often its not so easy to get them updated upstream, I end up soon having to maintain pretty much every charm in my private namespace or in my teams - effectively duplicating most of our used charms.

I’m more than willing to contribute to this process and have tons of ideas:

  • Some sort of “gamification” of active charmers? Badges, achievements, levels, promotions, etc.
  • Automatic rating of “compliance for charms and bundles”
  • Activity ratings.
  • Tracking of repository updates.
  • etc. etc.

Looking forward to help.

1 Like
8

As a reference, Nextcloud has a community very much like juju-charms.

They have about the same problematics and has addressed some of the issues like described in this text:

We want to avoid duplication and make it harder to ship broken releases by mistake, therefore we went for the following solution:

Your app’s source code is hosted on GitHub or a similar service

You should use Git tags to create new releases on these services

GitHub release downloads do not match the required folders structure. This is because GitHub appends a version to the top folder name. Therefore you need to create a separate release which conforms to the expected structure.

This keeps your repository up to date and satisfies the needs of maintainers, developers and experienced users.

Quoted from : App Developer Guide — nextcloudappstore 2.0.0 documentation

While this doesn’t cover things like rewarding maintainers, quality-tagging, gamifying contributions etc. I think its worth taking a very good look at.

I run Nextcloud myself and their “Apps” are like plugins which normally has quite a good quality.

@rick_h @seffyroff @timClicks

1 Like
9

Seeing how other communities have tackled this situation is revealing. I remember similar issues with Sandstorm and its app market when I used it extensively. Nextcloud’s approach seems very pragmatic.

I suppose there are a few models:

  • staff only
    as seen in the Debian community, where only trusted packagers (Debian developers) are permitted to add packages to the distro
  • gated community
    such as the curation provided by Canonical in its snapstore. You have the ability to publish what you want, but privileges can be revoked
  • mirror upstream
    develop some system, such as Nextcloud’s, that will ensure that the code that’s available from the “app store” is the same that’s currently being released by them “app store”
  • laissez faire
    people uploading packages are responsible for them, and the level of support that they provide is up to them
1 Like
10

Where does systems with badges, gamification etc fit in or are there more models?

11

Oh good point! I will need to fit reputation in there too.

1 Like
12

Another somewhat successful model is that which Apple’s gradually adopted on their app store, since Free Apps became the norm:

  • Anyone can release anything after paying a small fee (which is arguably unnecessary, and probably only funds their app-review system) - so removing the gated initial publishing manual human review could also remove the small fee to become a ‘registered developer’
  • There are some basic rules on what can and cannot be released on the app store, in terms of acceptable content. This doesn’t always work so well, so let’s skip that too
  • An automated packaging step performs some critical validation, and prevents anything going on the store that doesn’t meet basic standards in terms of coding and presentation (Juju already does some of this with the charm-tools, which is great!)

This things are the main things that Apple makes a deal out of with developers. But I think the more critical parts to encourage a healthy community driven content store are:

  • Curation. They have a massive focus on putting their name to content, endorsing it, and making it front and center. This exposure is so valuable to an app’s popularity (whether it is an app that generates revenue or not) that every developer interested in folk using their app make the effort to catch the curator’s eye.
  • Visible updates. An app’s update date is one of the first things you see. This I think is even more critical than curation - once again this indicates an app’s worth to me if it’s been updated recently. If the Charm store had this+comments (perhaps aggregated from a public VC repo) the typical bounce rate from a user discovering apps would decrease significantly.
  • Community reviews. A simple comment system that gives another indication of an app’s worth to me. I can quickly identify if my use-case is one that has issues within the app, or if there’s something else I should know that has already been called out.
2 Likes
13

Amazing contribution to the discussion.

There is alot of content in here now which would be great to put up for a community workshop?

14

Perhaps it would be worthwhile to host a video conference periodically?

1 Like
15

Yes, that would be good. Perhaps with themes?

Also, charming workshops.