Proposal: Charmcraft Analyze

charmcraft
1

Abstract

Have a mechanism for Charmcraft to validate that the charm was built using the best charm crafting practices as defined by the ecosystem, in addition to exposing if it is using the Operator Framework.

Rationale

Today a charm is consumed as a blob without much information exposed by its internals. Charmcraft can expose some of this information if the charm were to be assembled with it. This information, carried as ancillary data in the charm, can help an operator’s decision analysis on how good the charm is, as well as help the charm author keep up with the latest best practices for charms.

Specification

The goal is to have an analysis method that can act as a gating system as well as an information and warning system. This specification will only introduce three information level messages to bootstrap the tool with its immediate requirements. In the case of acting as a warning system, also add the results of this process to Charmcraft’s manifest.yaml.

The proposal is then to have two ways to exercise this functionality:

  • explicitly, by calling charmcraft analyze <charm-file> [--format=json]
  • implicit as part of charmcraft pack [--force]

Checks

Form of a check

Two types of checks are introduced

  • attribute for characteristics or attributes identified from the charm
  • lint for potential issues with the charm (warnings and errors)

Attributes can hold a value specific to the check (e.g.; python) or two special values:

  • ignored if the check has been explicitly ignored by user action
  • unknown if the analysis was unable to determine the value for the attribute.

Linting results can hold any of these values for a given check:

  • ok for a check that has run with no issues
  • warning for lining warnings
  • errors for lint errors
  • fatal for issues with the check itself
  • ignored if the check has been explicitly ignored by user action

Attribute Checks

These two checks introduced here are part of this specification to bootstrap the tool, future checks can be introduced by proposing them in a new document with the rationale for.

Language:

If through analysis, the charm can be detected as being a python based charm, then language shall be set to python. If not, it shall be set to unknown.

When working with python, it is possible to only publish byte-code. By doing so, troubleshooting is a harder task. Charms with python sources delivered are preferred.

This attribute meets the requirements to be set to python when:

  • the charm has a text dispatch which executes a .py
  • the charm has a .py entry point
  • the entry point file is executable

Framework

When using the operator framework, it is best to import it from a common path and not make customisation or package forks from it. If the operator framework is detected in the charm sources, this attribute’s value shall be set to operator. If not, it shall be set to unknown.

This check hint meets the requirements of operator when:

  • language attribute is set to python
  • the charm contains venv/ops
  • the charm imports ops in the entry point.

This check hint meets the requirements of reactive when:

  • has a has a metadata.yaml with a declared name
  • has a reactive/<name>.py file that imports charms.reactive
  • has a file name that starts with charms-reactive- inside the charm’s wheelhouse directory

Inhibiting checks

For the cases when a check is not desired by the charm author, a mechanism similar to most linting tools is introduced. The charm author must add in their charmcraft.yaml the following:

analysis:
    ignore:
        linters: [<check]>,...]
        attributes: [<check]...]

Analyzing

When charmcraft analyze ... is run, stricter rules would apply for a successful run. No warnings or errors would need to be found for the analysis to be successful.

It shall also be possible to suppress the ignored checks by running with --force.

Text output

When analyzing a charm file (charmcraft analyze apache.charm), the default outputs shall be as follows,

  • charm using charmcraft’s default operator framework setup:
charmcraft analyze apache.charm
Attributes:
- language: python (https://juju.is/docs/sdk/analysis/language)
- framework: operator (https://juju.is/docs/sdk/analysis/framework)
  • charm using charmcraft’s python setup but not using the operator framework:
Attributes:
- language: python (https://juju.is/docs/sdk/analysis/language)
- framework: unknown (https://juju.is/docs/sdk/analysis/framework)
  • charm using charmcraft’s python setup but not using the operator framework but ignoring attributes-framework:
Attributes:
- language: python (https://juju.is/docs/sdk/analysis/language)
- framework: ignored (https://juju.is/docs/sdk/analysis/framework)
  • charm using the operator framework with a hypothetical warning and error:
Attributes:
- language: python (https://juju.is/docs/sdk/analysis/language)
- framework: operator (https://juju.is/docs/sdk/analysis/framework)
Lint Warnings:
- foo: <text> (https://juju.is/docs/sdk/analysis/foo)
Lint Errors:
- bar: <text> (https://juju.is/docs/sdk/analysis/bar)
  • charm using charmcraft’s default operator framework setup with verbose output (showing all checks):
Attributes:
- language: python (https://juju.is/docs/sdk/analysis/language)
- framework: operator (https://juju.is/docs/sdk/analysis/framework)
Lint OK:
- foo: no issues found (https://juju.is/docs/sdk/analysis/foo)
- bar: no issues found (https://juju.is/docs/sdk/analysis/bar)

JSON output

To output the message in a machine format, the command shall offer a --format option with the only possible value of json, which using the example above shall print to stdout:

When analyzing a charm file (charmcraft analyze apache.charm), the json outputs shall be as follows,

  • charm using charmcraft’s default operator framework setup:
[
    {
        "name": "language",
        "result": "python",
        "url": "https://juju.is/docs/sdk/analysis/language",
        "type": "attribute"
    },
    {
        "name": "framework",
        "result": "operator",
        "url": "https://juju.is/docs/sdk/analysis/framework",
        "type": "attribute"
    },
    {
        "name": "foo",
        "result": "ok",
        "url": "https://juju.is/docs/sdk/analysis/foo",
        "type": "lint"
    },
    {
        "name": "bar",
        "result": "ok",
        "url": "https://juju.is/docs/sdk/analysis/bar",
        "type": "lint"
    }
]
  • charm using charmcraft’s python setup but not using the operator framework:
[
    {
        "name": "language",
        "result": "python",
        "url": "https://juju.is/docs/sdk/analysis/language",
        "type": "attribute"
    },
    {
        "name": "framework",
        "result": "unknown",
        "url": "https://juju.is/docs/sdk/analysis/framework",
        "type": "attribute"
    },
    {
        "name": "foo",
        "result": "ok",
        "url": "https://juju.is/docs/sdk/analysis/foo",
        "type": "lint"
    },
    {
        "name": "bar",
        "result": "ok",
        "url": "https://juju.is/docs/sdk/analysis/bar",
        "type": "lint"
    }
]
  • charm using charmcraft’s python setup but not using the operator framework but ignoring framework:
[
    {
        "name": "language",
        "result": "python",
        "url": "https://juju.is/docs/sdk/analysis/language",
        "type": "attribute"
    },
    {
        "name": "framework",
        "result": "ignored",
        "url": "https://juju.is/docs/sdk/analysis/framework",
        "type": "attribute"
    },
    {
        "name": "foo",
        "result": "ok",
        "url": "https://juju.is/docs/sdk/analysis/foo",
        "type": "lint"
    },
    {
        "name": "bar",
        "result": "ok",
        "url": "https://juju.is/docs/sdk/analysis/bar",
        "type": "lint"
    }
]
  • charm using the operator framework with a hypothetical warning and error:
[
    {
        "name": "language",
        "result": "python",
        "url": "https://juju.is/docs/sdk/analysis/language",
        "type": "attribute"
    },
    {
        "name": "framework",
        "result": "operator",
        "url": "https://juju.is/docs/sdk/analysis/framework",
        "type": "attribute"
    },
    {
        "name": "foo",
        "result": "warning",
        "url": "https://juju.is/docs/sdk/analysis/foo",
        "type": "lint"
    },
    {
        "name": "bar",
        "result": "error",
        "url": "https://juju.is/docs/sdk/analysis/bar",
        "type": "lint"
    }
]

Return codes

These return code apply, matching those of review-tools:

  • 0, found no errors or warnings
  • 1, checks not run due to fatal error
  • 2, found only errors or errors and warnings
  • 3, found only warnings

Packing

During pack, any non ignored check is tested for. Informational messages are not shown unless packing with --verbose (this includes attributes and lint-warning type messages). Errors and warnings are printed.

To pack regardless of errors you shall be able to use --force.

Return codes

These return codes apply, matching those of review-tools:

  • 0, found no errors or warnings
  • 2, found only errors or errors and warnings

Use of --force overrides the error or warning while displaying.

Manifest

Only attributes shall make it into manifest.yaml with their result, leaving the manifest snippet looking like

analysis:
   attributes:
   - name: language
     result: python
   - name: framework
     result: operator

if ignored,

analysis:
   attributes:
   - name: language
     result: ignored
   - name: framework
     result: ignored

Opens and considerations

  • Add text to JSON output.
  • Multiple occurrences of the same check (no identified case for it yet, but implementation might need to consider it).
2 Likes
2

+1 to being opinionated, as long as we clearly document that venv is the default path for virtualenvs, and preferably support a flag to point it somewhere else. Charms build pretty quickly, but being able to analyze before packing may be useful in some cases, and it’s easy enough to look at requirements.txt to see if ops is there. This would also allow us to easily check whether it’s being built off of a fork.

In a similar vein, which linters? Are we looking for specific versions of black or pyflakes? If we are, we should have been in the default scaffold.

Should we look for a pass of ./run_tests as part of charmcraft pack || build ?

3

./run_tests could be part of the default upcoming charm plugin behavior.

1 Like
4

The presence of the “reactive” subdirectory with files in it would make the framework be reactive. Also, reactive would imply python as the chosen language. IMO.