Review `spark-integration-hub-k8s` for listing

[deusebio (Enrico Deusebio)]

The Integration hub for Charmed Apache Spark is the component of the Charmed Spark solution that enables the integration with existing charms (PostgreSQL, S3-integrator, etc) and bundles (COS, Identity, etc) and properly configure the corresponding option in the Apache Spark ecosystem.

The Server Integration Hub Operator deploys and operates a deployment of the Integration Hub component, that takes care of managing configurations for Charmed Spark Service accounts.

It is part of the Charmed Spark Solution.

Documentation Links

The Integration hub for Charmed Apache Spark operator is part of the Charmed Apache Spark solution. Please refer to the solution docs for more information about how to deploy it and configure it as part of the solution. More information about how the Spark History server fit in the Charmed Spark solution can be found in the explanation section and in the README.md.

Metadata links

• charmhub page
• Github project repository
• Contribution
• Licensing
• Publisher: Canonical

Codebase

PR for code review

CI Links

• Code linting or-style checks
• Release automation
• Unit test implementation
• Unit test results
• Integration test implementation
• Integration test results

Further testing when part of the solution can also be found in the bundle repository:

• Implementation
• Results

[deusebio (Enrico Deusebio)]

Hi @tony-meyer

I have a companion review request as the previous one (BTW, we are also addressing your comments and suggestions in a PR that I’m about to link shortly in the other thread).

Some of the comments there are likely to apply here as well, and I’m already addressing them in a similar PR, so hopefully by the time you will get to this review they will already be fixed, but I wanted to start to start raising the requests as those comments were nevertheless minor and I’d like to get the charm reviewed by end of July when there will the be GA for the solution, such that the various charms are listed (beside this charm, only one more charm will be required which is consolidating its features as we speak)

[tony-meyer]

When I do the deploy & integrate in the instructions, the charm stays in blocked:

Model           Controller      Cloud/Region        Version  SLA          Timestamp
history-server  spark-tutorial  microk8s/localhost  3.6.7    unsupported  17:58:43+12:00

App                        Version  Status   Scale  Charm                      Channel        Rev  Address         Exposed  Message
s3-integrator                       active       1  s3-integrator              1/stable       145  10.152.183.248  no       
spark-history-server-k8s            active       1  spark-history-server-k8s   3.4/stable      38  10.152.183.127  no       
spark-integration-hub-k8s           blocked      1  spark-integration-hub-k8s  latest/edge     58  10.152.183.44   no       Integration Hub is not running. Please check logs.
traefik-k8s                2.11.0   active       1  traefik-k8s                latest/stable  236  10.152.183.99   no       Serving at 10.21.40.249

(I’m re-using the model I had for the history server review, but I assume that shouldn’t matter).

It says to check the logs, which I’m assuming means the Juju logs (I haven’t integrated with Loki), but I don’t see anything interesting there. For example:

unit-spark-integration-hub-k8s-0: 17:52:03 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch)
unit-spark-integration-hub-k8s-0: 17:52:03 INFO juju.worker.uniter found queued "leader-elected" hook
unit-spark-integration-hub-k8s-0: 17:52:05 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch)
unit-spark-integration-hub-k8s-0: 17:52:06 INFO unit.spark-integration-hub-k8s/0.juju-log Updating integration hub config...
unit-spark-integration-hub-k8s-0: 17:52:06 INFO unit.spark-integration-hub-k8s/0.juju-log Layer: {"summary": "spark integration hub layer", "description": "pebble config layer for spark integration hub", "services": {"integration-hub": {"environment": {"SPARK_PROPERTIES_FILE": "/etc/hub/conf/spark-properties.conf"}}}}
unit-spark-integration-hub-k8s-0: 17:52:08 WARNING unit.spark-integration-hub-k8s/0.juju-log No Loki endpoints available
unit-spark-integration-hub-k8s-0: 17:52:08 INFO juju.worker.uniter.operation ran "integration-hub-pebble-ready" hook (via hook dispatching script: dispatch)
unit-spark-integration-hub-k8s-0: 17:52:10 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch)
unit-spark-integration-hub-k8s-0: 17:52:10 INFO juju.worker.uniter found queued "start" hook
unit-spark-integration-hub-k8s-0: 17:52:12 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch)

Am I doing something wrong?

Also: once I fix this and do the integrate command, how do I know that everything is working as expected? I assume I could figure this out from the Spark docs, but if you could give me a hint so I don’t have to learn much Spark that would be great!

Checked	Review item	Objective	Review criteria
	Intended functionality	The charm must work!	Charm does what it is meant to do. For charms that are easily used without specialised infrastructure, successfully completing the tutorial is sufficient. For more complex charms, the submitter should provide a (possibly prerecorded) demo.
✓	Charmhub.io charm detail page	A complete and consistent appearance of the charm is required for a quality impression of the charm collection.	The overall appearance looks good and the documentation looks reasonable.
✓	Release automation implementation	An implementation for automated releasing to charmhub.io improves the ability to provide updates covering vulnerabilities quickly.	Release automation for unstable channels to enable testing when new versions of the charm code or the workload become available.
✓	Integration tests implemented	In particular for the review of charms, assuring a reasonable test suite is important to allow for automated releases in future. With this test, for every build, it is ensured that the integration with other charms is successful.	Integration tests verify that the charm can be deployed and ends up in a success state, and that the charm can be integrated with at least one example for each ‘provides’ and ‘requires’ specified (including optional, excluding tracing) ending up in a success state. The integration tests are currently passing.
✓	Coding conventions	The source code of the charm is accessible in the sense of approachability. Consistent source code style and formatting are also considered a sign of being committed to quality.	The implemented checks for coding conventions are reasonable and implemented in the regular CI/CD implementation.
✓	Documentation for contributing	The documentation for contributing to the charm should be separate from the documentation for developing or using the charm.	URL to this documentation is present and resolves with 2xx.
✓	Licensing statement	For the charm shared, OSS or not, the licensing terms of the charm are clarified (which also implies an identified authorship of the charm).	URL to the ruling licensing statement is present and resolves with 2xx.
✓	Metadata Links	A complete and consistent appearance of the charm is required.	charmcraft.yaml includes name, title, summary, and description fields that are not the default profile values, and a links field that contains documentation, issues, source, and contact fields, which resolve with 2xx.

Best practices:

• Use absolute paths in subprocesses to prevent security issues.
• Execute processes directly rather than via the shell.
• Name the repository using the pattern <charm name>-operator for a single charm, or <base charm name>-operators when the repository will hold multiple related charms.
• Ensure that tooling is configured to automatically detect new versions, particularly security releases, for all your dependencies.
• Ensure that the pyproject.toml and the lock file are committed to version control, so that exact versions of charms can be reproduced.
• Capture output to stdout and stderr in your charm and use the logging and warning functionality to send messages to the charm user, rather than rely on Juju capturing output. In particular, you should avoid print() calls, and ensure that any subprocess calls capture output.
• Do not build log strings yourself: allow the logger to do this for you as required.
• Ensure that log messages are clear, meaningful, and provide enough information for the user to troubleshoot any issues. Avoid spurious logging. For instance, try not to log when event handlers are called, as the Juju controller does this automatically.
• Never log credentials or other sensitive information.
• The quality assurance pipeline of a charm should be automated using a continuous integration (CI) system.
• Libraries should never mutate the status of a unit or application. Instead, use return values, or raise exceptions and let them bubble back up to the charm for the charm author to handle as they see fit.
• Just like Juju, a charm is an opinionated tool. Configure the application with the best defaults (ideally the application is deployable without providing any configuration at deploy time), and only expose application configuration options when necessary.
• Documentation links should apply to the charm, and not to the application that is being charmed. Assume that the user already has basic competency in the use of the application.
• charm name: The name should be slug-oriented (ASCII lowercase letters, numbers, and hyphens) and follow the pattern <workload name in full>[<function>][-k8s].
• Config: Prefer lowercase alphanumeric names, separated with dashes if required.
• requires/provides in charmcraft.yaml: Always include the optional key, rather than relying on the default value to indicate that the relation is required. Although this field is not enforced by Juju, including it makes it clear to users (and other tools) whether the relation is required.
• All charms should provide the commands configured by the Charmcraft profile, to allow easy testing across the charm ecosystem. It’s fine to tweak the configuration of individual tools, or to add additional commands, but keep the command names and meanings that the profile provides.
• Set the requires-python version in your pyproject.toml so that tooling will detect any use of Python features not available in the versions you support.

[deusebio (Enrico Deusebio)]

Hi @tony-meyer

Thank you for your kind review and feedbacks. I have addressed most of your comments in this PR: [DPE-7484] Address comments raised during charm review process by deusebio · Pull Request #88 · canonical/spark-integration-hub-k8s-operator · GitHub

Also I have updated the README with the correct information on how to deploy and validate that the charm is working. Please have a look at the “Usage” section and let me know if it is clear and if you can run it without problem.

The issue you faced is that in the documentation there was a --trust missing when deploying the charm. The charm was rightfully getting into blocked status, but the messages and logs were far from informative. I have opened an issue in the charm to fix this, see here

Let me know if you still face issue

Thanks Enrico

[tony-meyer]

For what it’s worth, doing a remove-application when not deployed with --trust also fails, with:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-spark-integration-hub-k8s-0/charm/src/charm.py", line 46, in <module>
    ops.main(SparkIntegrationHub)
  File "/var/lib/juju/agents/unit-spark-integration-hub-k8s-0/charm/venv/lib/python3.10/site-packages/ops/__init__.py", line 356, in __call__
    return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
  File "/var/lib/juju/agents/unit-spark-integration-hub-k8s-0/charm/venv/lib/python3.10/site-packages/ops/_main.py", line 475, in main
    manager.run()
  File "/var/lib/juju/agents/unit-spark-integration-hub-k8s-0/charm/venv/lib/python3.10/site-packages/ops/_main.py", line 459, in run
    self._emit()
  File "/var/lib/juju/agents/unit-spark-integration-hub-k8s-0/charm/venv/lib/python3.10/site-packages/ops/_main.py", line 403, in _emit
    self._emit_charm_event(self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-spark-integration-hub-k8s-0/charm/venv/lib/python3.10/site-packages/ops/_main.py", line 438, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-spark-integration-hub-k8s-0/charm/venv/lib/python3.10/site-packages/ops/framework.py", line 351, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-spark-integration-hub-k8s-0/charm/venv/lib/python3.10/site-packages/ops/framework.py", line 924, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-spark-integration-hub-k8s-0/charm/venv/lib/python3.10/site-packages/ops/framework.py", line 1030, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-spark-integration-hub-k8s-0/charm/src/events/integration_hub.py", line 55, in _remove_resources
    self.integration_hub.workload.exec(
  File "/var/lib/juju/agents/unit-spark-integration-hub-k8s-0/charm/src/common/k8s.py", line 79, in exec
    raise e
  File "/var/lib/juju/agents/unit-spark-integration-hub-k8s-0/charm/src/common/k8s.py", line 75, in exec
    output, _ = process.wait_output()
  File "/var/lib/juju/agents/unit-spark-integration-hub-k8s-0/charm/venv/lib/python3.10/site-packages/ops/pebble.py", line 1833, in wait_output
    raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
ops.pebble.ExecError: non-zero exit code 1 executing ['kubectl', 'delete', 'secret', '-l', 'app.kubernetes.io/managed-by=integration-hub', '

Not sure if you want to add some sort of handling for that.

(I was removing it and reinstalling rather than just adding trust).

[tony-meyer]

Everything looks good to me now (in the PR, and testing it out). Thanks @deusebio !

@adityagoel28 this one is good for listing, thanks!

[adityagoel28 (Aditya Goel)]

Hi,

I have listed the charm, but charms will not show up unless they are defaulted to a channel that has a stable release.

I did some investigation and your two options would be to change the default track to 3 (which I can do). Option two would be to release a revision to latest/stable.

Thanks, Aditya

[deusebio (Enrico Deusebio)]

Yes, we are going to promote the latest revisions we have in latest/edge in the 3 track and then switch the default track to 3. It is fine that the charm will not yet appear, until we do so.

FWIW, I don’t think anyone should use the latest/stable, as we cannot ensure that people on that track can upgrade seamlessly. We should commit to seamless upgrade within a track, not across tracks

Thanks for your support Enrico