Spark Charm -> Spark 2.4

framework
1

Created this ticket here https://issues.apache.org/jira/browse/BIGTOP-3100

@magicaltrout @kwmonroe ^^ thank you

2

I was planning a big fat refresh along side the Bigtop 1.3.0 release. I’ll check to see whats going down there and what spark rev they are shipping.

2 Likes
3

@magicaltrout bump

@hecklerchris follow this thread to find out when it happens

4

What will make you a bit sad is the Spark bigtop release is 2.2.1 I believe.

Bear with me just been catching up post Big Data London but I’d like to put some cycles into figuring out where we’re at with the Bigtop 1.3.0 release and what we can do about a newer spark version as that is now 12 months old.

5

It would be awesome if we could get spark bumped up to 2.4 (latest). Thanks!

1 Like
6

@kwmonroe @hecklerchris @magicaltrout

I’m thinking what this means for juju deployed spark is that we can easily build a spark k8s charm by using base images e.g. gcr.io/spark-operator/spark:v{2.2x, 2.3.x,2.4.x} or our own custom images built FROM the desired upstream ones.

Possibly this is a way to go about the BigTop Vs. Spark 2.4 issue going forward.

7

:open_mouth: the lightbulbs are really starting to illuminate with what could be done in this direction with the onset of k8s charms

Pretty good idea going here for a pyspark/jupyter notebook image minus the mesos bits https://github.com/jupyter/docker-stacks/blob/master/pyspark-notebook/Dockerfile …

The k8s charms allow us to tie into the whole ecosystem of spark/bigdata stuff coming to fruition in the greater community but from a juju perspective!

8

Yeah as a quick win, and a good way to test the K8S stuff, I personally think a K8S Spark charm complements spark well. Although I’d like to get non - K8S done also, as there’s plenty of folk who don’t have or want to run a K8S cluster for spark jobs.

9

I’ve got a few hours this evening and fancy a change, I’ll see what I can prototype for the Spark K8S stack.

10

See the latest comment on the Jira ticket above … it doesn’t look like there is a way to deviate from the bigtop release of spark here in the context of a traditional charm considering the current build chain process.

11

Yeah there’s a few options though aren’t there.

a) K8S spark charm
b) Non bigtop Spark in a native charm
c) Bump spark to 2.x in bigtop, get it in and then build some edge charms off of the Bigtop master branch.

And in reality, the answer could be all of them :wink: