A subordinate of a subordinate

framework
1

We’re trying to build a druid config charm, and basically want it to be subordinate and we would also like that charm to utilise the hadoop plugin layer to give us HDFS connectivity…

When you connect the Hadoop plugin to the Druid config charm we get the connection but the plugin isn’t installed.

Is there a technical limitation on having a subordinate of a subordinate or are we just doing something silly?

2

I believe it is the case that each subordinate has to have one direct relation to a principal (non-subordinate) in order to be installed. IIUC, this is because it needs to know what machine to install to, and the machine info isn’t available in the subordinate record information, so it would have to walk up the relation tree to find the info.

You could use the juju-info relation that all charms implicitly support to connect the Hadoop Plugin to the principal in addition to the normal relation to the Druid config charm. The Druid charm would then get the expected data on the normal relation and the Hadoop Plugin would be installed on the machine due to the juju-info relation.

I’ve run into this before as well, and would like to see “nested subordinates” supported, but I have to admit that it’s a pretty uncommon use-case and would add some complexity to the Juju code, so I don’t expect it to change.

3

I don’t think relating hadoop-plugin to a principal via juju-info is going to help here. The plugin wont do anything until a hadoop-plugin relation is joined.

What you could do is modify the hadoop-client to include a druid relation (like we do with mahout and giraph, deploy it, and then relate both druid and the hadoop-plugin subordinates to the client. You won’t get hadoop-plugin flags in druid, but if necessary, you could also update hadoop-client to set a new flag when both druid and the plugin were related. For example, in hadoop_client.py

    @when('hadoop.installed', 'druid.joined')
    def druid_ready():
        druid = charms.reactive.endpoint_from_flag('druid.joined')
        druid.set_ready() # <-- need a method in interface-druid

The druid charm could then react to whatever flag interface-druid sets that signifies hadoop is ready on the principal.

Of course, now you’re limited to relating druid to hadoop-client things (currently hbase, hive, pig, spark, zeppelin, and of course hadoop-client itself). This may or may not be an issue for you.

4

@kwmonroe My understanding was that the Druid charm has a hadoop-plugin relation to the hadoop-plugin charm, but because Druid is also subordinate, it doesn’t trigger the hadoop-plugin charm install to be started. So doing:

juju add-relation hadoop-plugin:juju-info druids-prinicpal:juju-info

Would trigger the install and the existing hadoop-plugin relation between the Druid and hadoop-plugin charms would satisfy the requirements for the hadoop-plugin charm code.

I’m not saying this is the best approach, but I think it would work without any changes in the code.

5

@magicaltrout Unfortunately, @kwmonroe has shown me how wrong I am. Without an explicit relation endpoint defined in either the principal or hadoop-plugin, the add-relation won’t work, and I’m also now seriously doubting if it would trigger the install either if it wasn’t scope: container as well.

I’m not a big fan of having charm-specific relations in hadoop-client, though, so maybe we can generalize the relation endpoints that mahout and giraph use so that druid could use it as well?

6

Unfortunately, @kwmonroe has shown me how wrong I am.

First of all, let’s frame this. The fact that in 1998, the Undertaker threw Mankind off Hell In A Cell, and plummeted 16 ft through an announcer’s table, is mostly irrelevant. A principal can’t relate to a subordinate that doesn’t define what to do in a juju-info situation:

$ juju relate hadoop-plugin:juju-info ubuntu:juju-info
ERROR no relations found

Subordinates like telegraf and the beats explicitly do something with juju-info. At present, hadoop-plugin does not. That said,

maybe we can generalize the relation endpoints that mahout and giraph use so that druid could use it as well?

The thing about mahout and giraph is that they really don’t care if hadoop is there – they just put jars on the filesystem. In druid’s case, i think we need to know a bit more about the principals that @magicaltrout expects to use. If they’re already hadoop-client charms, easy breezy, update hadoop-client as needed.

If not, I’m +1 on figuring out a generic “passthrough” of hadoop-plugin flags to other subordinates on the same principal – maybe that’s flag-management on principals like hadoop-client, maybe that’s subordinates having access to the principal unitdata.db to discover other subordinate flags, or maybe that’s proper nested subordinates.

7

Didn’t the undertaker just become a Mayor somewhere? Thats epic.

It’s nice to see such a question has generated this discussion :wink:

Basically, and Ryan may correct me if I’m wrong, but we need the various XML config files that hadoop configures itself by and probably some namenode stuff.

In the short term we can flick our charm to non subordinate and have it do its thing, but its a bit of a waste of a box.

8

If it runs in production, and a standalone process, it probably isn’t the worst thing to throw it a box of its own.

Also if resources are scarce, you could deploy it as a principal to a lxd and make the Hadoop-plugin relation there.

1 Like
9

If it can be switched to be a non-subordinate that easily, then it seems like it’s not really modifying the principal and shouldn’t really be a subordinate. If all you care about is saving an instance, placement is exactly what you want, especially with @jamesbeedy’s suggestion of placing it to a lxd container.

1 Like