Filebeat subordinate charm: use with existing Elastic/Logstash

charm
, ,
1

Hi,

i deployed a ubuntu VM and added a filebeat client: my target was to use it with existing Elastic and Logstash infra. At the moment the charm is stuck waiting for elasticsearch, logstash or kafka charms, which are out of juju control. Is there a way to mark the relation clean/down and inject the configuration it expects somehow? for instance inject elastic.mydomain:9200 as target for the filebeat agent.

Thank you

App       Version  Status   Scale  Charm     Channel  Rev  Exposed  Message
filebeat  6.8.23   waiting      1  filebeat             1  no       Waiting for: elasticsearch, logstash or kafka.
ubuntu    20.04    active       1  ubuntu               1  no

Unit           Workload  Agent  Machine  Public address  Ports  Message
ubuntu/1*      active    idle   1        10.10.4.10
  filebeat/1*  waiting   idle            10.10.4.10           Waiting for: elasticsearch, logstash or kafka.
2

I don’t know about filebeat, but we moved from it to using Fluentbit, and we charmed it: Deploy Fluentbit using Charmhub - The Open Operator Collection . With this charm, you can pass the configuration directly without the need for a relation.

For example, this is what we use to forward logs to a graylog server, that do not have a relation definition:

$ cat config
[{"output": [["name", "gelf"],
             ["match", "*"],
             ["host", "10.107.185.90"],
             ["port", "12201"],
             ["mode", "udp"],
 {"input": [("name",     "tail"),
            ("path",     "/var/log/foo/bar.log"),
            ("path_key", "filename"),
            ("tag",      "foo")}]

$ juju config fluentbit custom-config="$(cat config)"
3

thank you, but actually we should use the filebeat in order to have a full elastic support

4

Well, Fluent Bit can forward logs to an ElasticSearch instance: Elasticsearch - Fluent Bit: Official Manual

But I think you can specify a logstash host via charm configuration: Deploy Filebeat using Charmhub - The Open Operator Collection. Something like juju config filebeat logstash-host="the.server.com:1234"

5

with fluent bit it looks like i can only set 1 host, while i would like to pass a list.

For the filebeat charm

  1. it get stuck as it’s waiting a relation to a master charm like elasticsearch server
  2. i can set logstash as you mentioned but not directly the elastic servers list
6

You can have multiple inputs/outputs with FluentBit:

$ cat config
[{"output": [["name", "gelf"],
             ["match", "*"],
             ["host", "10.107.185.90"],
             ["port", "12201"],
             ["mode", "udp"]},
 {"output": [["name", "es"],
             ["match", "*"],
             ["host", "10.107.185.123"],
             ["port", "9200"],
             ["index", "my_index"],
             ["type", "my_type"]},
 {"input": [("name",     "tail"),
            ("path",     "/var/log/foo/bar.log"),
            ("path_key", "filename"),
            ("tag",      "foo")},
 {"input": [("name",     "tail"),
            ("path",     "/var/log/baz.log"),
            ("path_key", "filename"),
            ("tag",      "foo")}]

$ juju config fluentbit custom-config="$(cat config)"

I think you would need multiple outputs, one for each ES server.

I can’t help much with filebeat, we replaced it with fluentbit due to its many limitations. Did you try to contact the development team?

7

not yet, but i will.

Thank you for your suggestions