Filebeat and elasticsearch version

1

I deploy graylog+elasticsearch+filebeat,but graylog can’t riceive logs from filebeat.
Then I deploy rsyslog+rsyslog-ha-forward,then I the graylog recivied logs.
The log is:

juju-0de0d7-1-lxd-0 filebeat[554771]: ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://10.0.9.3:9200)): Connection marked as failed because the onConnect callback failed: This Beat requires the default distribution of Elasticsearch. Please install the default distribution of Elasticsearch from elastic.co, or install the oss-only distribution of beats

juju status
\Model    Controller       Cloud/Region    Version  SLA          Timestamp
graylog  maas-controller  mymaas/default  2.8.10   unsupported  20:31:19+08:00

SAAS             Status  Store            URL
primary-rsyslog  active  maas-controller  admin/rsyslog.primary-rsyslog

App                   Version  Status       Scale  Charm                 Store       Channel  Rev  OS      Message
apache2                        unknown          1  apache2               charmstore            35  ubuntu
elasticsearch         5.6.16   active           1  elasticsearch         charmstore            49  ubuntu  Unit is ready
filebeat              6.8.16   active           1  filebeat              charmstore            33  ubuntu  Filebeat ready.
grafana                        active           1  grafana               charmstore            40  ubuntu  Started grafana-server
graylog               2.5.1    active           1  graylog               local                  0  ubuntu  Ready with: elasticsearch, mongodb
mongodb               3.6.8    active           1  mongodb               charmstore            59  ubuntu  Unit is ready
prometheus2                    active           1  prometheus2           charmstore            22  ubuntu  Ready
rsyslog-forwarder-ha           maintenance      1  rsyslog-forwarder-ha  charmstore            20  ubuntu  installing charm software
telegraf                       active           4  telegraf              charmstore            41  ubuntu  Monitoring apache2/0 (source version/commit dec0633)

Unit                       Workload     Agent      Machine  Public address  Ports                                    Message
apache2/0*                 unknown      idle       2        10.0.0.157      80/tcp
  telegraf/0*              active       idle                10.0.0.157      9103/tcp                                 Monitoring apache2/0 (source version/commit d                          ec0633)
elasticsearch/0*           active       idle       1        10.0.9.3        9200/tcp                                 Unit is ready
grafana/0*                 active       idle       2/lxd/0  10.0.3.118      3000/tcp                                 Started grafana-server
  telegraf/1               active       idle                10.0.3.118      9103/tcp                                 Monitoring grafana/0 (source version/commit d                          ec0633)
graylog/0*                 active       idle       0        10.0.9.13       9000/tcp,9001/tcp                        Ready with: elasticsearch, mongodb
mongodb/0*                 active       idle       1/lxd/0  10.0.3.115      27017/tcp,27019/tcp,27021/tcp,28017/tcp  Unit is ready
  filebeat/0*              active       idle                10.0.3.115                                               Filebeat ready.
  rsyslog-forwarder-ha/0*  maintenance  executing           10.0.3.115                                               (install) installing charm software
  telegraf/2               active       idle                10.0.3.115      9103/tcp                                 Monitoring mongodb/0 (source version/commit d                          ec0633)
prometheus2/0*             active       idle       2/lxd/1  10.0.3.119      9090/tcp,12321/tcp                       Ready
  telegraf/3               active       idle                10.0.3.119      9103/tcp                                 Monitoring prometheus2/0 (source version/comm                          it dec0633)

Machine  State    DNS         Inst id              Series  AZ       Message
0        started  10.0.9.13   vm-159-1             focal   default  Deployed
1        started  10.0.9.3    vm-156-1             focal   default  Deployed
1/lxd/0  started  10.0.3.115  juju-0de0d7-1-lxd-0  focal   default  Container started
2        started  10.0.0.157  node6                focal   default  Deployed
2/lxd/0  started  10.0.3.118  juju-0de0d7-2-lxd-0  focal   default  Container started
2/lxd/1  started  10.0.3.119  juju-0de0d7-2-lxd-1  focal   default  Container started

root@vivodo-3:~#  juju add-relation rsyslog-forwarder-ha primary-rsyslog
root@vivodo-3:~#
root@vivodo-3:~#
root@vivodo-3:~# juju status
Model    Controller       Cloud/Region    Version  SLA          Timestamp
graylog  maas-controller  mymaas/default  2.8.10   unsupported  20:31:30+08:00

SAAS             Status  Store            URL
primary-rsyslog  active  maas-controller  admin/rsyslog.primary-rsyslog

App                   Version  Status       Scale  Charm                 Store       Channel  Rev  OS      Message
apache2                        unknown          1  apache2               charmstore            35  ubuntu
elasticsearch         5.6.16   active           1  elasticsearch         charmstore            49  ubuntu  Unit is ready
filebeat              6.8.16   active           1  filebeat              charmstore            33  ubuntu  Filebeat ready.
grafana                        active           1  grafana               charmstore            40  ubuntu  Started grafana-server
graylog               2.5.1    active           1  graylog               local                  0  ubuntu  Ready with: elasticsearch, mongodb
mongodb               3.6.8    active           1  mongodb               charmstore            59  ubuntu  Unit is ready
prometheus2                    active           1  prometheus2           charmstore            22  ubuntu  Ready
rsyslog-forwarder-ha           maintenance      1  rsyslog-forwarder-ha  charmstore            20  ubuntu  installing charm software
telegraf                       active           4  telegraf              charmstore            41  ubuntu  Monitoring apache2/0 (source version/commit dec0633)

Unit                       Workload     Agent      Machine  Public address  Ports                                    Message
apache2/0*                 unknown      idle       2        10.0.0.157      80/tcp
  telegraf/0*              active       idle                10.0.0.157      9103/tcp                                 Monitoring apache2/0 (source version/commit d                          ec0633)
elasticsearch/0*           active       idle       1        10.0.9.3        9200/tcp                                 Unit is ready
grafana/0*                 active       idle       2/lxd/0  10.0.3.118      3000/tcp                                 Started grafana-server
  telegraf/1               active       idle                10.0.3.118      9103/tcp                                 Monitoring grafana/0 (source version/commit d                          ec0633)
graylog/0*                 active       idle       0        10.0.9.13       9000/tcp,9001/tcp                        Ready with: elasticsearch, mongodb
mongodb/0*                 active       idle       1/lxd/0  10.0.3.115      27017/tcp,27019/tcp,27021/tcp,28017/tcp  Unit is ready
  filebeat/0*              active       idle                10.0.3.115                                               Filebeat ready.
  rsyslog-forwarder-ha/0*  maintenance  executing           10.0.3.115                                               (config-changed) installing charm software
  telegraf/2               active       idle                10.0.3.115      9103/tcp                                 Monitoring mongodb/0 (source version/commit d                          ec0633)
prometheus2/0*             active       idle       2/lxd/1  10.0.3.119      9090/tcp,12321/tcp                       Ready
  telegraf/3               active       idle                10.0.3.119      9103/tcp                                 Monitoring prometheus2/0 (source version/comm                          it dec0633)

Machine  State    DNS         Inst id              Series  AZ       Message
0        started  10.0.9.13   vm-159-1             focal   default  Deployed
1        started  10.0.9.3    vm-156-1             focal   default  Deployed
1/lxd/0  started  10.0.3.115  juju-0de0d7-1-lxd-0  focal   default  Container started
2        started  10.0.0.157  node6                focal   default  Deployed
2/lxd/0  started  10.0.3.118  juju-0de0d7-2-lxd-0  focal   default  Container started
2/lxd/1  started  10.0.3.119  juju-0de0d7-2-lxd-1  focal   default  Container started

The question is how can I degrade the beatfile version to oss-only version?

Thank you.

2

I am curious if this is related to the version of filebeat being 6.x or the relation model you’re using, or perhaps an incidence of not collecting the syslog file.

In case the issue really is just the filebeat version, I’ve included the install_sources and install_keys to pick up the elastic.co 5.x version in the below example bundle.

Here is some additional information about Filebeat, Graylog, Elasticsearch and how it differs from the standard ELK bundle.

Graylog is a non-data-holding member of the elasticsearch cluster and is the ingestion point of logstash data when it’s used as part of the filebeat/elasticsearch environment. It is not possible for Graylog to index or access data within elasticsearch that it didn’t ingest and index within ES itself, so all data from filebeat must pass through Graylog to be stored in Elasticsearch in a format Graylog understands.

This is opposed to the typical ELK stack where filebeat feeds and manages indicies in elasticsearch and kibana queries the filebeat indicies from elasticsearch.

A bundle snippet for a Graylog implementation would look like the following (assuming you want to collect logs from the “mongodb” application. Note the inclusion of graylog-mongodb as the backend for the graylog configuration database. While this could use your mongodb application, it may be better to not mix graylog’s backend configuration database with your non-infrastructure-supporting mongodb application.

applications:
  graylog:
    charm: cs:graylog
    num_units: 1
  graylog-mongodb:
    charm: cs:mongodb
    num_units: 1
  filebeat:
    charm: cs:filebeat
      options:
        install_keys: |-
        - |
          -----BEGIN PGP PUBLIC KEY BLOCK-----
          Version: SKS 1.1.6
          Comment: Hostname: keyserver.ubuntu.com

          mQENBFI3HsoBCADXDtbNJnxbPqB1vDNtCsqhe49vFYsZN9IOZsZXgp7aHjh6CJBDA+bGFOwy
          hbd7at35jQjWAw1O3cfYsKAmFy+Ar3LHCMkV3oZspJACTIgCrwnkic/9CUliQe324qvObU2Q
          RtP4Fl0zWcfb/S8UYzWXWIFuJqMvE9MaRY1bwUBvzoqavLGZj3SF1SPO+TB5QrHkrQHBsmX+
          Jda6d4Ylt8/t6CvMwgQNlrlzIO9WT+YN6zS+sqHd1YK/aY5qhoLNhp9G/HxhcSVCkLq8SStj
          1ZZ1S9juBPoXV1ZWNbxFNGwOh/NYGldD2kmBf3YgCqeLzHahsAEpvAm8TBa7Q9W21C8vABEB
          AAG0RUVsYXN0aWNzZWFyY2ggKEVsYXN0aWNzZWFyY2ggU2lnbmluZyBLZXkpIDxkZXZfb3Bz
          QGVsYXN0aWNzZWFyY2gub3JnPokBOAQTAQIAIgUCUjceygIbAwYLCQgHAwIGFQgCCQoLBBYC
          AwECHgECF4AACgkQ0n1mbNiOQrRzjAgAlTUQ1mgo3nK6BGXbj4XAJvuZDG0HILiUt+pPnz75
          nsf0NWhqR4yGFlmpuctgCmTD+HzYtV9fp9qW/bwVuJCNtKXk3sdzYABY+Yl0Cez/7C2GuGCO
          lbn0luCNT9BxJnh4mC9h/cKI3y5jvZ7wavwe41teqG14V+EoFSn3NPKmTxcDTFrV7SmVPxCB
          cQze00cJhprKxkuZMPPVqpBS+JfDQtzUQD/LSFfhHj9eD+Xe8d7sw+XvxB2aN4gnTlRzjL1n
          TRp0h2/IOGkqYfIG9rWmSLNlxhB2t+c0RsjdGM4/eRlPWylFbVMc5pmDpItrkWSnzBfkmXL3
          vO2X3WvwmSFiQbkBDQRSNx7KAQgA5JUlzcMW5/cuyZR8alSacKqhSbvoSqqbzHKcUQZmlzNM
          KGTABFG1yRx9r+wa/fvqP6OTRzRDvVS/cycws8YX7Ddum7x8uI95b9ye1/Xy5noPEm8cD+hp
          lnpU+PBQZJ5XJ2I+1l9Nixx47wPGXeClLqcdn0ayd+v+Rwf3/XUJrvccG2YZUiQ4jWZkoxsA
          07xx7Bj+Lt8/FKG7sHRFvePFU0ZS6JFx9GJqjSBbHRRkam+4emW3uWgVfZxuwcUCn1ayNgRt
          KiFv9jQrg2TIWEvzYx9tywTCxc+FFMWAlbCzi+m4WD+QUWWfDQ009U/WM0ks0KwwEwSk/UDu
          ToxGnKU2dQARAQABiQEfBBgBAgAJBQJSNx7KAhsMAAoJENJ9ZmzYjkK0c3MIAIE9hAR20mqJ
          WLcsxLtrRs6uNF1VrpB+4n/55QU7oxA1iVBO6IFu4qgsF12JTavnJ5MLaETlggXY+zDef9sy
          TPXoQctpzcaNVDmedwo1SiL03uMoblOvWpMR/Y0j6rm7IgrMWUDXDPvoPGjMl2q1iTeyHkMZ
          EyUJ8SKsaHh4jV9wp9KmC8C+9CwMukL7vM5w8cgvJoAwsp3Fn59AxWthN3XJYcnMfStkIuWg
          R7U2r+a210W6vnUxU4oN0PmMcursYPyeV0NX/KQeUeNMwGTFB6QHS/anRaGQewijkrYYoTNt
          fllxIu9XYmiBERQ/qPDlGRlOgVTd9xUfHFkzB52c70E=
          =92oX
          -----END PGP PUBLIC KEY BLOCK-----
      install_sources: |
        - 'deb https://artifacts.elastic.co/packages/5.x/apt stable main'
      logpath: /var/log/*.log /var/log/*/*.log /var/log/syslog

  mongodb:
    charm: cs:mongodb
    num_units: 1
  elasticsearch:
    charm: cs:elasticsearch
    num_units: 1

relations:
- - filebeat:beats-host
  - mongodb:juju-info
- - graylog:beats
  - filebeat:logstash
- - graylog:elasticsearch
  - elasticsearch:client
- - graylog:mongodb
  - graylog-mongodb:database
3

Thank you@ afreiberger。

Your suggestion is so nice and reasonable ,I will try it.

1 Like
4

Hi yanxiaomu,

I have that stack deployed successfully. Filebeat will use default install_sources
deb https://artifacts.elastic.co/packages/6.x/apt stable main

But elasticsearch is using default install_source
deb https://artifacts.elastic.co/packages/5.x/apt stable main

You need to define install_source to 6.x for elasticsearch to be in compatible version with filebeat, https://www.elastic.co/support/matrix#matrix_compatibility

juju deploy cs:elasticsearch-49 --to 6 --series focal --config apt-repository=“deb https://artifacts.elastic.co/packages/6.x/apt stable main”

You dont need relation between elasticsearch and filebeat but you need relation between filebeat and graylog:
juju add-relation filebeat:logstash graylog:beats

And of course you need relation between graylog and elasticsearch.

I hope this helps you.
Regards,
Bane

1 Like
5

Thank you a lot@ branislav.neskovic.

Your said is so helpful , I will try it.

I deploy fellow your said and success,but with a little difference :
juju deploy cs:elasticsearch-49 --to 1 --series focal --config apt-repository=“deb https://artifacts.elastic.co/packages/6.x/apt stable main”
ERROR unrecognized args: [“stable” “main””]

Then I config elasticsearch fellow Install Elasticsearch 6.x on Ubuntu 18.04 LTS (Bionic Beaver) Linux

juju deploy cs:elasticsearch-49 --to 1 --series focal --config apt-repository="deb https://artifacts.elastic.co/packages/oss-6.x/apt stable main"

Located charm “elasticsearch” in charm-store, revision 49
Deploying “elasticsearch” from charm-store charm “elasticsearch”, revision 49 in channel stable

And Graylog received logs. It is awesome.

Thank you again