docs/elasticsearch/

[giscus[bot]]

docs/elasticsearch/

How to deploy Elasticsearch and Kibana in our Pi Kuberentes cluster.

https://picluster.ricsanfre.com/docs/elasticsearch/

[Hubert2208]

Thanks for your tutorial.
One small thing: in the manifest-file for kibana is the referenz for elasticsearch wrong. Have to be "efk" like in the manifest for elasticsearch.

[ricsanfre]

Hi @Hubert2208,

You are right. Thanks for the heads up. I have already fixed it.

Regards
Ricardo

[sbatururimi]

1. @ricansfree, could you add a section or help to get logs in kibana from a custom deployed pod, for example from /var/log/?
2. secondly, is there soething that should be done to clean automatically elasticsearch? Looks like my 5gb on longhorn filled and then the efk startes to crash till the volume is incremented

[sbatururimi]

wrong place and can't remove

[i5Js]

Thanks!

When I try to create a data view in kibana, it doesn't find fluentd-* or logstash-*

Regards,

[i5Js]

Hi,

I have just followed the tutorial, I don't know if I need to do something else before.

Thanks!

[ricsanfre]

Hi,
The logging tutorial is compose of 3 different parts as it is explaining in https://picluster.ricsanfre.com/docs/logging/

1. Deploy ES and Kibana
2. Loki Installation
3. Fluentbit/Fluentd installation

Kibana data view configuration can be only completed when step 3 is finished. Currently there is a note in the documentation indicating that data need to be inserted before configuring the data view.

If it is not enough clear, I can add a reference to the fluentbit/fluentd installation page.

Thanks

[i5Js]

Yeah, I figured it out later... I'll follow the step 3 :)

[i5Js]

After following all steps, I can't logstash-* in kibana, only fluentd-*

[i5Js]

Checking the logs in the pod, I see the. following:
[2024-03-29T06:12:19.132+00:00][INFO ][plugins.securitySolution.endpoint:metadata-check-transforms-task:0.0.1] no endpoint installation found

I don't know if it's related.

[ilbarone87]

Hi @ricsanfre, thank you for your tutorial, I rebased all my GitOps strategy thanks to you. I recently deployed the ELK stack, do you have any good recommendations for kibana dashboard or some templates to share? I'd like to separate the logs for the hosts and kubernetes, and have something for the errors with some nice visualiser (eg pie, table, metrics, etc...)

[ricsanfre]

Thanks @ilbarone87.

Currently I am not using any Kibana dashboard in my implementation. I am only using Kibana's logs Discover features for doing deep analysis of the logs. I am using Grafana as main reporting component where I can define dashboards containing information from logs (Loki), metrics (Prometheus) and traces (Tempo).

[ilbarone87]

@ricsanfre after ArgoCD in my cluster updated to 2.10.4 elasticsearch applications was stuck in progressing status because the ES cluster was in yellow state, given the single replica deployed. So had to up the replicas to 2. Do you have the same issue? Do you know by any chance a workaround for it?

[ricsanfre]

Hi @ilbarone87. I am not facing that issue.
My elasticsearh cluster is in 'green' state using just one cluster node. which eck-operator version are you using?

[ilbarone87]

8.12 for elastic and 2.11.1 eck-operator

[ricsanfre]

I was using an old version of elastic stack (8.6). After upgrading my Elastic stack to 8.13 I am still not having that issue. Cluster is in 'green' state using just one single node.

[sbatururimi]

@ricansfree,

1. could you add a section or help to get logs in kibana from a custom deployed pod, for example from /var/log/?

2. secondly, is there something that should be done to clean automatically elasticsearch? Looks like my 5gb on longhorn filled and then the efk starts to crash till the volume is incremented