HELK with Kafka (not winlogbeats but Filebeat)

[mvasuraja]

Describe the problem

I am running helk with install option #4. IP of machine is 10.180.7.188, with all settings default.

I am able to send my system logs generated on localhost, picked up by filebeat, to elastic/ kibana.

What I would want is to have the logs come through Kafka (Filebeat-> Kafka-> Logstash(optional)->Elastic/Kibana)

So I go to Kafka bash as given here:
https://thehelk.com/how-to/kafka/topic-ingestion.html
and run the commands on the bash.

I get the following error:

Connection to node -1 (helk-kafka-broker/172.20.0.10:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

What do I need to do, to get the filebeat logs into Kafka and then into Kibana (logstash- optional)?
Idea is to have filebeat eventually pick up logs from Zeek, and push into Kibana via Kafka.

cat /etc/os-release
Ubuntu (Jammy) 22.04.2

echo -e "\nDocker Space:" && df -h /var/lib/docker; echo -e "\nMemory:" && free -g; echo -e "\nCores:" && getconf _NPROCESSORS_ONLN
/dev/sda1 916G 669G 201G 77% /
Mem: 62 42 1 0 17 18
Cores:
16

Get output of the HELK docker containers:
docker ps --filter "name=helk"

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0fcdd775df83 confluentinc/ksqldb-cli:latest "/bin/sh" 3 weeks ago Up 3 weeks helk-ksql-cli 06610d2dc968 confluentinc/ksqldb-server:latest "/usr/bin/docker/run" 3 weeks ago Up 3 weeks 0.0.0.0:8088->8088/tcp, :::8088->8088/tcp helk-ksql-server 175cc81b6a35 otrf/helk-spark-worker:2.4.5 "./spark-worker-entr…" 3 weeks ago Up 3 weeks helk-spark-worker d4e9037b8f2c otrf/helk-kafka-broker:2.4.0 "./kafka-entrypoint.…" 3 weeks ago Up 3 weeks 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp helk-kafka-broker 0ed243275620 docker_helk-jupyter "/opt/jupyter/script…" 3 weeks ago Up 3 weeks 8000/tcp, 8888/tcp helk-jupyter 987442d5f4aa otrf/helk-spark-master:2.4.5 "./spark-master-entr…" 3 weeks ago Up 3 weeks 7077/tcp, 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp helk-spark-master 3d1965622f31 otrf/helk-zookeeper:2.4.0 "./zookeeper-entrypo…" 3 weeks ago Up 24 hours 2181/tcp, 2888/tcp, 3888/tcp helk-zookeeper 9a417c2b8c46 otrf/helk-elastalert:latest "./elastalert-entryp…" 3 weeks ago Up 3 weeks helk-elastalert 0609c98210b0 otrf/helk-logstash:7.6.2.1 "/usr/share/logstash…" 3 weeks ago Up 3 weeks 0.0.0.0:3515->3515/tcp, :::3515->3515/tcp, 0.0.0.0:5044->5044/tcp, :::5044->5044/tcp, 0.0.0.0:5514->5514/tcp, 0.0.0.0:5514->5514/udp, :::5514->5514/tcp, :::5514->5514/udp, 0.0.0.0:8515-8516->8515-8516/tcp, :::8515-8516->8515-8516/tcp, 0.0.0.0:8531->8531/tcp, :::8531->8531/tcp, 0.0.0.0:8515-8516->8515-8516/udp, :::8515-8516->8515-8516/udp, 9600/tcp helk-logstash cc41c014c934 otrf/helk-nginx:0.3.0 "/opt/helk/scripts/n…" 3 weeks ago Up 3 weeks 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp helk-nginx 9d753b0b3695 docker.elastic.co/kibana/kibana:7.6.2 "/usr/share/kibana/s…" 3 weeks ago Up 3 weeks 0.0.0.0:5601->5601/tcp, :::5601->5601/tcp helk-kibana e06b4115ef16 docker.elastic.co/elasticsearch/elasticsearch:7.6.2 "/usr/share/elastics…" 3 weeks ago Up 3 weeks 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp helk-elasticsearch

HELK version:

ad752b2 (HEAD -> master, origin/master, origin/HEAD) Update jvm.options (#563)

Thanks & Regards,