I Broke It and I Don't Know Where to Start

[MHammett]

I spent some time adding info the setup guide said I should fill out (except for core classifiers, because I didn't understand what I was supposed to do).

I threw some sflow at it, but nothing showed up. Not understanding if I properly added multiple SNMP communities, I removed all but one SNMP community. I downed and uped the system via docker compose, but now three containers keep restarting, one is unhealthy, and the rest are fine. Something is listening on port 8081, but it just throws a 404 error.

I don't know where to begin trying to troubleshoot this.

root@akvorado:~/akvorado# docker ps
CONTAINER ID   IMAGE                               COMMAND                  CREATED          STATUS                          PORTS                                                      NAMES
14f1d9129e27   ghcr.io/akvorado/akvorado:1.8.1     "/usr/local/bin/akvo…"   14 minutes ago   Restarting (1) 17 seconds ago                                                              akvorado_akvorado-console_1
26ed8009aa64   bitnami/kafka:3.3                   "/opt/bitnami/script…"   14 minutes ago   Up 14 minutes (healthy)         9092/tcp                                                   akvorado_kafka_1
02822ee4a4c8   clickhouse/clickhouse-server:22.8   "/entrypoint.sh"         14 minutes ago   Up 14 minutes (unhealthy)       8123/tcp, 9000/tcp, 9009/tcp                               akvorado_clickhouse_1
3f00fa2ccc06   bitnami/redis:7.0                   "/opt/bitnami/script…"   14 minutes ago   Up 14 minutes                   6379/tcp                                                   akvorado_redis_1
b85711e0c3f4   ghcr.io/akvorado/akvorado:1.8.1     "/usr/local/bin/akvo…"   14 minutes ago   Restarting (1) 15 seconds ago                                                              akvorado_akvorado-inlet_1
0c834443f68e   ghcr.io/akvorado/akvorado:1.8.1     "/usr/local/bin/akvo…"   14 minutes ago   Up 14 minutes                                                                              akvorado_akvorado-conntrack-fixer_1
3243ee7f3311   provectuslabs/kafka-ui:v0.5.0       "/bin/sh -c 'java $J…"   14 minutes ago   Up 14 minutes                   8080/tcp                                                   akvorado_kafka-ui_1
a0aea40f42e2   bitnami/zookeeper:3.7               "/opt/bitnami/script…"   14 minutes ago   Up 14 minutes                   2181/tcp, 2888/tcp, 3888/tcp, 8080/tcp                     akvorado_zookeeper_1
be40864582d4   ghcr.io/akvorado/akvorado:1.8.1     "/usr/local/bin/akvo…"   14 minutes ago   Restarting (1) 25 seconds ago                                                              akvorado_akvorado-orchestrator_1
014bca8b77f8   traefik:2.6                         "/entrypoint.sh --ap…"   14 minutes ago   Up 14 minutes                   80/tcp, 127.0.0.1:8080->8080/tcp, 0.0.0.0:8081->8081/tcp   akvorado_traefik_1

[vincentbernat]

docker-compose logs -f should help. Likely something it does not like about the configuration.

[ashortz-carbon]

I'm running into similar behavior with a first time setup on Ubuntu 22.04.

Here's a snippet from docker-compose logs -f:

clickhouse-1                | Downloading ClickHouse init script...
akvorado-orchestrator-1     | Error: invalid configuration:
akvorado-orchestrator-1     | invalid key "geoip"
clickhouse-1                | Downloading ClickHouse init script...
akvorado-orchestrator-1 exited with code 1
akvorado-conntrack-fixer-1  | {"level":"info","id":"cfbac5f929f9d2a7430845bdc49d65393c4c7f2ed689670fff5c507e633f1ebf","from":"ghcr.io/akvorado/akvorado:latest","time":"2024-04-01T22:29:08Z","caller":"akvorado/conntrackfixer/root.go:133","module":"akvorado/conntrackfixer","message":"new container started"}
akvorado-conntrack-fixer-1  | {"level":"info","binding":"2055/udp -> 2055","time":"2024-04-01T22:29:08Z","caller":"akvorado/conntrackfixer/root.go:193","module":"akvorado/conntrackfixer","message":"clear conntrack for UDP port"}
akvorado-conntrack-fixer-1  | {"level":"info","binding":"6343/udp -> 6343","time":"2024-04-01T22:29:08Z","caller":"akvorado/conntrackfixer/root.go:193","module":"akvorado/conntrackfixer","message":"clear conntrack for UDP port"}
akvorado-inlet-1            | Error: unable to fetch configuration file: Get "http://akvorado-orchestrator:8080/api/v0/orchestrator/configuration/inlet": dial tcp: lookup akvorado-orchestrator on 127.0.0.11:53: server misbehaving
akvorado-inlet-1 exited with code 1
akvorado-console-1          | Error: unable to fetch configuration file: Get "http://akvorado-orchestrator:8080/api/v0/orchestrator/configuration/console": dial tcp: lookup akvorado-orchestrator on 127.0.0.11:53: server misbehaving
akvorado-console-1 exited with code 1

It seems the culprit is the orchestrator container's geoip configuration.

[vincentbernat]

Did you follow the quick start documentation to get there?

[ashortz-carbon]

Hi Vincent,

No, I cloned the entire repo and ran docker compose up -d in the docker directory. I should have paid closer attention to the documentation. Thanks for the quick assistance!

[vincentbernat]

Yes, it does not work as the "latest" image does not match the content of the repository. I'll fix that, but it's better to follow the quickstart.

[otronomo]

Hi all,
I found the same problem.
I followed the quickstart guide and the GUI throws a 404.
docker-compose logs -f output is extremely verbose and I honestly don't know where to begin. Any help is much appreciated.

akvorado-akvorado-inlet-1 | {"level":"error","error":"kafka: client has run out of available brokers to talk to: dial tcp 172.19.0.5:9092: connect: connection refused","brokers":"kafka:9092","time":"2024-04-04T15:10:19Z","caller":"akvorado/inlet/kafka/root.go:89","module":"akvorado/inlet/kafka","message":"unable to create async producer"}
akvorado-akvorado-inlet-1 | {"level":"error","error":"kafka: client has run out of available brokers to talk to: dial tcp 172.19.0.5:9092: connect: connection refused","brokers":"kafka:9092","time":"2024-04-04T15:10:23Z","caller":"akvorado/inlet/kafka/root.go:89","module":"akvorado/inlet/kafka","message":"unable to create async producer"}
akvorado-akvorado-inlet-1 | {"level":"error","error":"kafka: client has run out of available brokers to talk to: dial tcp 172.19.0.5:9092: connect: connection refused","brokers":"kafka:9092","time":"2024-04-04T15:10:27Z","caller":"akvorado/inlet/kafka/root.go:89","module":"akvorado/inlet/kafka","message":"unable to create async producer"}
akvorado-akvorado-orchestrator-1 | {"level":"error","error":"dial tcp 172.19.0.3:9000: connect: connection refused","time":"2024-04-04T15:10:17Z","caller":"akvorado/orchestrator/clickhouse/migrations.go:50","module":"akvorado/orchestrator/clickhouse","message":"unable to query database"}
akvorado-akvorado-orchestrator-1 | {"level":"error","error":"unable to query database: dial tcp 172.19.0.3:9000: connect: connection refused","time":"2024-04-04T15:10:17Z","caller":"akvorado/orchestrator/clickhouse/root.go:90","module":"akvorado/orchestrator/clickhouse","message":"database migration error"}
akvorado-akvorado-orchestrator-1 | {"level":"error","error":"dial tcp 172.19.0.3:9000: connect: connection refused","time":"2024-04-04T15:10:18Z","caller":"akvorado/orchestrator/clickhouse/migrations.go:50","module":"akvorado/orchestrator/clickhouse","message":"unable to query database"}
akvorado-akvorado-orchestrator-1 | {"level":"error","error":"unable to query database: dial tcp 172.19.0.3:9000: connect: connection refused","time":"2024-04-04T15:10:18Z","caller":"akvorado/orchestrator/clickhouse/root.go:90","module":"akvorado/orchestrator/clickhouse","message":"database migration error"}
akvorado-akvorado-orchestrator-1 | {"level":"error","error":"kafka: client has run out of available brokers to talk to: dial tcp 172.19.0.5:9092: connect: connection refused","brokers":"kafka:9092","time":"2024-04-04T15:10:19Z","caller":"akvorado/orchestrator/kafka/root.go:67","module":"akvorado/orchestrator/kafka","message":"unable to get admin client for topic creation"}
akvorado-akvorado-orchestrator-1 | {"level":"info","time":"2024-04-04T15:10:22Z","caller":"akvorado/orchestrator/clickhouse/migrations_helpers.go:321","module":"akvorado/orchestrator/clickhouse","message":"raw flows errors view already exists, skip migration"}
akvorado-akvorado-orchestrator-1 | {"level":"error","error":"kafka: client has run out of available brokers to talk to: dial tcp 172.19.0.5:9092: connect: connection refused","brokers":"kafka:9092","time":"2024-04-04T15:10:23Z","caller":"akvorado/orchestrator/kafka/root.go:67","module":"akvorado/orchestrator/kafka","message":"unable to get admin client for topic creation"}
akvorado-akvorado-orchestrator-1 | {"level":"info","time":"2024-04-04T15:10:26Z","caller":"akvorado/orchestrator/clickhouse/migrations_helpers.go:321","module":"akvorado/orchestrator/clickhouse","message":"raw flows errors view already exists, skip migration"}
akvorado-akvorado-orchestrator-1 | {"level":"error","error":"kafka: client has run out of available brokers to talk to: dial tcp 172.19.0.5:9092: connect: connection refused","brokers":"kafka:9092","time":"2024-04-04T15:10:26Z","caller":"akvorado/orchestrator/kafka/root.go:67","module":"akvorado/orchestrator/kafka","message":"unable to get admin client for topic creation"}
akvorado-akvorado-conntrack-fixer-1 | {"level":"error","error":"error while watching for Docker events: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","component":"conntrack-fixer","time":"2024-04-04T15:10:14Z","caller":"akvorado/common/daemon/root.go:68","module":"akvorado/common/daemon","message":"component error, quitting"}
akvorado-akvorado-conntrack-fixer-1 | {"level":"error","error":"error while watching for Docker events: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","time":"2024-04-04T15:10:14Z","caller":"akvorado/cmd/components.go:21","module":"akvorado/cmd","message":"unable to stop component, ignoring"}
akvorado-akvorado-conntrack-fixer-1 | {"level":"error","error":"error while watching for Docker events: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","component":"conntrack-fixer","time":"2024-04-04T15:10:15Z","caller":"akvorado/common/daemon/root.go:68","module":"akvorado/common/daemon","message":"component error, quitting"}