Loki crashing on docker

[imclint21]

Describe the bug
The instance crash after few hours

Configuration:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

storage_config:
  filesystem:
    directory: /loki/

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  max_query_lookback: 2100h
  retention_period: 2100h

compactor:
  working_directory: /loki/retention
  delete_request_store: filesystem
  retention_enabled: true

ruler:
  alertmanager_url: http://localhost:9093

analytics:
 reporting_enabled: false

Logs I get:

[chaudum]

Hi @imclint21
Thanks for reporting. Can you also post the full docker run command the the startup logs?

[imclint21]

Here you go @chaudum I exported the whole logs.
loki.csv

I restarted the instance for now it seems UP I'll wait a bit.

[imclint21]

Okay no it again crashed, and this time I don't have logs.

The only things I can say it's it crash when I restart promtail on all my servers:

May 15 12:26:34 fra3 promtail[30227]: level=warn ts=2024-05-15T10:26:34.891870836Z caller=client.go:419 component=client host=loki.xxx msg="error sending batch, will retry" status=429 tenant= error="server returned HTTP status 429 Too Many Requests (429): Ingestion rate limit exceeded for user fake (limit: 4194304 bytes/sec) while attempting to ingest '4383' lines totaling '1048442' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased"
May 15 12:26:37 fra3 promtail[30227]: level=warn ts=2024-05-15T10:26:37.652537111Z caller=client.go:419 component=client host=loki.xxx msg="error sending batch, will retry" status=429 tenant= error="server returned HTTP status 429 Too Many Requests (429): Ingestion rate limit exceeded for user fake (limit: 4194304 bytes/sec) while attempting to ingest '4380' lines totaling '1048435' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased"
May 15 12:26:39 fra3 promtail[30227]: level=warn ts=2024-05-15T10:26:39.696488947Z caller=client.go:419 component=client host=loki.xxx msg="error sending batch, will retry" status=429 tenant= error="server returned HTTP status 429 Too Many Requests (429): Ingestion rate limit exceeded for user fake (limit: 4194304 bytes/sec) while attempting to ingest '4386' lines totaling '1048532' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased"

[chaudum]

After some testing I could not reproduce this issue yet.

[imclint21]

I can propose you an access to our NAS if you want!

I guess it's related to compactor but I'm not sure, we have like 1GB currently in /loki.