Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loki ruler SERVER_ERROR out of memory storing object warnings #3047

Open
slimes28 opened this issue Mar 28, 2024 · 0 comments
Open

Loki ruler SERVER_ERROR out of memory storing object warnings #3047

slimes28 opened this issue Mar 28, 2024 · 0 comments

Comments

@slimes28
Copy link

Hello,
I am using loki-distributed helm chart with configured memcachedChunks, memcachedFrontend and memcachedIndexQueries but the ruler has a lot of warnings: memcache: unexpected response line from \"set\": \"SERVER_ERROR out of memory storing object\\r\\n\.

Here is how memcached is configured:

         memcachedChunks:
            enabled: true
            extraArgs:
              - -m 2048
              - -I 2m
              - -v
            resources:
              limits:
                cpu: "2"
                memory: 3Gi
              requests:
                cpu: 500m
                memory: 3Gi

          memcachedFrontend:
            enabled: true
            resources:
              limits:
                cpu: 100m
                memory: 200Mi
              requests:
                cpu: 10m
                memory: 10Mi

          memcachedIndexQueries:
            enabled: true
            resources:
              limits:
                cpu: "1"
                memory: 200Mi
              requests:
                cpu: 100m
                memory: 100Mi
       
          loki:
            structuredConfig:
              storage_config:
                index_queries_cache_config:
                  memcached:
                    batch_size: 100
                    parallelism: 100
                  memcached_client:
                    consistent_hash: true

              chunk_store_config:
                chunk_cache_config:
                  memcached:
                    batch_size: 100
                    parallelism: 100
                  memcached_client:
                    consistent_hash: true
                    timeout: 60s
                  background:
                    writeback_goroutines: 1
                    writeback_buffer: 1000
                    writeback_size_limit: 500MB

              query_range:
                parallelise_shardable_queries: true
                results_cache:
                  cache:
                    memcached_client:
                      consistent_hash: true
                      max_idle_conns: 16
                      timeout: 500ms
                      update_interval: 1m
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant