[loki-distributed] extraVolumes and extraVolumeMounts for memcached statefulsets

[justinbwood]

Currently in the loki-distributed Helm chart, the memcached-chunks service is the only memcached statefulset that supports extraVolumeMounts, and none of the memcached statefulsets support extraVolumes

Example of mounting a TLS cert to the ingester service:

ingester:
  volumes:
    - name: loki-rsa-cert
      secret:
        secretName: loki-rsa-cert
  volumeMounts:
    - mountPath: /var/loki-tls
      name: loki-rsa-cert
      readOnly: true

I'd like to mount a cert-manager certificate secret as a volume to all the memcached pods to simplify the TLS setup, but I don't see a way of doing so. Loki's memcached_client config only supports file paths for TLS, so I can't inject the cert directly as an environment variable.

https://grafana.com/docs/loki/latest/configure/#cache_config

I'd like to be able to use the secret to configure memcached:

memcached_client:
  # ...
  tls_enabled: true
  tls_cert_path: "/var/loki-tls/tls.crt"
  tls_key_path: "/var/loki-tls/tls.key"
  tls_ca_path: "/var/loki-tls/ca.crt"
  tls_server_name: "loki.svc.cluster.local"