Corrupt Ingester pod name in the ring

[rishabhkumar92]

Describe the bug

We encounted a incident where we saw corrupt Ingester pod name in the ring for couple of pods.
There was character flip(c-k) in the name of pod.

To Reproduce

Steps to reproduce the behavior:

Reproduction of issue might be an issue as it was one off behavior, describing version and other details:

1. Mimir version - 2.10.5
2. Distributor/Rulers saw corrupt pod name in the ring and we saw some % of write and read failure which was expected.

Expected behavior

Pod name should be correct.

Environment

• Infrastructure: Kubernetes

Additional Context

We uses memberlist which uses gossip based protocol for discovering pods. During time of corruption we did see some degraded probes(counter_memberlist_degraded_probe) around that time but that doesn't explain why pod were named in wrong way.

e.g: we say teleskope-kortex-ingester-zone-c-p-sequoia-204, correct name should be telescope-cortex-ingester-zone-c-p-sequoia-204

[rishabhkumar92]

Attaching screenshot describing memberlist tcp packet received errors around that time. cc: @krajorama