INVALID COPIES for chunk

[struthio]

hi,

Recently I wanted to access files that were stored in MFS and haven't been touched for longer time. I was surprised to see that some of chunks got error:

<file>: chunk 0: 000000000006844D_00000002 / (id:427085 ver:2) copy 1: 192.168.X.Y:9422 (status:INVALID) copy 2: 192.168.X.Z:9422 (status:INVALID) no valid copies !!!

How it is possible that two chunks on two different servers have 'INVALID COPIES' status ?

[oszafraniec]

Read #106 because the data can be still there to restore by hand. Don't try to repair this file with mfsfilerepair!!! Read #106 first!

[struthio]

Thanks for warning. Luckily those end up to be less important files (those that can be recreated somehow).
But question is how two copies lying on two different machines might get corrupted ?
I thought that reason of having more than one copy of chunk is to avoid this situation, if one is invalidated, then second one can be used to repair first one.

[chogata]

It's a question of probability. MooseFS checks copies correctness - periodically. If a copy becomes invalid for some reason (broken HDD, faulty RAM, power surge in your server room, any number of similar reasons), it will be detected - eventually. If this chunk is used for i/o, it will be detected as soon as the chunk is accessed. If it is not used, it will be detected in the scrubbing process, but that is slow and depends on the total number of chunks on your system. So if another copy is corrupted before the first one is detected as invalid, then yes, both will be lost.
That's why we suggest to all our clients to keep their data in at least 3 copies - the probability of loosing all 3 drops dramatically. Also, it's a good idea to buy hard drives from different production series (famous problems of 2-disk raids, when both disk were bought at the same time, from the same manufacturer, and their serial number were only 1 digit apart - when one broke, half the time the other broke within DAYS) and to monitor their SMART readings.
Also, if at any point any of your machines suffered from power loss or another similar event that caused the machines to shut down dirty and you did not check your disks (fsck or equivalent for the drive, followed by mfschunktool) after starting it up and before starting the chunk server process again, you might have had those invalid chunks since then.

TL;DR: if a long unaccessed copy of a chunk becomes corrupted due to outside hardware factors, MooseFS doesn't learn about it instantly, some time needs to pass first. The probability of two copies of the same chunk becoming corrupted before detection, while very low, does not equal zero. Each additional copy lowers the probability by an order of magnitude.