HDD level replication

[alpharde]

Is it possible to have chunks replicated to more than one HDD on the same chunkserver?

[tianon]

I accomplish this in my small two-server, six-drive cluster by running one chunkserver per hard drive (which isn't perfect, but it works pretty well for my use case). I think CHUNKS_UNIQUE_MODE is probably a useful setting here?

moosefs/mfsdata/mfsmaster.cfg.in

Lines 114 to 120 in 023198e

	# avoid using same ip/rack for different chunk copies
	# 0 - ignore ip and rackid (standard behaviour)
	# 1 - avoid storing more than one copy on chunkservers using same IP number
	# 2 - avoid storing more than one copy on chunkservers using IP number from the same rack id
	# CHUNKS_UNIQUE_MODE = 0

[alpharde]

Thanks, I've split some of my HDDs into separate chunkservers and they became new replicas. The "main" chunkserver could have some setting to enable this behavior but for now this works.

[chogata]

What would be the purpose of having copies of the same chunk on one physical chunk server? The purpose of MooseFS is to keep data safe. Copies on different machines means if one fails, the other has the data. Copies on one machine means if this machine fails, all copies fail. Your desire seems counterintuitive to me.

[onlyjob]

Having server as a unit of failure is great, but sometimes there are not enough servers so one might design storage configuration to have a drive bay, disk controller, or even a single HDD as a unit of failure. It is not an ideal (perhaps temporary?) setting, but sometimes still better than RAID. Another case is disproportional storage capacity on one server -- I had exactly that situation with one machine having 3 times more disks than others. Of course, running few chunkservers on different ports does the trick.

[chogata]

We do have some users that have more than one chunk server process on one physical machine, but they have more than one machine also and they mostly want to make sure that copies of one chunk will be on different machines, not on the same one. So they use labels and/or CHUNKS_UNIQUE_MODE =1 setting in mfsmaster.cfg to make sure copies of one chunk will land on different machines, rather than purposefully try to keep 2 chunk copies on the same machine. The only scenario with chunks on the same machine would be if you had less physical machines than the maximum goal of your storage classes. To purposefully keep 2 chunk copies on one machine when it's possible to keep them on 2 machines - this I can't see a scenario for and this is how I understood the OP's question.
Of course it CAN be done: 2 chunk server processes on 1 machine, each with its own hard drive, both labeled and a storage class using those labels. But why? ;)

[eleaner]

I guess the idea was about protection from a failure of the HDD
I am running my home backup machine on moosefs
14HDD, 14chunkservers + one master server, one physical machine
14 chunkservers have certain CPU demand though

[onlyjob]

Having more than few chunkservers on one machine makes no sense. One chunkserver is perfectly capable of handling multiple HDDs as well as having multiple labels. IMHO 14 chunkservers looks like misconfiguration. What could possibly justify such setup?

[eleaner]

It is precisely for HDD-level protection. And I need a few chunk servers to allow ec encoding.Sent from my iPhoneOn 13 Dec 2022, at 05:41, onlyjob ***@***.***> wrote: Having more than few chunkservers on one machine makes no sense. One chunkserver is perfectly capable of handling multiple HDDs as well as having multiple labels. IMHO 14 chunkservers looks like misconfiguration. What could possibly justify such setup? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[jkiebzak]

You could have different MFS chunk server processes point to different speed drives: 5400, 7200, SSD, NVMe, assigning them different labels. Then you can choose to tier however you like.

…

Message ID: ***@***.*** com>

[Blackpaw]

Its about having software raid pooling your storage on one machine. I have a media server with 14 disks of various sizes (total 50TB) running 4 chunkservers. It pools the disks into one namespace and provides all the moosefs features such as checksuming, copies = 2, 3 etc to protect against disk failures. Over the years I've had the occasional disk failure, but no data loss. It's a lot easier to manage and more flexible that zfs.

[Blackpaw]

Its about having software raid pooling your storage on one machine. I have a media server with 14 disks of various sizes (total 50TB) running 4 chunkservers. It pools the disks into one namespace and provides all the moosefs features such as checksuming, copies = 2, 3 etc to protect against disk failures. Over the years I've had the occasional disk failure, but no data loss. It's a lot easier to manage and more flexible that zfs.

[eleaner]

+1 to @Blackpaw
moosefs is a pretty good software raid - even if it is not the use was designed for

[VideoFX]

I can confirm that running more than 1 chunk server on a single machine WORKS amazingly to achieve the goal of having more than one copy of data per drive.

You can still have multiple machines, and practice redundancy there in addition.

For example, You can run 2 (or more) chunk servers on a single server, and assign some drives to one chunk server, and other drives to the other chunk server(s). The result is, as expected, you get redundancy, and also has some nice perks like fast replication, and excellent per-drive management.

Another use case is related to drive diversity. You may have a mix of spinning disks and SSDs. You can assign one chunk server to the spinning disks, and another chunk server to SSDs, in the SAME machine! You can continue to do that with more machines too! The result is, you have both fast and slow drives in the same machines, each managed by its own chunk instance. You can then replicate that to additional machines, each having multiple chuck servers if needed.

MooseFS has incredible flexibility. It blows my mind how far you can go to manage data using this tool. Bravo!