Replies: 2 comments 3 replies
-
|
I'm afraid I'm not much help on how to best accomplish it within Kubernetes, but in case it's helpful I think what you probably want as part of the "ideal end result" would be a separate moosefs/mfsmanpages/mfsmount.8 Lines 70 to 73 in 023198e |
Beta Was this translation helpful? Give feedback.
-
|
What about exporting a block device with "mfsbdev" ? This is what I am using with lxd containers, creating a big file on moosefs, exporting with mfsbdev, making a zfs filesystem on the device and every containers will have its own dataset: mfsbdev start -H mfsmaster zfs list Cheers Max |
Beta Was this translation helpful? Give feedback.
-
|
Sorry about the delay replying but I've been contemplating my options including this. I haven't sorted out what I'm actually going to do yet, but some variation on your idea might be the way I go. The issue with what you're doing in k8s is that you're mounting the block device on the host itself, which means you can only mount a particular block device on a single host, and would need to restrict containers to running on that particular host, which defeats some of the ability of k8s to just move containers around between nodes. Unless the block device contains a cluster filesystem you couldn't mount the same block device on all your nodes, and running a cluster filesystem on top of a distributed filesystem seems like an interesting prospect (though probably possible). Now, a variation on what you're doing is to just map the block device on multiple hosts but not mount it at the host level. Instead expose the block device itself to the container and let the container mount it. I think k8s can be told to not let more than one container run at the same time with access to the block device. If all nodes can see the block device but only one can access at a time then a container could be scheduled on any node. For containers that don't need to create a bazillion small files I can just directly map a directory in the filesystem to that container. I'm not using moosefs to store the actual container images - just their persistent storage. k8s just caches locally on each node the images and all container images are ephemeral. Another option would be to get moosefs installed in each container and let the container do its own mapping, assuming I can give them the necessary permissions. I'd need to keep two containers from mounting the same image though unless moosefs can lock them. Fixing the moosefs CSI driver might end up being the easiest option. It seems like it is just out of date with respect to the API. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @rich0, we are in progress of testing a newer version of MooseFS CSI and we are planning to release it as soon as the tests are completed, so please bear with us and stay tuned! |
Beta Was this translation helpful? Give feedback.
-
|
@pkonopelko I'm definitely happy to hear with that. I've been experimenting with the Kunde21 fork and that seems closer to working, though it actually fails to mount moosefs (already reported as an issue on his repo). I'm fairly new to k8s and figuring this out as I go, but I'm definitely getting the impression that some of these APIs aren't entirely stable so a number of CSIs end up needing fixes to work in later k8s versions. Longhorn for example won't work on 1.25 yet, at least not the stable version. |
Beta Was this translation helpful? Give feedback.
-
(FYI - I'm brand new to both docker and k8s, though I have been running containers for a while using nspawn.)
I want to access data on MooseFS within some k8s containers, but am not sure what the best-supported way of going about this is.
I see there is a CSI for MooseFS, but it doesn't seem to be configurable and mounts the entire filesystem. Obviously I can't just drop that into some published container where it expects to write otherwise everything will just go into the root of the filesystem.
Options I can think of offhand are to either try to get symlinks or bind mounts into the container image so that I can point things deeper into moosefs. I'd prefer not to have the entire filesystem mounted though, so another option would be to mount moosefs on the hosts, and then use hostpath to mount specific paths under that to each place they need to go, which would accomplish exactly what I want to do. Hostpath seems to be discouraged in the k8s community, but in this case a container could run on any host I believe as long as they all mount moosefs in the same place, and containers would see the same thing.
Beta Was this translation helpful? Give feedback.
All reactions