The documentation suggests using ReadWriteMany, but that fails

[davesargrad]

I am using rook/nfs in a small k3s cluster.

I am following these instructions.

I saw this error message:

The fix was to replace ReadWriteMany with ReadWriteOnce.

Now the PVC is properly bound.

Am I losing something with having to use ReadWriteOnce? Should the documentation be changed?

Please clarify. Is this a documentation bug? Or should I actually be using ReadWriteMany? If so, how?

[travisn]

The error message indicates that the storage class "local-path" doesn't support RWX volumes. Are you sure you were using the correct storage class? The nfs should be capable of RWX volumes since the whole point is to have shared storage between pods. A RWO volume can't be shared between pods.

[davesargrad]

@travisn

This is the default storage class used to establish the persistent volume that the NFS server itself uses.

The documentation says that any storage class will do. I am using a local-storage storage class.

The documentation doesnt say there is a "correct" storage class. It suggests that as long as there is a default storage class.. things are good.

The issue I've written, I believe, is just an issue on the documentation. I struggled for a while before realizing that I could use ReadWriteOnce (with local storage). I think the documentation should talk clearly to this option.

[TonyBogdanov]

@davesargrad In my experience I haven't found a major cloud provider which supports ReadWriteMany for its native storage. This is because a storage device / volume can only be attached to one node at a time. It would be physically impossible to attach a hard drive to two nodes in different regions.

If your cloud provider did support ReadWriteMany, why would you even need to use rook/nfs in the first place...

The problem in this particular case is that rook/nfs doesn't really provide a "cluster" solution to NFS storage, but merely exposes an underlying cloud native storage via an NFS export. Simply put, with rook/nfs you are installing an NFS server and pointing it to a folder on one of your nodes.

The fact that you can have more than one replica of the NFS server, doesn't really mean you are running an NFS cluster, you are just running N servers, all reading and writing to the same folder. If you've installed this and played with it, you'll notice all replicas you create live on the exact same node, you can't have them on different nodes or they won't be able to access the underlying storage.

A proper NFS cluster would be one where each replica has its own persistent volume containing a copy of all stored data, constantly in contact with other replicas and constantly replicating the data. Unfortunately rook/nfs is NOT such a solution.

On the other hand rook/ceph IS such a solution, or at least it was, 1 year ago when I tried it out. Unfortunatelly to run it you'll need a very strong machine with upwards of 4 cores and 16 GB of RAM just for your storage solution.