Add support for CEPH ceph.file.layout and ceph.dir.layout xattr

[fabiomirmar]

Ceph supports using extended attributes (xattrs) to configure a file or directory layout, so as to tell Ceph which pool to place the objects related to that file or directory. This is mostly relevant for when you are using CephFS and have multiple pools in the Ceph volume, as documented here [1] and [2].

However, if you have NFS-Ganesha serving the CephFS filesystem to NFS clients, it will fail if you try to setfattr

ubuntu@jammy-183048:~$ sudo setfattr -n ceph.dir.layout.pool -v ceph-fs_ecdata /mnt/nfs1/ecpool
setfattr: /mnt/nfs1/ecpool: Operation not supported

So, it seems there's a limitation on what kind of Ceph xattr ganesha would support.

In this bug, there are some discussions around some CEPH FSALs that were enabled.

#526

And we can see them at:

nfs-ganesha/src/FSAL/FSAL_CEPH/internal.h

Lines 201 to 204 in f495f2f

	#define CEPH_SETTABLE_ATTRIBUTES ((const attrmask_t) ( \
	ATTR_MODE | ATTR_OWNER | ATTR_GROUP | ATTR_ATIME | \
	ATTR_CTIME | ATTR_MTIME | ATTR_SIZE | ATTR_MTIME_SERVER | \
	ATTR_ATIME_SERVER | ATTR4_SEC_LABEL | POSIX_ACL_ATTR))

However none of them are related to the xattr required by Ceph for disk layouts to work (i.e.: ceph.dir.layout.pool)

Hence, although ceph-fs supports file-layouts, it doesn't seem that NFS Ganesha contains the necessary features to support it in a Ceph environment.

It would be good to have implemented in NFS-Ganesha, so we can relay the CephFS requirements to configure file-layouts.

[1] https://docs.ceph.com/en/latest/cephfs/file-layouts/#adding-a-data-pool-to-the-file-system
[2] https://docs.ceph.com/en/latest/cephfs/file-layouts/#adding-a-data-pool-to-the-file-system

[ffilz]

Ganesha does have the xattr support from NFSv4.2 and FSAL_CEPH is hooked up to use it.

[xiaods]

@fabiomirmar any update on it?

[fabiomirmar]

Hi @ffilz and @xiaods thanks for the reply. My environment is running Ubuntu 22.04 + nfs-ganesha 3.5-1ubuntu1:

ii  nfs-ganesha                          3.5-1ubuntu1                            amd64        NFS server in User Space
ii  nfs-ganesha-ceph:amd64               3.5-1ubuntu1                            amd64        nfs-ganesha fsal ceph libraries

And my client is mounting the share using nfs 4.2:

10.5.100.1:/volumes/_nogroup/f35a6430-bc6c-4202-85a7-48711b9ea639/f5e9f783-e4c9-49bb-9cd8-8c0ee8edbd03 on /mnt type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.21.148,local_lock=none,addr=10.5.100.1)

But I still can't set this xattr:

root@jammy-130612:/mnt# setfattr -n ceph.dir.layout.pool -v ceph-fs_ecdata ecpool
setfattr: ecpool: Operation not supported

Any idea what I might be missing here?

Not sure if it matters, but this is how ganesha.conf looks like: https://paste.ubuntu.com/p/hzSRzwMDGF/

[dang]

FSAL Ceph didn't get xattr support until 4.0.

[mattbenjamin]

also, though, is this a user xattr? nfs only permits setting those, iirc

[fabiomirmar]

OK, thanks for the information. Do you have a list of commits that enabled this?

And, to answer Matt's question, the xattr that I'm looking to use are ceph.file.layout and ceph.dir.layout as outlined here: https://docs.ceph.com/en/latest/cephfs/file-layouts/

[ffilz]

They were added in V4-dev.55, looks like a bunch of patches.