Skip to content

Kerberos setup for NFS Ganesha in Ceph

Jump to bottom Edit New page
Sachin Punadikar edited this page Nov 8, 2023 · 1 revision

This document provides steps on how to setup KDC, Kerberos client and NFS specific changes in Kerberos. Treat this document is guide for standalone kerberized NFS setup. In most of the cases KDC setup already exists, and we have use that KDC information for Kerberos setup. This document has multiple sections:

  • Correcting /etc/hosts file (if needed)
  • KDC setup - Setup new KDC (as per need)
  • Kerberos client setup
  • NFS specific kerberos steps
  • Ceph container changes for NFS Ganesha
  • NFS client side setup
  • Setup validation

Generic changes in "/etc/hosts" (if needed)

Observed that in lab setup, /etc/hosts needs some changes to add domain name. Domain name usually used to have FQDN for every node involved in Kerberos setup. Using Domain Name = redhat.com

[root@ceph-mani-o7fdxp-node4 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.208.97	        ceph-mani-o7fdxp-node1-installer.redhat.com	ceph-mani-o7fdxp-node1-installer
10.0.210.243	ceph-mani-o7fdxp-node2.redhat.com	ceph-mani-o7fdxp-node2
10.0.208.63	        ceph-mani-o7fdxp-node3.redhat.com	ceph-mani-o7fdxp-node3
10.0.210.222	ceph-mani-o7fdxp-node4.redhat.com	ceph-mani-o7fdxp-node4
10.0.210.235	ceph-mani-o7fdxp-node5.redhat.com	ceph-mani-o7fdxp-node5
10.0.209.87	        ceph-mani-o7fdxp-node6.redhat.com	ceph-mani-o7fdxp-node6
10.0.208.89	        ceph-mani-o7fdxp-node7.redhat.com	ceph-mani-o7fdxp-node7

Ensure that domain name is present for all involved nodes in the setup (all nodes in Ceph cluster and all NFS client nodes)

Setup the KDC (as per need)

KDC stands for Key Distribution Center. This holds information about all its clients (user principals, service principals), hence needs to be secure. In a kerberos setup KDC is single point of failure, hence it is recommended to have one master KDC and multiple slave KDCs. Lets start installing and configuring KDC. Skip this part if you already have KDC installed and configured.

Check the required RPMs are installed on the machine where you want to setup KDC

[root@ceph-mani-o7fdxp-node4 ~]# rpm -qa | grep krb5
krb5-libs-1.20.1-9.el9_2.x86_64
krb5-pkinit-1.20.1-9.el9_2.x86_64
krb5-server-1.20.1-9.el9_2.x86_64
krb5-server-ldap-1.20.1-9.el9_2.x86_64
krb5-devel-1.20.1-9.el9_2.x86_64
krb5-workstation-1.20.1-9.el9_2.x86_64

Decide below things

KDC server - - ceph-mani-o7fdxp-node4.redhat.com Kerberos works with FQDN names. And it is better to have domain name in accordance with kerberos REALM name. Realm - PUNE.REDHAT.COM Admin principal - admin/admin

Edit the installed conf files to reflect our new KDC. Note that KDC can be provided as either IP address or DNS name.

Update the krb5.conf file

[root@ceph-mani-o7fdxp-node4 ~]# cat /etc/krb5.conf
# To opt out of the system crypto-policies configuration of krb5, remove the
# symlink at /etc/krb5.conf.d/crypto-policies which will not be recreated.
includedir /etc/krb5.conf.d/
[logging]
    default = [FILE:/var/log/krb5libs.log](file:///var/log/krb5libs.log)
    kdc = [FILE:/var/log/krb5kdc.log](file:///var/log/krb5kdc.log)
    admin_server = [FILE:/var/log/kadmind.log](file:///var/log/kadmind.log)
[libdefaults]
    dns_lookup_realm = false
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = true
    rdns = false
    pkinit_anchors = [FILE:/etc/pki/tls/certs/ca-bundle.crt](file:///etc/pki/tls/certs/ca-bundle.crt)
    spake_preauth_groups = edwards25519
    dns_canonicalize_hostname = fallback
    qualify_shortname = ""
    default_realm = PUNE.REDHAT.COM
    default_ccache_name = KEYRING:persistent:%{uid}
[realms]
    PUNE.REDHAT.COM = {
       kdc = 10.0.210.222:88
       admin_server = 10.0.210.222:749
    }
[domain_realm]
  .redhat.com = PUNE.REDHAT.COM
  redhat.com = PUNE.REDHAT.COM

Update the kdc.conf file

 [root@ceph-mani-o7fdxp-node4 ~]# cat /var/kerberos/krb5kdc/kdc.conf
 [kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88
 spake_preauth_kdc_challenge = edwards25519
[realms]
   PUNE.REDHAT.COM = {
      master_key_type = aes256-cts-hmac-sha384-192
      acl_file = /var/kerberos/krb5kdc/kadm5.acl
      dict_file = /usr/share/dict/words
      default_principal_flags = +preauth
      admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
      supported_enctypes = aes256-cts-hmac-sha384-192:normal aes128-cts-hmac-sha256-128:normal aes256-cts-hmac-sha1-96:normal aes128-cts-hmac-sha1-96:normal camellia256-cts-cmac:normal camellia128-cts-cmac:normal arcfour-hmac-md5:normal
     # Supported encryption types for FIPS mode:
     #supported_enctypes = aes256-cts-hmac-sha384-192:normal aes128-cts-hmac-sha256-128:normal
  }

Create the KDC database using kdb5_util

Note: Make sure that host name can be resolved via either DNS or /etc/hosts

[root@ceph-mani-o7fdxp-node4 ~]# kdb5_util create -s -r [PUNE.REDHAT.COM](http://pune.ibm.com/)
Initializing database '/var/kerberos/krb5kdc/principal' for realm 'PUNE.REDHAT.COM',
master key name 'K/M@PUNE.REDHAT.COM'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
Re-enter KDC database master key to verify:

Add administrators to the ACL file

[root@ceph-mani-o7fdxp-node4 ~]# cat /var/kerberos/krb5kdc/kadm5.acl
*/admin@PUNE.REDHAT.COM	*

Above line indicates any principal in the PUNE.REDHAT.COM realm with an admin instance has all administrative privileges.

Add administrators to the Kerberos database

[root@ceph-mani-o7fdxp-node4 ~]# kadmin.local
Authenticating as principal root/admin@PUNE.REDHAT.COM with password.
kadmin.local:  addprinc admin/admin@PUNE.REDHAT.COM
No policy specified for admin/admin@PUNE.REDHAT.COM; defaulting to no policy
Enter password for principal "admin/admin@PUNE.REDHAT.COM":
Re-enter password for principal "admin/admin@PUNE.REDHAT.COM":
Principal "admin/admin@PUNE.REDHAT.COM" created.
kadmin.local:

Start kdc & kadmind

# krb5kdc
# kadmind

Check if kdc & kadmind are running properly

# ps -eaf | grep krb
root     27836     1  0 07:35 ?        00:00:00 krb5kdc
root     27846 13956  0 07:35 pts/8    00:00:00 grep --color=auto krb
# ps -eaf | grep kad
root     27841     1  0 07:35 ?        00:00:00 kadmind
root     27851 13956  0 07:36 pts/8    00:00:00 grep --color=auto kad

Check if setup is correct

[root@ceph-mani-o7fdxp-node4 ~]# kinit admin/admin
Password for admin/admin@PUNE.REDHAT.COM:
[root@ceph-mani-o7fdxp-node4 ~]# klist
Ticket cache: KCM:0
Default principal: admin/admin@PUNE.REDHAT.COM

Valid starting     Expires            Service principal
10/25/23 06:37:08  10/26/23 06:37:01  krbtgt/PUNE.REDHAT.COM@PUNE.REDHAT.COM
    renew until 10/25/23 06:37:08

Setup the kerberos client

The Kerberos client machine should be time synced with KDC. It is advised to sync KDC and clients by using NTP. Time difference of 5 minutes or more leads to Kerberos authentication failure and throws clock skew error. This step is prerequisites on all systems which are going to participate in Kerberos authentication like NFS clients, hosts where NFS Ganesha containers are going to run.

Check the required RPMs

[root@ceph-mani-o7fdxp-node5 ~]# rpm -qa | grep krb5
krb5-libs-1.20.1-9.el9_2.x86_64
krb5-pkinit-1.20.1-9.el9_2.x86_64
krb5-workstation-1.20.1-9.el9_2.x86_64

Update the krb5.conf file similar to that of on KDC server

[root@ceph-mani-o7fdxp-node5 ~]# cat /etc/krb5.conf
# To opt out of the system crypto-policies configuration of krb5, remove the
# symlink at /etc/krb5.conf.d/crypto-policies which will not be recreated.
includedir /etc/krb5.conf.d/
[logging]
    default = [FILE:/var/log/krb5libs.log](file:///var/log/krb5libs.log)
    kdc = [FILE:/var/log/krb5kdc.log](file:///var/log/krb5kdc.log)
    admin_server = [FILE:/var/log/kadmind.log](file:///var/log/kadmind.log)
[libdefaults]
    dns_lookup_realm = false
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = true
    rdns = false
    pkinit_anchors = [FILE:/etc/pki/tls/certs/ca-bundle.crt](file:///etc/pki/tls/certs/ca-bundle.crt)
    spake_preauth_groups = edwards25519
    dns_canonicalize_hostname = fallback
    qualify_shortname = ""
    default_realm = PUNE.REDHAT.COM
   default_ccache_name = KEYRING:persistent:%{uid}
[realms]
  PUNE.REDHAT.COM = {
     kdc = 10.0.210.222:88
     admin_server = 10.0.210.222:749
  }
[domain_realm]
  .redhat.com = PUNE.REDHAT.COM
  redhat.com = PUNE.REDHAT.COM

Validate the client settings

[root@ceph-mani-o7fdxp-node5 ~]# kinit admin/admin
Password for admin/admin@PUNE.REDHAT.COM:
[root@ceph-mani-o7fdxp-node5 ~]# klist
Ticket cache: KCM:0
Default principal: admin/admin@PUNE.REDHAT.COM

Valid starting     Expires            Service principal
10/25/23 08:49:12  10/26/23 08:49:08  krbtgt/PUNE.REDHAT.COM@PUNE.REDHAT.COM
  renew until 10/25/23 08:49:12

Here basic configuration of KDC and client is done.

NFS specific Kerberos setup

First we need to create service principals for both NFS server and client. The respective keys are stored in the keytab files. These principals are required to setup the initial security context (required by GSS_RPCSEC). These service principals have format like nfs/@REALM. One can copy the /etc/krb5.conf from working system to the Ceph nodes.

Create service principal for that host.

[root@ceph-mani-o7fdxp-node1-installer ~]# kinit admin/admin
Password for admin/admin@PUNE.REDHAT.COM:
[root@ceph-mani-o7fdxp-node1-installer ~]# kadmin
Authenticating as principal admin/admin@PUNE.REDHAT.COM with password.
Password for admin/admin@PUNE.REDHAT.COM:
kadmin:  addprinc -randkey nfs/ceph-mani-o7fdxp-node1-installer.redhat.com
No policy specified for nfs/ceph-mani-o7fdxp-node1-installer.redhat.com@PUNE.REDHAT.COM; defaulting to no policy
Principal "nfs/ceph-mani-o7fdxp-node1-installer.redhat.com@PUNE.REDHAT.COM" created.

Add the key to the keytab file.

Note that here we are already on the NFS server and using kadmin interface. Here the keytab operations reflect on the keytab of the NFS server.

kadmin:  ktadd nfs/ceph-mani-o7fdxp-node1-installer.redhat.com
Entry for principal nfs/ceph-mani-o7fdxp-node1-installer.redhat.com with kvno 2, encryption type aes256-cts-hmac-sha384-192 added to keytab [FILE:/etc/krb5.keytab](file:///etc/krb5.keytab).
Entry for principal nfs/ceph-mani-o7fdxp-node1-installer.redhat.com with kvno 2, encryption type aes128-cts-hmac-sha256-128 added to keytab [FILE:/etc/krb5.keytab](file:///etc/krb5.keytab).
Entry for principal nfs/ceph-mani-o7fdxp-node1-installer.redhat.com with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab [FILE:/etc/krb5.keytab](file:///etc/krb5.keytab).
Entry for principal nfs/ceph-mani-o7fdxp-node1-installer.redhat.com with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab [FILE:/etc/krb5.keytab](file:///etc/krb5.keytab).
Entry for principal nfs/ceph-mani-o7fdxp-node1-installer.redhat.com with kvno 2, encryption type camellia256-cts-cmac added to keytab [FILE:/etc/krb5.keytab](file:///etc/krb5.keytab).
Entry for principal nfs/ceph-mani-o7fdxp-node1-installer.redhat.com with kvno 2, encryption type camellia128-cts-cmac added to keytab [FILE:/etc/krb5.keytab](file:///etc/krb5.keytab).
Entry for principal nfs/ceph-mani-o7fdxp-node1-installer.redhat.com with kvno 2, encryption type arcfour-hmac added to keytab [FILE:/etc/krb5.keytab](file:///etc/krb5.keytab).
kadmin:

Carry out above 2 steps on all ceph nodes where NFS Ganesha containers would be running and all the NFS clients.

NFS Ganesha container settings in Ceph environment

Get existing NFS Ganesha container config

[ceph: root@ceph-msaini-qp49z7-node1-installer /]# ceph orch ls --service-type nfs --export
service_type: nfs
service_id: c_ganesha
service_name: nfs.c_ganesha
placement:
  hosts:
  - ceph-msaini-qp49z7-node1-installer
  - ceph-msaini-qp49z7-node2
  - ceph-msaini-qp49z7-node3
spec:
  port: 2049

Modify the container config

Modify the container configuration to pass on /etc/krb5.conf and /etc/krb5.keytab files to the container from the host. These files will be referred by NFS Ganesha at runtime to validate incoming service tickets and secure the communication between Ganesha and NFS client (krb5p).

[root@ceph-msaini-qp49z7-node1-installer ~]# cat nfs.yaml
service_type: nfs
service_id: c_ganesha
service_name: nfs.c_ganesha
placement:
  hosts:
  - ceph-msaini-qp49z7-node1-installer
  - ceph-msaini-qp49z7-node2
  - ceph-msaini-qp49z7-node3
spec:
  port: 2049
extra_container_args:
  - "-v"
  - "/etc/krb5.keytab:/etc/krb5.keytab:ro"
  - "-v"
  - "/etc/krb5.conf:/etc/krb5.conf:ro"

Make this modified nfs.yaml file available inside cephadm shell

[root@ceph-msaini-qp49z7-node1-installer ~]# cephadm shell --mount nfs.yaml:/var/lib/ceph/nfs.yaml
Inferring fsid ff1c1498-73ec-11ee-af38-fa163e9a17fd
Inferring config /var/lib/ceph/ff1c1498-73ec-11ee-af38-fa163e9a17fd/mon.ceph-msaini-qp49z7-node1-installer/config
Using ceph image with id 'fada497f9c5f' and tag 'ceph-7.0-rhel-9-containers-candidate-73711-20231018030025' created on 2023-10-18 03:03:39 +0000 UTC
registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:e66e5dd79d021f3204a183f5dbe4537d0c0e4b466df3b2cc4d50cc79c0f34d75

Validate the file has required changes.

[ceph: root@ceph-msaini-qp49z7-node1-installer /]# cat /var/lib/ceph/nfs.yaml
service_type: nfs
service_id: c_ganesha
service_name: nfs.c_ganesha
placement:
  hosts:
  - ceph-msaini-qp49z7-node1-installer
  - ceph-msaini-qp49z7-node2
  - ceph-msaini-qp49z7-node3
spec:
  port: 2049
extra_container_args:
  - "-v"
  - "/etc/krb5.keytab:/etc/krb5.keytab:ro"
  - "-v"
  - "/etc/krb5.conf:/etc/krb5.conf:ro"

Apply the required changes to NFS Ganesha container and redeploy the container

[ceph: root@ceph-msaini-qp49z7-node1-installer /]# ceph orch apply -i /var/lib/ceph/nfs.yaml
Scheduled nfs.c_ganesha update...
[ceph: root@ceph-msaini-qp49z7-node1-installer /]# ceph orch redeploy nfs.c_ganesha
Scheduled to redeploy nfs.c_ganesha.1.0.ceph-msaini-qp49z7-node1-installer.sxzuts on host 'ceph-msaini-qp49z7-node1-installer'
Scheduled to redeploy nfs.c_ganesha.2.0.ceph-msaini-qp49z7-node2.psuvki on host 'ceph-msaini-qp49z7-node2'
Scheduled to redeploy nfs.c_ganesha.0.0.ceph-msaini-qp49z7-node3.qizzvk on host 'ceph-msaini-qp49z7-node3'

Validate that redeployed service is having required changes.

[ceph: root@ceph-msaini-qp49z7-node1-installer /]# ceph orch ls --service-type nfs --export
service_type: nfs
service_id: c_ganesha
service_name: nfs.c_ganesha
placement:
  hosts:
  - ceph-msaini-qp49z7-node1-installer
  - ceph-msaini-qp49z7-node2
  - ceph-msaini-qp49z7-node3
extra_container_args:
- -v
- /etc/krb5.keytab:/etc/krb5.keytab:ro
- -v
- /etc/krb5.conf:/etc/krb5.conf:ro
spec:
  port: 2049

Modify the export definition to have krb5* (krb5, krb5i, krb5p) security flavor.

One can create such export after doing all above setup.

[ceph: root@ceph-msaini-qp49z7-node1-installer /]# ceph nfs export info c_ganesha /exp1
{
  "access_type": "RW",
  "clients": [],
  "cluster_id": "c_ganesha",
  "export_id": 1,
  "fsal": {
    "fs_name": "fs1",
    "name": "CEPH",
    "user_id": "nfs.c_ganesha.1"
  },
  "path": "/volumes/_nogroup/exp1/81f9a67e-ddf1-4b5a-9fe0-d87effc7ca16",
  "protocols": [
    4
  ],
  "pseudo": "/exp1",
  "sectype": [
    "krb5"
  ],
  "security_label": true,
  "squash": "none",
  "transports": [
    "TCP"
  ]
}

NFS Client side actions

Create service principal

kadmin:  addprinc -randkey nfs/ceph-msaini-qp49z7-node5.redhat.com@PUNE.REDHAT.COM
No policy specified for nfs/ceph-msaini-qp49z7-node5.redhat.com@PUNE.REDHAT.COM; defaulting to no policy
Principal "nfs/ceph-msaini-qp49z7-node5.redhat.com@PUNE.REDHAT.COM" created.
kadmin:  ktadd nfs/ceph-msaini-qp49z7-node5.redhat.com@PUNE.REDHAT.COM
Entry for principal nfs/ceph-msaini-qp49z7-node5.redhat.com@PUNE.REDHAT.COM with kvno 2, encryption type aes256-cts-hmac-sha384-192 added to keytab [FILE:/etc/krb5.keytab](file:///etc/krb5.keytab).
Entry for principal nfs/ceph-msaini-qp49z7-node5.redhat.com@PUNE.REDHAT.COM with kvno 2, encryption type aes128-cts-hmac-sha256-128 added to keytab [FILE:/etc/krb5.keytab](file:///etc/krb5.keytab).
Entry for principal nfs/ceph-msaini-qp49z7-node5.redhat.com@PUNE.REDHAT.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab [FILE:/etc/krb5.keytab](file:///etc/krb5.keytab).
Entry for principal nfs/ceph-msaini-qp49z7-node5.redhat.com@PUNE.REDHAT.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab [FILE:/etc/krb5.keytab](file:///etc/krb5.keytab).
Entry for principal nfs/ceph-msaini-qp49z7-node5.redhat.com@PUNE.REDHAT.COM with kvno 2, encryption type camellia256-cts-cmac added to keytab [FILE:/etc/krb5.keytab](file:///etc/krb5.keytab).
Entry for principal nfs/ceph-msaini-qp49z7-node5.redhat.com@PUNE.REDHAT.COM with kvno 2, encryption type camellia128-cts-cmac added to keytab [FILE:/etc/krb5.keytab](file:///etc/krb5.keytab).
Entry for principal nfs/ceph-msaini-qp49z7-node5.redhat.com@PUNE.REDHAT.COM with kvno 2, encryption type arcfour-hmac added to keytab [FILE:/etc/krb5.keytab](file:///etc/krb5.keytab).

Restart rpc.gssd service to take effect of modified/new keytab file.

# systemctl restart rpc-gssd

Mount the NFS export

[root@ceph-msaini-qp49z7-node5 ~]# mount -t nfs 10.0.210.146:/exp1 /mnt

Create users

Once the NFS export is mounted, regular users are used to work with mounted exports. These regular users (generally local users on the system or users from centralised system like LDAP/AD) need to be part of Kerberos setup. Based on the kind of setup, local users need to be created in KDC as well.

Validating the setup

Access the export as normal user (Sachin), without Kerberos tickets

[sachin@ceph-msaini-qp49z7-node5 ~]$ klist
klist: Credentials cache 'KCM:1001' not found
[sachin@ceph-msaini-qp49z7-node5 ~]$ cd /mnt
-bash: cd: /mnt: Permission denied

With Kerbeors ticket

[sachin@ceph-msaini-qp49z7-node5 ~]$ kinit sachin
Password for sachin@PUNE.REDHAT.COM:
[sachin@ceph-msaini-qp49z7-node5 ~]$ klist
Ticket cache: KCM:1001
Default principal: sachin@PUNE.REDHAT.COM

Valid starting     Expires            Service principal
10/27/23 12:57:21  10/28/23 12:57:17  krbtgt/PUNE.REDHAT.COM@PUNE.REDHAT.COM
  renew until 10/27/23 12:57:21
[sachin@ceph-msaini-qp49z7-node5 ~]$ cd /mnt
[sachin@ceph-msaini-qp49z7-node5 mnt]$ klist
Ticket cache: KCM:1001
Default principal: sachin@PUNE.REDHAT.COM

Valid starting     Expires            Service principal
10/27/23 12:57:21  10/28/23 12:57:17  krbtgt/PUNE.REDHAT.COM@PUNE.REDHAT.COM
renew until 10/27/23 12:57:21
10/27/23 12:57:28  10/28/23 12:57:17  nfs/ceph-msaini-qp49z7-node1-installer.redhat.com@
renew until 10/27/23 12:57:21
Ticket server: nfs/ceph-msaini-qp49z7-node1-installer.redhat.com@PUNE.REDHAT.COM

Ticket for service nfs/ are observed on the client.

Add a custom footer
  1. Release Cycle
  2. Downloads
  3. Architecture
  4. Developer Guide
  5. Continuous Integration
  6. Static Analysis
  7. Configuration Files
  8. FSALs:
  9. DBus:
  10. Kerberos Authentication
  11. High Availability
  12. SELinux
  13. Talks, Papers and Blogs
Clone this wiki locally