Installation of ganesha NFS and SMB packages from the protcol node software suite. SMB packages must be installed as well otherwise mmces will not start.

[root@gprot1 ~]# rpm -qa | egrep "gan|smb"
nfs-ganesha-gpfs-2.3.0-0.1ibm11.el7.x86_64
nfs-ganesha-2.3.0-0.1ibm11.el7.x86_64
nfs-ganesha-utils-2.3.0-0.1ibm11.el7.x86_64
gpfs.smb-4.3.0_gpfs_9-3.el7.x86_64
[root@gprot1 ~]# 

set cesSharedRoot in the global config. cesSharedRoot can be located on an existing GPFS filesystem but it's recommended to place it in an own small GPFS filesystem for security reasons. This filesystem needs only to be mounted on the protoocol node.

mmcrfs cesroot -F /repo/config/stanza/titan2_cesroot.stanza -A yes \
       -B 64k -j scatter -m 1 -M 1 -R 1 -r 1 -n 128 -D posix -n 2 -T /mnt/cesroot

Set the CES root in the global config. It can be located inside an existing filesystem, but for security reasons it should be placed into a separated filesystem. The min size of the cesSharedRoot is 4GB.

mmchconfig cesSharedRoot=/mnt/cesroot

enable ccr in the cluster; mandatory to configure CES

mmchcluster --ccr-enable

set the protocol node to be ces enabled. The node must have the CES shared root mounted.

mmchnode --ces-enable -N 10

start up CES NFS service

mmces service enable nfs
mmces service start nfs

set shared IP to be used as exporter interface for NFS and/or SMB. The IP must be resolvable (at least on the CES nodes and on the node where the command is issued)

mmces address add --ces-node gprot1 --ces-ip 10.0.2.230

check CES status

[root@gpfs1 ~]# mmlscluster --ces

GPFS cluster information
========================
GPFS cluster name:         gpfs1.ib.bm.nec.de
GPFS cluster id:           1768562925710693003

Cluster Export Services global parameters
-----------------------------------------
Shared root directory:                /mnt/cesroot
Enabled Services:                     NFS
Log level:                            3
Address distribution policy:          even-coverage

Node  Daemon node name            IP address       CES IP address list
-----------------------------------------------------------------------
10   gprot1                      10.0.0.223       10.0.2.230

[root@gpfs1 ~]# 

create user authentication using mmuserauth. This is the basic and mandatory setting. connections to LDAP or NIS or AD to be defined here.

mmuserauth service create --type userdefined --data-access-method file

[root@rwthpn2 ~]# mmuserauth service check -N cesNodes --rectify
File configuration is USER-DEFINED
Object not configured
[root@rwthpn2 ~]# 

[root@rwthpn2 ~]# mmnfs export add /rwth/vol1/nfsroot/ -c "10.0.2.0/24(Access_Type=RW,Protocols=3:4)"
192.168.2.192:  Redirecting to /bin/systemctl stop  nfs-ganesha.service
192.168.2.191:  Redirecting to /bin/systemctl stop  nfs-ganesha.service
192.168.2.191:  Redirecting to /bin/systemctl start  nfs-ganesha.service
192.168.2.192:  Redirecting to /bin/systemctl start  nfs-ganesha.service
NFS Configuration successfully changed. NFS server restarted on all NFS nodes.
[root@rwthpn2 ~]# 

[root@rwthpn2 ~]# mmnfs export list
Path               Delegations Clients     
-------------------------------------------
/rwth/vol1/nfsroot none        10.0.2.0/24 
[root@rwthpn2 ~]#  


[root@rwthpn2 ~]# mmnfs export list --nfsdefs /rwth/vol1/nfsroot/
Path               Delegations Clients     Access_Type Protocols Transports Squash      Anonymous_uid Anonymous_gid SecType PrivilegedPort DefaultDelegations Manage_Gids NFS_Commit 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/rwth/vol1/nfsroot none        10.0.2.0/24 RW          3,4       TCP        ROOT_SQUASH -2            -2            SYS     FALSE          none               FALSE       FALSE      

[root@rwthpn2 ~]#