The NEC ExpressServer5800 R120f-2M systems have the following default IP adresses and accounts configured. In multi server environments the server can be identified by the S/N printed on a small flag at the front of the server.

Default IP: 192.168.1.1 
account: Administrator/Administrator

access via ipmitool on default config.
read S/N from the IPMI interface. This is the same as printed on the flag.

ipmitool -U Administrator -P Administrator -H 192.168.1.1 fru

set IP address of identified interface

ipmitool -U Administrator -P Administrator -H 192.168.1.1 lan set 2 ipaddr 192.168.2.203 

Access the GUI via the IPMI IP address and set the KVM license

The following changes shall be done in the BIOS settings of the server to ensure a proper operation.

• Switch boot from UEFI to legacy
• Check that all option roms except for LSI RAID controller are disabled
• Create RAID1 over both boot disks
• BBU setting to write through in LSI config menu

The follwing procedure is valid for the environment of the EHPCTC in Leinfelden-Echterdingen. The PXE boot environment provides an installation for RHEL and other operating system. The system cann be installed as well using the internal DVD drive (if available) or via remote ISO mount available in ExpressScope Menu. Installation of RHEL 7.2 proposed here.

• Pxe boot from network
• Choose OS installation/RHEL7.2 from boot menu
• Software selection ’Infrastructure server’

File system layout:
Standard partitions, xfs file system
/boot                                           500 MiB
/                                                50 GiB
/home                                           100 GiB
/usr                                            200 GiB
Swap                                          31,44 GiB
/var                                         (remaining space) 176,35GiB/600GB or 82,66GiB/500GB disks

* Network config: just set the hostname incl domain. <WRAP center round important 60%> Do not set IP addressing fixed to the final one. This results in aborted installation procedure. The final fixed IP needs to be set after the reboot. </WRAP>

• Ssh-keygen on one server and copy keys to other systems; edit known_hosts file and distribute. Passwordless login via ssh is mandatory between the systems
• Configure RHEL7.1 repo from inst server
• Install additional packages:

telnet 
iotop
lsscsi
sysfsutils
sg3-utils
device-mapper-multipath (dependent on Device-mapper-multipath-libs)
nfs-utils (only used for nfs mount installation export)
ntp and depending autogen-libopts
kernel-headers
kernel-devel
cpp
gcc
gcc-c++
ksh

SMutil (dependent on SMruntime) (from NetApp site or on gpfs-master)

consolidated command line:

yum install telnet iotop lsscsi sysfsutils sg3-utils device-mapper-multipath \
            nfs-utils ntp kernel-headers kernel-devel cpp gcc gcc-c++ ksh

• create LUN mapping and host access rules on storage box

• display local SAS host port identifyer on GxFS Server

lsscsi -t -H

• configure multipathd

systemctl enable multipathd
systemctl start multipathd
mpathconf --enable
systemctl restart multipathd
multipath –l

• Use build_multipath_conf.sh tool on each nsd Server to create user friendly names for the system and amend these to /etc/multipath.conf
• Check for correct name resolution using multipath –l

• configure ntp

• Provide repo for OFED 3.2 for RHEL7.1 by yum (yum.repo.d/xxx.repo)

yum grouplist | grep MLNX_OFED
yum groupinstall MLNX_OFED ALL
mlnxofedinstall --fw-update-only (theoretisch, hat er aber nicht gefunden)

Software repository can be mounted from gpfs-master:/repo
GPFS packages to be found in /repo/software/IBM/
install intended version by executing the package presented as script
after accepting the license the rpm will be copied to /usr/lpp/mmfs/<version>
delete unused *deb files in /usr/lpp/mmfs/<version>

cd /usr/lpp/mmfs/<version>
rm -f *deb

• install packages for NSD Server

use follwing script to install needed packages:

echo -n "yum install "; for i in $( for files in base adv crypto docs ext gpl \
         gskit msg; do ls | grep $files; done ); do echo -n "$i "; done; echo ""

• install GPFS packages for protocol node

echo -n "yum install "; for i in $( for files in base adv crypto docs ext gpl \
         gskit msg ganesha smb; do ls | grep $files; done ); do echo -n "$i "; done; echo ""

• install optional GPFS packages on NSD server

Just one node in a cluster needs the GUI package to be installed. The other nodes only need pmcollector and pmsensors. Please take care to take the package for the correct OS. After that start services and make startup persistent.
The scripts can as be used to install the packages on a protocol node, but the packages are distributed under various subdirectories in the installation directory.

yum install gpfs.gss.pmcollector-4.2.0-3.el7.x86_64.rpm \
            gpfs.gss.pmsensors-4.2.0-3.el7.x86_64.rpm \
            gpfs.gui-4.2.0-3.el7.x86_64.rpm

start performance monitoring on the nodes, where the collectors and sensors are installed

mmchnode --perfmon -N 1,2,3,4

start services and make them persistent

systemctl start pmcollector
systemctl start pmsensors
systemctl start gpfsgui
systemctl enable pmcollector
systemctl enable pmsensors
systemctl enable gpfsgui

same for nodes without GUI

yum install gpfs.gss.pmcollector-4.2.0-3.el7.x86_64.rpm \
            gpfs.gss.pmsensors-4.2.0-3.el7.x86_64.rpm
systemctl start pmcollector
systemctl start pmsensors
systemctl enable pmcollector
systemctl enable pmsensors

*add GPFS bin dir to $PATH

cat << EOF > /etc/profile.d/gpfsvars.sh
  export PATH=/usr/lpp/mmfs/bin:$PATH
EOF

*network configuration use nmtui to configure IPoIB device (works best in Linux shell) configure NTP (enable service persistent) disable selinux disable firewalld

• build GPFS compatibility layer

cd /usr/lpp/mmfs/src
make Autoconfig
make World
make InstallImages
optional: make rpm, in this case development tools need not to be installed on all NSD servers

rpm to distribute to be found on /root/rpmbuild

OR: use single command 'mmbuildgpl' to create the and install the compatibility layer.

mmcrcluster -N nsd00:quorum-manager,nsd01,nsd10,nsd11 -r /usr/bin/ssh -R /usr/bin/scp --ccr-disable -p nsd00 -s nsd10 -C GxFS_KULeuven
mmchlicense server --accept -N1-4

• config NSD disks

use build_stanza.sh script to create stanza file

mkdir /var/mmfs/ccr (muss wohl nicht)
firewall-cmd --permanent --add-port=1191/tcp / firewalld restart
mmcrcluster -N nsd00:quorum-manager,nsd01:quorum-manager,nsd10:quorum-manager,nsd11 -r /usr/bin/ssh -R /usr/bin/scp --ccr-enable -p nsd00 -s nsd10 -C GxFS_KULeuven
mmstartup –a
     

client config lxc3 vnfs clone <image> <image>-gpfs lxc3 vnfs mount <image>-gpfs install gpfs packages on master server use yum to install GPFS packages inside the vnfs capsule: yum –installroot /mnt/centos-7.2-1.stateful-gxfs/ install /usr/lpp/mmfs/4.2.0.1/gpfs.base-4.2.0-1.x86_64.rpm gpfs.docs-4.2.0-1.noarch.rpm gpfs.ext-4.2.0-1.x86_64.rpm gpfs.gpl-4.2.0-1.noarch.rpm gpfs.gs

Manual create link /usr/lib64 libibverbs.so.1 /usr/lib64/libibverbs.so