Site Tools


New release available: 2022-07-31 "Igor". upgrade now! [52.2] (what's this?)
bmsystemconf_new

good performing BMsystem setup for LE GPFS test system

[root@gpfs-master ~]# ssh gxfs1
Last login: Thu Apr  4 09:47:34 2019 from 10.0.0.177
[root@gxfssrv1 ~]# mmlsconfig
Configuration data for cluster nec-gxfs-cluster.gxfs1:
------------------------------------------------------
clusterName nec-gxfs-cluster.gxfs1
clusterId 12947237399892212317
autoload no
dmapiFileHandleSize 32
minReleaseLevel 5.0.1.0
cipherList AUTHONLY
maxblocksize 16M
verbsRdma enable
cesSharedRoot /gpfs/ddpfs/ces
ccrEnabled yes
[gxfs1bm,gxfs2bm,hsw001,hsw003,hsw004,hsw005,hsw025,hsw027,hsw028,hsw031,sky21,skye1,skye2,skye3]
verbsPorts mlx5_0
[hsw017,hsw019,hsw021]
verbsPorts mlx4_0
[skye4]
verbsPorts mlx5_1
[common]
pagepool 8G
[gxfs1bm,gxfs2bm]
pagepool 64G
[common]
workerThreads 1k
nsdMaxWorkerThreads 2k
nsdSmallThreadRatio 1
maxFilesToCache 200000
maxMBpS 20000
prefetchPct 60
ignorePrefetchLUNCount yes
adminMode central

File systems in cluster nec-gxfs-cluster.gxfs1:
-----------------------------------------------
/dev/ddpfs
[root@gxfssrv1 ~]#

optimized config for Infiniband in BM Center. <hi #fff200>Timestamp: 20220503</hi>

[root@gxfs01 ~]# mmlsconfig
Configuration data for cluster gxfs1.cluster:
---------------------------------------------
clusterName gxfs1.cluster
clusterId 12947237400007371149
minReleaseLevel 5.1.1.0
ccrEnabled yes
autoload no
dmapiFileHandleSize 32
verbsRdma enable
nsdSmallThreadRatio 1
nsdMaxWorkerThreads 2048
maxblocksize 16M
cipherList AUTHONLY
prefetchPct 50
workerThreads 1024
ignorePrefetchLUNCount yes
pagepool 8G
[gxfs01,gxfs02]
pagepool 64G
[cnode00,cnode01,cnode02,cnode03]
verbsPorts hfi1_0
[cnode10,cnode11,cnode12,cnode13,gxfs01,gxfs02]
verbsPorts mlx5_0
[common]
maxFilesToCache 200000
maxMBpS 22000
nsdMinWorkerThreads 2048
nsdThreadsPerDisk 12
nsdThreadsPerQueue 10
numaMemoryInterleave yes
prefetchAggressiveness 3
scatterBufferSize 524288
verbsRdmasPerConnection 16
tiebreakerDisks SNA800_DDP01_01
adminMode central

File systems in cluster gxfs1.cluster:
--------------------------------------
/dev/sna800
[root@gxfs01 ~]#

tuning results with different settings from the last config from 20020503 - verbsRdmasPerConnection: good results with 16, acceptable results with 20, worse results with 24

bmsystemconf_new.txt · Last modified: 2022/07/10 09:34 by 127.0.0.1