Site Tools


New release available: 2022-07-31 "Igor". upgrade now! [52.2] (what's this?)
xcat_debugging

after initial xcat install most recent version of pcp / dstat is installed. But no Inifiniband support is installed b default. So we have to carry out some steps

[root@ece02 ~]# yum install pcp-pmda-infiniband
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.

Last metadata expiration check: 1:48:06 ago on Mon 20 Dec 2021 05:09:54 AM EST.
Dependencies resolved.
===========================================================================================================================================================================================
 Package                                           Architecture                         Version                                    Repository                                         Size
===========================================================================================================================================================================================
Installing:
 pcp-pmda-infiniband                               x86_64                               5.1.1-3.el8                                xCAT-rhels8.3-path1                                52 k

Transaction Summary
===========================================================================================================================================================================================
Install  1 Package

Total download size: 52 k
Installed size: 42 k
Is this ok [y/N]: y
Downloading Packages:
pcp-pmda-infiniband-5.1.1-3.el8.x86_64.rpm                                                                                                                  33 MB/s |  52 kB     00:00
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                      5.1 MB/s |  52 kB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                   1/1
  Installing       : pcp-pmda-infiniband-5.1.1-3.el8.x86_64                                                                                                                            1/1
  Running scriptlet: pcp-pmda-infiniband-5.1.1-3.el8.x86_64                                                                                                                            1/1
  Verifying        : pcp-pmda-infiniband-5.1.1-3.el8.x86_64                                                                                                                            1/1
Installed products updated.

Installed:
  pcp-pmda-infiniband-5.1.1-3.el8.x86_64

Complete!
[root@ece02 ~]#

This just copies the metric files to the disk, but does not install th metrics. This needs to be doine manually.

[root@ece02 ~]# rpm -ql pcp-pmda-infiniband
/usr/lib/.build-id
/usr/lib/.build-id/84
/usr/lib/.build-id/84/237d4c7158b85da252414dcb3eadfe45095f60
/var/lib/pcp/pmdas/ib
/var/lib/pcp/pmdas/infiniband
/var/lib/pcp/pmdas/infiniband/Install
/var/lib/pcp/pmdas/infiniband/Remove
/var/lib/pcp/pmdas/infiniband/domain.h
/var/lib/pcp/pmdas/infiniband/help
/var/lib/pcp/pmdas/infiniband/pmdaib
/var/lib/pcp/pmdas/infiniband/pmns
/var/lib/pcp/pmdas/infiniband/root
[root@ece02 ~]# cd /var/lib/pcp/pmdas/infiniband/
[root@ece02 infiniband]# ls -l
total 64
-rw-r--r-- 1 root root    54 Jun 22  2020 domain.h
-rw-r--r-- 1 root root  8952 Jun 22  2020 help
-rwxr-xr-x 1 root root  1020 Jun 22  2020 Install
-rwxr-xr-x 1 root root 31416 Jun 22  2020 pmdaib
-rw-r--r-- 1 root root  1278 Jun 22  2020 pmns
-rwxr-xr-x 1 root root   675 Jun 22  2020 Remove
-rw-r--r-- 1 root root   118 Jun 22  2020 root
[root@ece02 infiniband]# pminfo | grep infi
[root@ece02 infiniband]# ./Install
Updating the Performance Metrics Name Space (PMNS) ...
Terminate PMDA if already installed ...
Updating the PMCD control file, and notifying PMCD ...
Check infiniband metrics have appeared ... 3 warnings, 44 metrics and 96 values
[root@ece02 infiniband]#

after that the available metrics can be displyed.

[root@ece02 infiniband]# pminfo | grep infi
infiniband.hca.type
infiniband.hca.ca_type
infiniband.hca.numports
infiniband.hca.fw_ver
infiniband.hca.hw_ver
infiniband.hca.node_guid
infiniband.hca.system_guid
infiniband.port.guid
infiniband.port.gid_prefix
infiniband.port.lid
infiniband.port.state
infiniband.port.phystate
infiniband.port.rate
infiniband.port.capabilities
infiniband.port.linkspeed
infiniband.port.linkwidth
infiniband.port.in.bytes
infiniband.port.in.packets
infiniband.port.in.errors.drop
infiniband.port.in.errors.filter
infiniband.port.in.errors.local
infiniband.port.in.errors.remote
infiniband.port.out.bytes
infiniband.port.out.packets
infiniband.port.out.errors.drop
infiniband.port.out.errors.filter
infiniband.port.total.bytes
infiniband.port.total.packets
infiniband.port.total.errors.drop
infiniband.port.total.errors.filter
infiniband.port.total.errors.link
infiniband.port.total.errors.recover
infiniband.port.total.errors.integrity
infiniband.port.total.errors.vl15
infiniband.port.total.errors.overrun
infiniband.port.total.errors.symbol
infiniband.port.switch.in.bytes
infiniband.port.switch.in.packets
infiniband.port.switch.out.bytes
infiniband.port.switch.out.packets
infiniband.port.switch.total.bytes
infiniband.port.switch.total.packets
infiniband.control.query_timeout
infiniband.control.hiwat
[root@ece02 infiniband]#

to be able to run dstat on the Infiniband interfaces pmlogger service needs to be started,

[root@ece02 ~]# systemctl start pmlogger
[root@ece02 ~]# systemctl status pmcd
● pmcd.service - Performance Metrics Collector Daemon
   Loaded: loaded (/usr/lib/systemd/system/pmcd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2021-12-20 06:59:17 EST; 2s ago
     Docs: man:pmcd(8)
 Main PID: 2562262 (pmcd)
    Tasks: 7 (limit: 821611)
   Memory: 25.4M
   CGroup: /system.slice/pmcd.service
           ├─2562262 /usr/libexec/pcp/bin/pmcd
           ├─2562267 /var/lib/pcp/pmdas/root/pmdaroot
           ├─2562268 /var/lib/pcp/pmdas/proc/pmdaproc -d 3
           ├─2562269 /var/lib/pcp/pmdas/xfs/pmdaxfs -d 11
           ├─2562270 /var/lib/pcp/pmdas/linux/pmdalinux
           └─2562271 /var/lib/pcp/pmdas/kvm/pmdakvm -d 95

Dec 20 06:59:17 ece02 systemd[1]: Starting Performance Metrics Collector Daemon...
Dec 20 06:59:17 ece02 systemd[1]: Started Performance Metrics Collector Daemon.
Dec 20 06:59:17 ece02 pmcd[2562178]: Starting pmcd ...
[root@ece02 ~]#

now the infiniband metrics can be displayed using pmrep

[root@ece02 ~]# pmrep infiniband.port.total.bytes
  i.p.t.bytes  i.p.t.bytes
     mlx5_0:1     mlx5_1:1
       byte/s       byte/s
          N/A          N/A
          N/A          N/A
  97646040.19  97651127.93
  99513503.32  105867137.8
  108217616.5  97610383.27
  97690812.72  99815321.40
  97589942.74  95467309.41
  95319535.02  97438435.35
  97600935.37  99715374.78
  97698589.43  95577573.43
  97455162.52  105936571.8
  108344906.7  97722089.75
  97525418.99  101770097.7
  50951377.69  48829485.63
  87192635.21  95396909.82
  69722740.06  59396511.78
  97480257.66  97481991.94
  67976250.82  67975069.17
  84854137.89  86973124.06
^C
[root@ece02 ~]#
xcat_debugging.txt · Last modified: 2022/07/10 09:34 by 127.0.0.1