1. Verifying Functionality and Performance

When installing the eXpressWare software stack (which includes SISCI, SuperSockets and the IPoPCIe (TCP/IP driver) via the SIA, the basic functionality and performance is verified at the end of the installation process by some of the same tests that are described in the following sections. This means, that if the tests performed by the SIA did not report any errors, it is very likely that both, the software and hardware work correctly.

The following sections describe how to verify that the interconnect is setup correctly, which means that all Cluster Nodes can communicate with all other Cluster Nodes via the PCI Express interconnect by sending low-level control packets and performing remote memory access.

1.1. Availability of Drivers and Services

The Cluster Management Node functionality is optional for SISCI based applications but currently mandatory for SuperSockets operation. The Cluster Management Node will automatically distribute configuration files and simplify diagnostic of the cluster. On the Cluster Management Node, only the user-space service dis_networkmgr (the central Network Manager) needs to be running.

Without the required drivers and services running on a Cluster Node, the node will fail to communicate with other nodes. On the Cluster Nodes, the kernel module dis_kosif (providing operating system dependent functionality to all other kernel modules), the kernel services dis_irm (interconnect resources driver) and dis_sisci (upper level hardware services) and dis_ssocks need to be running. Next to these kernel drivers, the user-space service dis_nodemgr (node manager, which talks to the central Network Manager) needs to be active for configuration and monitoring.

Because the drivers do also appear as services, you can query their status with the usual tools of the installed operating system distribution. I.e., for Red Hat-based Linux distributions, you can do

# service dis_irm status
Dolphin IRM 5.5.0 (  January 10th 2018 ) is running.

Dolphin provides a script dis_services that performs this task for all Dolphin services installed on a machine. It is used in the same way as the individual service command provided by the distribution:

# dis_services status
Dolphin KOSIF 5.5.0 is running
Dolphin MX 5.5.0 (  January 10th 2018 ) is running.
Dolphin IRM 5.5.0 (  January 10th 2018 ) is running.
Dolphin Node Manager is running (pid 3172).
Dolphin SISCI 5.5.0 (  January 10th 2018 ) is running.
Dolphin SuperSockets 5.5.0 "Express Train", January 10th 2018 (built January 10th
2018) running.

If any of the required services is not running, you will find more information on the problem that may have occurred in the system log facilities. Call dmesg to inspect the kernel messages, and check /var/log/messages for related messages.

1.2. PCIe Connection Test

To ensure that the cluster is cabled correctly, please perform the PCIe connection test as described in Chapter 4, Initial Installation, Section 2.8.4, “PCIe connection Test”.

1.3. Static PCIe Interconnect Test - dis_diag

The static interconnect test makes sures that all PCIe communication hardware are working correctly by performing a self-test, and determines if the setup and the PCIe routing is correct (matches the actual hardware topology). It will also check all PCIe connections, but this has already been done in the PCIe Connection Test. The tool to perform this test is dis_diag (default location /opt/DIS/sbin/dis_diag).

Running dis_diag on a Cluster Node will perform a self test on the local adapter(s) and list all remote adapters that this adapter can see via the PCI Express interconnect. This means, to perform the static interconnect test on a full cluster, you will basically need to run dis_diag on each Cluster Node and see if any problems with the adapter are reported, and if the adapters in each Cluster Node can see all remote adapters installed in the other Cluster Nodes.

Normally you should invoke dis_diag with no arguments, and it will do a general test and only show the most interesting information. Advanced users may want to enable the full verbose mode by using the -V 9 command line option:

	  dis_diag -V 9	

The -V 9 option will generate a lot of information, some parts of the information requires knowledge about the PCIe chipset and the PCIe specification in general. The diagnostic module will collect various usage and error information over time. This can be cleared by using the -clear command line option:

	  dis_diag -clear	

An example output of dis_diag for a Cluster Node which is part of a 2 node cluster and using one MXH830 adapter per Cluster Node looks like this:

[root@Hetty ~]# /opt/DIS/sbin/dis_diag 
  Dolphin diagnostic tool --  dis_diag version 5.5.0 ( Tue Jan 10th 14:54:02 CEST 2018 )
dis_diag compiled in 64 bit mode
Driver : Dolphin IRM (GX) Jan 10th 2018 (rev  39220)
Date   : Wed Mar 29 19:02:52 CEST 2018
System : Linux Maxwell 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC 2017
         x86_64 x86_64 x86_64 GNU/Linux

Number of configured local adapters found: 1

Adapter 0 > Type                       : MXH830
            NodeId                     : 4
            Serial number              : MXH830-AA-000008
            MXH chip family            : Microchip - PFX
            MXH chip vendorId          : 0x11f8
            MXH chip device            : 0x0830
            MXH chip revision          : 0x0 (ZA)
            EEPROM version NTB mode    : 01
            EEPROM vendor info         : 0x0000
            EEPROM swmode[3:0]         : 0111
            Card revision              : AA
            Topology type              : Not determined
            Topology Autodetect        : No
            Number of enabled links    : 1
            Max payload size (MPS)     : 256
            Multicast group size       : 2 MB
            Prefetchable memory size   : 4096 MB (BAR2)
            Non-prefetchable size      : NA
            Clock mode slot            : Global
            Clock mode link            : Global
            PCIe slot state            : x0, Gen3 (8 GT/s)
            PCIe slot capabilities     : x16, Gen3 (8 GT/s)

*************************  MXH ADAPTER 0 LINK 0 STATE  *************************

            Link 0 downtime            : 2 seconds
            Link 0 state               : DISABLED
            Link 0 configuration       : NTB
            Link 0 required            : x4, Gen1 (2.5 GT/s)
            Link 0 capabilities        : NA

***************  MXH ADAPTER 0, PARTNER INFORMATION FOR LINK 0  ***************

            Partner information not available

*****************************  TEST OF ADAPTER 0  *****************************

OK: MXH chip alive in adapter 0.
OK: Link alive in adapter 0.
==> Local adapter 0 ok.
[WARN] Node 8 not reachable, but is an entry in the dishosts.conf file

************************  TOPOLOGY SEEN FROM ADAPTER 0  ************************

Adapters found: 2
----- List of all nodes found:

Nodes detected:   0004  0008 

***********************  SESSION STATUS FROM ADAPTER 0  ***********************

Node 4: Session valid
Node 8: Session valid

dis_diag discovered 0 note(s).
dis_diag discovered 0 warning(s).
dis_diag discovered 0 error(s).

The static interconnect test passes if dis_diag delivers TEST RESULT: *PASSED* and reports the same topology (remote adapters) on all Cluster Nodes.

1.4. Interconnect Load Test

While the static interconnect test sends very a few packets over the links to probe remote nodes, the Interconnect Load Test puts significant stress on the interconnect and observes if any data transmissions have to be retried due to link errors. This can happen if cables are not correctly connected, i.e. plugged in without connector latches locking correctly. Before running this test, make sure your cluster is connected and configured correctly by running the tests described in the previous sections.

1.4.1. Test Execution from Dolphin dis_admin GUI

This test can be performed from within the Dolphin dis_admin GUI tool. Please refer to Appendix B, dis_admin Reference for details.

1.5. Interconnect Performance Test

Once the correct installation and setup and the basic functionality of the interconnect have been verified, it is possible to perform a set of low-level benchmarks to determine the base-line performance of the interconnect without any additional software layers. The tests that are relevant for this are scibench2 (streaming remote memory PIO access performance), scipp (request-response remote memory PIO write performance), dma_bench (streaming remote memory DMA access performance) and intr_bench (remote interrupt performance).

All these tests need to run on two Cluster Nodes (A and B) and are started in the same manner:

  1. Determine the NodeId of both Cluster Nodes using the query command (default path /opt/DIS/bin/query ). The NodeId is reported as "Local node-id".

  2. On node A, start the server-side benchmark with the options -server and -rn <NodeId of B>, like:

    $ scibench2 -server -rn 8
  3. On Cluster Node B, start the client-side benchmark with the options -client and -rn <NodeId of A>, like:

    $ scibench2 -client -rn 4
  4. The test results are reported by the client.


The following results are measured using the MXH930 card (Gen4, x16)

  Message    Total     Vector     Transfer     Transfer time      Bandwidth
   size      size      length       time        per message
     64      16384      256        69.55 us       0.27 us       235.57 MBytes/s
    128      32768      256        70.07 us       0.27 us       467.63 MBytes/s
    256      65536      256        66.13 us       0.26 us       991.08 MBytes/s
    512     131072      256        64.77 us       0.25 us      2023.65 MBytes/s
   1024     262144      256        65.98 us       0.26 us      3973.08 MBytes/s
   2048     524288      256        69.25 us       0.27 us      7570.95 MBytes/s
   4096    1048576      256        76.81 us       0.30 us      13651.38 MBytes/s
   8192    2097152      256        94.94 us       0.37 us      22090.40 MBytes/s
  16384    4194304      256       162.86 us       0.64 us      25753.73 MBytes/s
  32768    8388608      256       312.17 us       1.22 us      26871.84 MBytes/s
  65536    16777216      256       611.22 us       2.39 us      27448.83 MBytes/s
 131072    33554432      256      1209.15 us       4.72 us      27750.48 MBytes/s
 262144    67108864      256      2405.18 us       9.40 us      27901.78 MBytes/s
 524288    67108864      128      2402.11 us      18.77 us      27937.52 MBytes/s
1048576    67108864       64      2402.21 us      37.53 us      27936.29 MBytes/s
2097152    67108864       32      2400.34 us      75.01 us      27958.01 MBytes/s
4194304    67108864       16      2400.34 us     150.02 us      27958.12 MBytes/s
8388608    67108864        8      2400.29 us     300.04 us      27958.67 MBytes/s
16777216    67108864        4      2400.31 us     600.08 us      27958.42 MBytes/s
33554432    67108864        2      2400.15 us     1200.07 us      27960.29 MBytes/s
67108864    67108864        1      2400.20 us     2400.20 us      27959.73 MBytes/s

The scipp SISCI benchmark sends a message of the specified size to the remote system. The remote system is polling for incoming data and will send a similar message back to the first node.

The minimal round-trip latency for writing to remote memory is extremely low using PCI Express networks.

The following results are measured using a MXH830 card (Gen3 x16).

Ping Pong data transfer:
  size       retries  latency (usec)  latency/2 (usec)
     0           248           1.391             0.695
     4           495           1.258             0.629
     8           499           1.259             0.629
    16           499           1.260             0.630
    32           507           1.290             0.645
    64           516           1.304             0.652
   128           515           1.309             0.654
   256           532           1.376             0.688
   512           552           1.457             0.728
  1024           582           1.582             0.791
  2048           654           1.897             0.949
  4096           804           2.542             1.271
  8192          1106           3.854             1.927

The interrupt latency is affected by the operating system and can therefore vary.

Average unidirectional interrupt time :        2.515 us.
Average round trip     interrupt time :        5.030 us.

To simply gather all relevant low-level performance data, the script sisci_benchmarks.sh can be called in the same way. It will run all of the described tests.