Dolphin eXpressWare Installation and Reference Guide
		Next

Dolphin eXpressWare Installation and Reference Guide

For users of :SISCI, SuperSockets, IPoPCIe

Windows version

Dolphin Interconnect Solutions

This document describes the installation and usage of Dolphins PCI Express software stack (eXpressWare) version 5.19.x in combination with Dolphin MX interconnect hardware.

Copyright © 2022 Dolphin Interconnect Solutions

Published under Gnu Public License v2

March 8th, 2022

Table of Contents

1. Introduction & Overview

1. Who needs a PCI Express Network?

2. eXpressWare

2.1. SuperSockets
2.2. SISCI
2.3. Device Lending
2.4. Transparent Hot Add
2.5. TCP/IP Driver
2.6. eXpressWare Licensing

3. Dolphin PCI Express PX Adapter cards

3.1. PXH810
3.2. PXH830
3.3. PXH820
3.4. PXH840

4. Dolphin PCI Express MX Adapter cards

4.1. MXH830
4.2. MXH930
4.3. MXH940
4.4. MXH950
4.5. MXP924
4.6. MXP908
4.7. MXP909

5. Dolphin PCI Express IX Adapter cards

5.1. IXH610
5.2. IXH620

6. Dolphin PCI Express Switches

6.1. IXS600
6.2. MXS824
6.3. MXS924

7. PCI Express Cables

7.1. iPass Cables
7.2. SFF-8644 / PCIe 3.0 / Cables
7.3. SFF-8644 / PCIe 4.0 Cables

8. Supported OEM hardware

8.1. Curtiss-Wright SBCs and switches
8.2. Keysight PXIe Controllers

9. PCI Express networking basics

10. What's Special about PCI Express and SuperSockets?

11. Contents of this Document

12. Terminology

13. Support

2. Quick Installation Guide

3. Requirements and Planning

1. Supported Platforms

1.1. Application recommendations
1.2. Supported Platforms
1.3. Recommended Cluster Node Hardware
1.4. Recommended Cluster Management Hardware

2. Software Support

2.1. Linux
2.2. Windows
2.3. VxWorks

3. Interconnect Planning

3.1. Nodes to Equip with PCI Express Interconnect

4. Dolphin MX Interconnect Configurations and topologies

4.1. Switchless topologies using the MXH830 NTB host adapter card
4.2. PCIe switch based topologies using the MXH830 NTB host adapter card and the MXS824 switch

5. Physical Node Placement

4. Initial Installation

1. Installation Overview

1.1. System installation overview
1.2. Installing the Cluster Management software
1.3. Install the Cluster Node software
1.4. Installation Result

2. Software Installation

2.1. Running the Dolphin Installer

2.2. Commandline installation

2.2.1. Install Standard
2.2.2. Manager
2.2.3. Uninstall

2.3. Post installation

2.4. Working with the Dolphin Network Configurator, dis_netconfig

2.4.1. Cluster Edit
2.4.2. Node Arrangement
2.4.3. Cabling Instructions

2.5. Cluster Cabling

2.5.1. Connecting the PCI Express cables
2.5.2. Verifying the Cabling

2.6. Handling Installation Problems

2.7. Interconnect Validation using the management GUI

2.7.1. Installing dis_admin
2.7.2. Starting dis_admin
2.7.3. Cluster Overview
2.7.4. PCIe connection Test

2.8. Interconnect Validation using the command line

2.9. Making Cluster Application use PCI Express

2.9.1. Generic Socket Applications
2.9.2. Native SISCI Applications

5. Interconnect Maintenance

1. Verifying Functionality and Performance

1.1. Availability of Drivers and Services

1.2. PCIe Connection Test

1.3. Static PCIe Interconnect Test - dis_diag

1.4. Interconnect Load Test

1.4.1. Test Execution from Dolphin dis_admin GUI
1.4.2. Test Execution from Command Line

1.5. Interconnect Performance Test

2. Replacing Interconnect Cables

3. Replacing an Adapter

4. Physically Moving Nodes

5. Replacing a Node

6. Adding Nodes

7. Removing Nodes

6. Configuring the adapter card

1. DIP Switches

2. PCIe Prefetch Memory size

2.1. Determine the PCIe Prefetch Memory Size
2.2. Configuring the PCIe Prefetch Memory Size

7. Firmware upgrade

1. Dolphin Adapter card firmware upgrade.
2. MXS824 PCIe Gen3 Switch firmware upgrade
3. MXS924 PCIe Gen4 Switch firmware upgrade

8. SISCI API

1. SISCI Documentation and resources

2. Enable applications to use the SISCI API

3. How to compile your own SISCI application

4. SISCI API Demo and Example programs

4.1. SISCI API Example programs

4.1.1. shmem
4.1.2. memcopy
4.1.3. interrupt
4.1.4. data_interrupt
4.1.5. intcb
4.1.6. lsegcb
4.1.7. rsegcb
4.1.8. dma
4.1.9. dmacb
4.1.10. dmavec
4.1.11. rpcia
4.1.12. smartio_example
4.1.13. reflective_memory
4.1.14. reflective_device
4.1.15. reflective_device_receive
4.1.16. reflective_write
4.1.17. probe
4.1.18. query

4.2. SISCI API demo and benchmarks programs

4.2.1. scibench2
4.2.2. scipp
4.2.3. dma_bench
4.2.4. intr_bench
4.2.5. reflective_bench

9. Dolphin PCI Express TCP/IP Driver

1. Who should use the TCP/IP driver

10. SuperSockets

1. Installation

2. Make Generic Windows Applications use SuperSockets

3. SuperSockets Functionality and Performance

3.1. SuperSockets Benchmarks

3.1.1. latency_bench

11. SmartIO

12. Transparent Board Management

1. Installation
2. Diagnostic
3. Firmware upgrade

13. Advanced Topics

1. Installing eXpressWare OEM license

2. Managing PCIe and eXpressWare Resources

2.1. Allocating Large SISCI Segments
2.2. Connecting to Large Remote SISCI Segments
2.3. Creating large SISCI reflective memory Segments
2.4. Updates with Modified eXpressWare Configuration Files

3. Using dis_diag

1. Dolphin PCI Express Hardware
2. Software

A. dis_admin Reference

2. Interconnect Status View

2.2. Operation

2.2.1. Cluster Status
2.2.2. Node Status

3. Node and Interconnect Control

3.1. Admin Menu
3.2. Cluster Menu
3.3. Node Menu
3.4. Cluster Settings
3.5. Adapter Settings

4. Interconnect Testing & Diagnosis

4.1. Cable Test
4.2. Fabric Test

5. Troubleshooting Best Practice

B. Configuration Files

1. Cluster Configuration

1.1. dishosts.conf

1.1.1. Basic settings
1.1.2. Persistent adapter numbers
1.1.3. SuperSockets settings
1.1.4. Miscellaneous Notes

1.2. networkmanager.conf

1.3. cluster.conf

2. eXpressWare Driver Configuration

2.1. dis_mx.conf

2.1.1. Memory Preallocation
2.1.2. Cluster Page Size
2.1.3. Configuring multicast memory

2.2. dis_irm.conf

2.2.1. Resource Limitations
2.2.2. Interrupt and DMA polling
2.2.3. Real time behavior
2.2.4. Memory Preallocation
2.2.5. Logging and Messages

C. Platform Issues and Software Limitations

1. Platforms with Known Problems vs Dolphin PCI Express MX
2. Platforms with Known Problems vs Dolphin PCI Express software
3. IRM
4. SISCI
5. SuperSockets

D. Example Scripts

1. Software Stack Installation

E. eXpressWare License text

List of Figures

2.1. Windows Installer: Setup Type
2.2. Windows Installer: Custom Setup
2.3. Windows Installer: NDIS feature
4.1. Windows Installer: Icon
4.2. Windows Installer: Welcome Message
4.3. Windows Installer: Setup Type
4.4. Windows Installer: Custom Setup
4.5. Windows Installer: Installing Software
4.6. Windows Installer: Cancel Wizard on Windows XP and Windows Server 2003
4.7. Windows Installer: Continue Anyway on Windows XP and Windows Server 2003
4.8. Windows Installer: Configure Firewall
4.9. Windows Installer: Completed
4.10. Windows Installer: Device Manager
4.11. msiexec help menu
4.12. Cluster Edit dialog of dis_netconfig
4.13. Main dialog of dis_netconfig
4.14. Node dialog of dis_netconfig
4.15. Starting dis_admin
4.16. Dolphin dis_admin GUI - Connect to Network Manager
4.17. Dolphin dis_admin GUI - Representation of topology
A.1. Fabric has FAILED due to dead Cluster Nodes
A.2. Options in the Admin menu
A.3. Options in the Cluster menu
A.4. Options in the Node menu
A.5. Cluster configuration in dis_admin
A.6. Advanced settings for a Cluster Node
A.7. Result of running cable test on a good cluster
A.8. Result of cable test on a problematic cluster
A.9. Result of fabric test without installing all the necessary rpms
A.10. Result of fabric test on a proper fabric

List of Tables

A.1. Node or Adapter State
A.2. Link State

		Next
		Abstract