eXpressWare Software is a complete software suite for PCI Express
Dolphins PCI Express software suite is named eXpressWare and enables applications to communicate over PCI Express cables and backplanes. Several interfaces and APIs are supported, from standard TCP/IP networking to the lowest level direct remote memory access. Each API has its benefits and can be selected based on application requirements. The complete stack can be seen in the figure below:
SISCI - Shared memory API
The SISCI API (Software Infrastructure Shared-Memory Cluster Interconnect) enables customers to fully exploit the PCI Express programming model without having to spend months to develop device drivers and read low level PCIe switch chip specifications. The SISCI API offers a C programming API for shared/remote memory and device access, including reflective memory functionality. The SISCI API also supports direct peer to peer transfers and remote access to FPGAs, GPUs and NVMes communication over PCIe.
The SISCI API supports both local and networked communication, enabling applications to easily scale from a multiprocessor system to a larger PCIe network.
Dolphin’s SISCI Developers Kit consists of driver and API software, tools, benchmarks, documentation and source needed to develop your own embedded application utilizing the low latency and high performance of a PCI Express Network.
Applications that will benefit from using SISCI
- Embedded control and measurement systems
- Reflective memory applications
- HPC / GPU RDMA applications
- Low latency electronic trading
- Real time simulators
- High availability applications
Applications that need fast message passing and is already supporting regular networking should normally use SuperSockets.
The Dolphin software distribution package contains SISCI application example, demo and benchmark programs that easily can be used by customers to verify performance and functionality. The SISCI API is available for Linux, Windows, RTX and VxWorks on several hardware platforms. More details on the SISCI API.
SmartIO
The SmartIO software enables native access to PCIe devices installed in a remote server and offers a uniform way to access devices independent of their physical location. The SmartIO software currently consists of the core SmartIO drivers and the Device Lending software and the SISCI API SmartIO extension.
Device Lending
Device Lending software is a part of the SmartIO functionality and offers a flexible way to enable PCIe IO devices (NVMes, FPGAs, GPUs etc) to be accessed within a PCIe Network. Devices can be borrowed over the PCIe network without any software overhead at the performance of PCI Express. Device Lending is a simple way to reconfigure systems and reallocate resources. GPUs, NVMe drives or FPGAs can be added or removed without having to be physically installed in a particular system on the network. The result is a flexible and simple method of creating a pool of devices that maximizes usage. More details on Device Lending.
Device Lending with NVMes implements a very fast NVMe over PCIe Fabric solution.
SISCI API SmartIO extension
The SISCI API extension extends the existing SISCI API with functionality to enable programmers to access PCIe devices managed by the SmartIO driver. The API enables one or many computers to concurrently access a PCIe device and implement advanced types of device sharing. More details on SISCI API SmartIO extension.
SuperSockets
SuperSockets is Dolphins optimized networking software. It is compliant with the Berkeley sockets and the WinSock2 API, but offers a significant performance improvement over Ethernet based networks. The solution is recommended for commercial and embedded networked applications and can be deployed without any changes to the application.
SuperSockets enabled networked applications benefit from low latency, high throughput PCI Express networks without any modifications to the application code. SuperSockets is currently available for both Windows and Linux systems. For Linux, there are two implementations, the kernel implementation and a user space implementation.
Loopback support
SuperSockets is optimized for communication with systems connected via a PCIe network, but both the Windows and Linux kernel space implementation of SuperSockets will accelerate loopback/local address communication up to 10 times compared to standard networking stack. This feature is automatically enabled when the SuperSockets software is installed and configured. More details on loopback support.
Linux kernel Space Implementation of SuperSockets
This is our standard recommendation for Linux systems. It implements a new socket address family AF_SSOCKS, and support virtually all networked Linux applications. The software is also compliant with the Linux Kernel Sockets API and can also be used by kernel services that can be configured to use AF_SSOCKS (decimal value 27). This version of SuperSockets includes automatic fail-over to Ethernet if there is a failure with the PCIe network.
Applications that will benefit from using the Kernel Space implementation of SuperSockets
- General networked Linux applications
- DRBD
- MySQL Cluster / NDB
More information on SuperSockets for Linux.
Linux User Space Implementation of SuperSockets
Dolphins User Space implementation of SuperSockets is implemented to support embedded applications. The implementation is slightly faster than the Linux Kernel Space implementation, but does not support the full set of socket functions. Applications using the Linux User Space implementation can also communicate with applications running on Windows.
Applications that will benefit from using the User Space implementation of SuperSockets
- Embedded applications that need very low latency message passing
- Socket connectivity between Linux and Windows based systems.
Windows implementation of SuperSockets
Dolphin SuperSockets for Windows supports the basic WinSock2 feature set and is implemented to support embedded applications. This library also supports communication with Linux systems running the Linux User Space implementation of SuperSockets. SuperSockets on Windows supports regular Windows applications but can't be used by Windows system and kernel services (like Windows explorer map network drive).
Applications that will benefit from using SuperSockets on Windows
- General networked Windows applications
More information on SuperSockets for Windows.
IP driver for PCIe
The IPoPCIe driver supports regular networking over PCI Express and is typically used as a faster alternative to 10G/40/100G Ethernet. The IPoPCIe driver is integrated with the standard networking stack and fully support routing and ARP. The current driver is available for both Linux and Windows, but does not support communication between Linux and Windows (This is on the Roadmap, please contact Dolphin for details). Most regular applications should use SuperSockets for best possible performance.
Applications that will benefit from using the IP driver on Linux
- Linux kernel services that can't be adopted to use SuperSockets
- NFS
- Clustered File Systems
Applications that will benefit from using the IP driver on Windows
- All Windows kernel services
More details on the IPoPCIe driver.
Transparent Adapter Management Software
Starting from release DIS 5.5.0, eXpressWare now also includes support for managing and monitoring transparent PCIe adapter cards. More details on the Transparent Adapter Management Software.
MPI
NMPI is an open source version of MPICH2 with a native SISCI transport. More details on NMPI.
The SuperSockets software supports all MPI libraries configured for Ethernet communication.
IRM - Interconnect Resource Manager
The IRM is a central component of the Dolphin stack - it has the responsibility to manage all resources for all higher level software. This enables higher level software to be written in an independent way. When certain resources are required, the IRM will manage and distribute the available resources based on internal policies and configuration settings.
The IRM is also responsible for bringing up the interconnect after power up and hot plug events. The IRM implements its own low cost heart beat mechanisms over PCI Express to detect unresponsive systems and links. The IRM is also responsible for handling hot plugging of nodes, handling of nodes being rebooted.
The IRM services and resources are available through the GENIF interface. The SISCI Driver, TCP/IP driver, SuperSockets driver etc are typically clients of the GENIF interface. The GENIF interface specification is available and customers can write their own GENIF client if required. Typical resources managed by the IRM:
- Remote mapping resources / NTB Luts
- Interrupts
- DMA
PCI Express network management
Dolphin's eXpressWare software includes an optional PCI Express network manager that helps customers to install, configure and monitor the PCI Express network. The management software includes an graphical tool that is very convenient for larger systems. The graphical tool is written in QT and is supported on both Linux and Windows. You can e.g. easily use a Windows Laptop to monitor a Linux PCI Express network.
The network management software uses standard Ethernet as an out of band channel between all systems to easily collect and report issues with the high speed PCI Express network. The network management software is optional and designed to be light weight and will not put a high load on the Ethernet.
Support for multiple operating systems
The core Dolphin software stack is designed to be independent of the operating system. Special operating system dependent code is organized into a separate library. This gives huge benefits, both lowering the cost to include support for a new operating systems as well as being able to test independent code on several platforms. The following operating systems are currently supported by the Dolphin PCI Express software, pease follow the links for more information:
Supported hardware configurations and chipsets
The Dolphin software suite is written to be very portable and supports a variety of PCI Express NTB switches from Microchip, IDT and PLX/Broadcom as well as the built in Intel NTB functionality available with some Intel CPUs. The software is normally a bundle with Dolphin PCI Express hardware products, but can also be licensed by OEMs designing their own hardware.
The software supports setting up direct NTB to NTB connections, switched topologies and multi-port topologies.
Software road map
Dolphin is offering a complete solution for host-to-host communication over PCI express today. Future software plans includes virtualization support and integration with transparent IO devices to create a unified PCI Express environments where systems and IO devices can directly communicate over PCI Express. Please contact Dolphin for more information.