Flexible Access to PCIe Devices
The SISCI API has since 1998 provided a stable and super low latency API for various networks supporting remote access to memory. Dolphin has recently developed an extension to the SISCI API (SISCI API SmartIO), enabling customers to easily work with local and remote PCIe devices like FPGAs, GPUs and NVMes.
The SISCI API SmartIO extension makes it easy and convenient for SISCI application developers to access and use PCIe devices within a PCIe fabric. PCIe devices located in the local server, in a remote server or directly connected to the PCIe fabric can be managed by the API. The SmartIO software provides a solution where you don't need to care where the device is located, local or remote - it is available over PCIe. The result is extremely low latency and performance close to the wire speed or device speed. Accessing a remote PCIe device is just a few hundred nanoseconds extra (The cut through latency of 2 PCIe switches).
The SISCI API SmartIO extension doesn't follow the traditional device driver model, but is a flexible API that enables several systems concurrent access one or more devices. Once concurrent access is established, the application programmer is required to manage and control the concurrent access to the devices. Some devices, by nature, can not be concurrently shared, in this case, the programmer needs to ensure only a single system is accessing the device. (This can be done using a distributed lock manager running over PCIe). Other devices, like nonvolatile memory boards, FPGAs and NVMes may be easier so share.
Device Lending
If you would like to access a remote PCIe device but want to use an unmodified driver for your PCIe device, please consider the PCIe Device Lending software solution.
Status
The SISCI API SmartIO extension is implemented and is available as a part of the Dolphin Linux eXpressWare software 5.5.0 for PXH820, PXH830, PXH840 and MXH830 cards. eXpressWare version 5.19 or newer is required for the PCIe Gen4 MXH930, MXH940, MXH950 cards.
SISCI API SmartIO support for Windows is currently being tested, please contact Dolphin for more information.
The SISCI API Smart IO extension is undergoing enhancements and changes may still occur to the API based on feedback from customers. The current SISCI API can be found here - look for the SmartIO section.
NVMe Example Driver
As a test case of the new SmartIO API, we have implemented a user space NVMe driver that allows a single-function NVMe namespace to be shared concurrently among multiple computers connected via a PCIe fabric.
This is possible by exploiting the inherent parallel design of NVMe specification and assigning IO queue pairs to different hosts. SmartIO sets up NT mappings behind the scenes, making IO queues and user space data buffers accessible directly from the NVMe controller. In other words, the controller accesses command queues and buffers over the NTB.
While this requires implementing a custom driver or extending existing drivers with SmartIO support, it means that it is possible to share a single-function NVMe SSD concurrently between multiple nodes and move data using an optimal data path directly into an application buffer or a GPU buffer.
Future extensions
The current SISCI API SmartIO extension does not support forwarding Device Interrupts. This will be supported in the future.
The SISCI API is currently available for Linux, Windows, RTX and VxWorks. The SmartIO extension is currently only available with Linux and being tested with Windows.
Please contact Dolphin for more information.