The eXpressWare software stack and the PCIe NTB cards from Dolphin comes with a set of default settings that will make it easy to use for may use cases and PCIe fabric configurations. However, demanding application resource requirements and PCI Express configurations may require some level of tuning to maximize performance or resource usage.
A number of resources in the eXpressWare stack are run-time limited by parameter in the driver configuration file /opt/DIS/lib/modules/dis_irm.conf and /opt/DIS/lib/modules/dis_ix.conf (for the default installation path). These files contains numerous parameter settings; for those parameters that are relevant for changes by the user, please refer to Appendix C, Configuration Files, Section 3.2, “dis_irm.conf” , Section 3.1, “dis_ix.conf”.
Generally, to change (increase) default limits, the configuration files needs to be changed on each Cluster Node. Typically, you should edit and test the changes on one Cluster Node, and then copy the file over to all other Cluster Nodes. To make changes in the configuration file effective, you need to restart the full eXpressWare driver stack on all Cluster Nodes:
# dis_services restart
SCICreateSegment() will fail if the system can't allocate a large enough physical contiguous memory. The ability to do this will be reduced over time as the physical memory often will be fragmented over time.
To overcome the problem caused by memory fragmentation, the eXpressWare software supports "Memory preallocation" to allow the driver to allocate the required memory during the initial boot and driver load. More details about memory preallocation can be found in Appendix C, Configuration Files, Section 3.1, “dis_ix.conf”
SCIConnectRemteSegment() will fail if the local mapping resources (prefetch memory size) are exhausted.
All use of remote memory will consume addresses from the Prefetch Memory range, you need to ensure that the card Prefetch Memory Size is configured to be larger than the sum of all concurrent use. Some parts of the address space may be lost due to memory alignment and the eXpressWare internals is also consuming some resources. As a general recommendation is to double the size vs your maximum requirements.
Information on how to increase the Prefetch Space Size can be found at Section 2, “PCIe Prefetch Memory size”
The default setup is to support two reflective memory multicast groups/segments, each 2 Megabytes. By following this guide, you can increase the maximum size and/or number of reflective memory groups (segments). You typically need to change both the size of the preallocated multicast memory and the size of the PCIe Prefetch Space.
The maximum reflective memory size and number of multicast groups that can be used, depends on your PC, Operating system and type of NTB adapter card. Please consult the appropriate software release note for additional details.
If you make changes to the multicast memory size, these settings must be applied to all systems connected to the PCIe fabric.
If you are trying to use a larger multicast memory segments than currently supported, the typical symptom is that SCICreateSegment() will fail (the local multicast memory resources is exhausted). The max size is controlled by the ntb_mcast_max_grop_size set in the dis_ix.conf configuration file.
You will find more details on changing this configuration file Section 3.1, “dis_ix.conf”, Section 3.1.3, “Configuring multicast memory”
Accessing the multicast memory space buffer requires resources from the shared PCIe Prefetch Space pool. If you are exhausting this resource, SCIConnectRemteSegment() will fail.
All use of remote memory will consume space from the Prefetch Memory pool, you need to ensure the card Prefetch Memory Size is configured to be larger than the sum of all concurrent use. Some parts of the address space may be lost due to memory alignment and the eXpressWare internals is also consuming some resources. As a general recommendation is to double the size vs your maximum requirements.
Information on how to increase the Prefetch Space Size can be found at Section 2, “PCIe Prefetch Memory size”
You need to be careful when updating RPMs on the Cluster Nodes with a modified dis_irm.conf , , , dis_ix.conf, , dis_ssocks.conf. If you directly use RPMs to update the existing eXpressWare RPMs like using rpm -U, the existing and modified configuration file(s)will be moved to e.g. dis_irm.conf.rpmsave, and the default dis_irm.conf will replace previously modified version.
If you update your system with the SIA as described in Chapter 5, Update Installation, SIA will take care that the existing dis_irm.conf and , dis_ix.conf , dis_ssocks.conf. will be preserved and stay effective.