For Linux based systems, the recommended installation method is to use the automatic eXpressWare installer (SIA) to install all required drivers and software on all cluster nodes. You typically start the installation on one system and the installer will ask a number of questions to gather information about the Cluster Nodes, the Network Management Node and the underlying PCIe network and its configuration. If you answer all questions correctly and all requirements are met, the SIA will install the software on all nodes. The alternative method is to do a manual installation.
The installation of eXpressWare on a system that does not satisfy this requirement is described in Chapter 6, Manual Installation,Section 2, “Installation of a Heterogeneous Cluster”
For the automatic SIA-based eXpressWare installation of all Cluster Nodes and the Cluster Management Node, the following requirements have to be met:
Homogeneous Cluster Nodes: All Cluster Nodes of the cluster are of the same CPU architecture and run the same kernel version. The Cluster Management Node may be of different CPU architecture and kernel version!
RPM support: The Linux distribution on the Cluster Nodes, the Cluster Management Node and the installation machine needs to support RPM or DEB packages. Both major distributions from Red Hat and (SUSE) use RPM packages.
On platforms that do not support RPM packages, it is also possible to install the Dolphin PCI Express software. Please see Chapter 6, Manual Installation,Section 4, “Unpackaged Installation” for instructions.
Installed RPM packages: To build the Dolphin PCI Express software stack, a few RPM packages that are often not installed by default are required:
qt and qt-devel (> version 3.0.5), glibc-devel and libgcc (32- and 64-bit, depending on what binary formats should be supported), rpm-build, and the kernel header files and configuration (typically a kernel-devel or kernel-source RPM that exactly(!) matches the version of the installed kernel) |
The SIA will check for these packages, report what packages might be missing and will offer to install them if the yum RPM management system is supported on the affected machine. All required RPM packages are within the standard set of RPM packages offered for your Linux distribution, but may not be installed by default.
If the qt-RPMs are not available, the Dolphin PCI Express software stack can be built nevertheless, but the GUI applications to configure and manage the cluster will not be available. Please see below (Section 1.2.2, “Non-GUI Installation”) on how to install the software stack in this case.
GUI support: for the initial installation, the installation machine should be able to run GUI application via X.
If the required configuration files are already available prior to the installation, a GUI is not required (see section Section 1.2.2, “Non-GUI Installation”).
Disk space:
To build the eXpressWare RPM packages, about 500MB free disk space in the system's temporary directory (typically /tmp on Linux) are required on the kernel build machine and the Cluster Management Node.
To install eXpressWare on a Cluster Node; about 100MB free disk space is required for the installation directory (default /opt/DIS ).
Resolvable host names:
The nodes must have resolvable names associated with a network interface. Please note that the hostnames must not resolve to the loopback address for SuperSockets to work properly.
You can use ping to test the name resolution:
[root@XD1-CentOS7 ~]# ping -c 1 XD1-CentOS7 PING XD1-CentOS7 (192.168.1.30) 56(84) bytes of data. 64 bytes from XD1-CentOS7 (192.168.1.30): icmp_seq=1 ttl=64 time=0.035 ms --- XD1-CentOS7 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms
Password-less ssh-access
To use the eXpressWare installer (SIA) to install the software on all nodes, you need to enable password-less ssh-access to all systems from the Cluster Management Node (frontend). ' If you don't want to do this, you must manually install the software on each node using the --install-node option. The eXpressWare installer will ask if password-less ssh-access is enabled and offer to set it up during the installation process.
Firewall configuration
For centralized management of the PCIe network, the eXpressWare Network Manager must be able to communicate over Ethernet to the Node Manager running on each Cluster Node.
To allow this communication, you must ensure the Firewall on each Cluster Node allows incoming communication on TCP port 3443.
If you plan to utilize SuperSockets between Cluster Nodes, you need to disable the Firewall.
If you can't open the Firewall on the Cluster Nodes, you need to do a manual installation of eXpressWare on each node and distribute the configuration files yourself.
As an example, if you would like to disable the Firewall using RHEL/CentOS 7:
[root@XD1-CentOS7 ~]# systemctl stop firewalld [root@XD1-CentOS7 ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination
In order to permanently disable the firewall service, run the following command on all nodes:
[root@XD1-CentOS7 ~]# systemctl disable firewalld Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
Disable SELinux: RHEL/Centos 7 enables the SELinux security extensions by default. They should be disabled since they cause problems with SuperSockets. To disable SELinux at runtime run the following command:
[root@XD1-CentOS7 ~]# setenforce 0
In order to permanently disable SELinux edit the file /etc/sysconfig/selinux and change the line:
SELINUX=enforcing
to
SELINUX=disabled
Reboot for the changes to take effect.
If the Cluster Management Node does not support running GUI applications, but another machine in the network does, it is possible to run the installation on this machine. The only requirement is ssh-access from the installation machine towards the Cluster Management Node and all Cluster Nodes. This installation mode can be chosen by executing the SIA on the installation machine and specifying the Cluster Management Node name when being asked for it.
In this scenario, the dis_netconfig will be compiled, installed and executed on the installation machine, and the generated configuration files will be transferred to the Cluster Management Node by the installer. Optionally you can follow the instructions in the next section.
PCI Express can be installed into a cluster which is currently under operation without stopping the cluster application from working. This requires that the application running on the cluster can cope with single Cluster Nodes going down. It is only necessary to turn off each Cluster Node once to install the adapter. The software installation can be performed under load, although minor performance impacts are possible. For a description of this installation type, please proceed as described in Chapter 6, Manual Installation,Section 1, “Installation under Load”
If no machine in the network does have the capability to run GUI applications, you can still use the SIA-based installation. The SIA-based installation does not require that you use the GUI applications, and will use the command line client, dis_mkconf to create dishosts.conf and networkmanager.conf.
eXpressWare includes two GUI tools:
dis_netconfig is a tool that is used to create the interconnect configuration file /etc/dis/dishosts.conf and the Network Manager configuration file /etc/dis/networkmanager.conf. It is needed once on the initial cluster installation, and each time Cluster Nodes are added or removed from the cluster.
dis_admin is used to monitor and control the cluster interconnect.
These tools are not mandatory for the installation and operation of the cluster, but are very helpful. They can be run on machines separate from the cluster: the configuration files created by dis_netconfig can be copied over to the Cluster Management Node manually, and dis_admin can connect to the Network Manager on the Cluster Management Node via TCP sockets.
If the Cluster Management Node does not support running GUI applications, but another machine in the network does, it is possible to run the installation on this machine. The only requirement is ssh-access from the installation machine towards the Cluster Management Node and all Cluster Nodes. This installation mode can be chosen by executing the SIA on the installation machine and specifying the Cluster Management Node name when being asked for it.
In this scenario, the dis_netconfig will be compiled, installed and executed on the installation machine, and the generated configuration files will be transferred to the Cluster Management Node by the installer.
If no machine in the network does have the capability to run GUI applications, you can still use the SIA-based installation. The SIA-based installation does not require that you use the GUI applications, and will use the command line client, dis_mkconf to create dishosts.conf and networkmanager.conf.
If you want to verify your settings in the Dolphin Network Configuration, it is possible to create the correct configuration files on another machine and store them in /etc/dis on the Cluster Management Node before executing the SIA on the Cluster Management Node (not on another machine). In this scenario, no GUI application is run at all during the installation. To create the configuration files on another machine, you can either run the SIA with the --install-editor option if it is a Linux machine, or install a binary version of the dis_netconfig if it is a Windows-based machine. Finally, you can send the necessary information to create the configuration files to Dolphin support which will then provide you with the matching configuration files and the cabling instructions. This information includes:
External hostnames (or IP addresses) of all Cluster Nodes
Adapter type and number of fabrics (1 or 2)
Hostnames (or IP adresses/subnet) which should be accelerated with SuperSockets (default is the list of hostnames provided above)
Planned interconnect topology (default is derived from number of Cluster Nodes and adapter type)
Description of how Cluster Nodes are physically located (to avoid cabling problems)
To configure and operate a PCI Express cluster you need to select one computer to be the Cluster Management Node (frontend). The Cluster Management Node will automatically monitor and configure the PCI Express hardware based on configuration files created on the Cluster Management Node. The Cluster Management Node can be one of the Cluster Nodes in the cluster or it can be a separate machine. The Cluster Management Node will communicate with the Cluster Nodes through Ethernet. The burden of the Cluster Management Node is very modest. The Cluster Management Node machine needs to have a graphical display to be able to utilize the Dolphin Network Configurator and the Dolphin Admin.
The Cluster Management Node does not need to run the same operating system as the cluster. Windows and Linux is currently supported for running the Cluster Management Node software. It is possible to both use Windows and Linux to manage a Linux cluster (or a Linux box to manage a Windows cluster).
Please note that there is no need to manually configure the adapter cards in the cluster as this will be done automatically by eXpressWare.
The result of the installation will be that all necessary drivers and services are installed on the Cluster Nodes, that the Cluster Management Node has the Network Manager running and that the whole PCI Express interconnect is configured and functional. By default, the installation target of all Dolphin software is /opt/DIS .
In detail, the installation results on each Cluster Node are the following services:
dis_kosif: This service loads the kosif module which manages OS dependent functionality.
dis_ix: This service loads the dis_ix module which manages the IDT PCI Express hardware.
dis_px: This service loads the dis_px module which manages the PLX PCI Express hardware.
dis_mx: This service loads the dis_mx module which manages the Microsemi PCI Express chip.
dis_irm: This service loads the dis_irm module which performs configuration, and session and resource management.
dis_sisci: This service loads the dis_sisci module which provides abstractions and mappings for the SISCI user-level API.
dis_supersockets: This service loads the dis_msq, dis_mbox and dis_ssocks modules which together provide the SuperSockets functionality.
dis_nodemgr: This service controls the node manager daemon.
dis_networkmgr (only on Cluster Management Node): This service controls the node manager daemon.
Please note that you should only have one of the dis_mx, dis_ix or dis_px drivers installed on one node.
Next to this, a number of administration tools, benchmark and test programs are installed into /opt/DIS/bin and /opt/DIS/sbin .
On the Cluster Management Node, the service dis_networkmgr will be installed which controls the Network Manager daemon. Monitoring and test tools are installed to /opt/DIS/sbin.