“In a typical PCIe-based system, the PCIe bus is enumerated and resources are allocated to each PCIe endpoint device during system initialization. Due to the limitations of enumeration and resource allocation algorithms, the PCIe topology is repaired once it is initialized, which means that new endpoint devices or switches must not be connected to the system.
In a typical PCIe-based system, the PCIe bus is enumerated and resources are allocated to each PCIe endpoint device during system initialization. Due to the limitations of enumeration and resource allocation algorithms, the PCIe topology is repaired once it is initialized, which means that new endpoint devices or switches must not be connected to the system.
The IDT PCIe hot-swappable driver is a loadable Linux module that uses IDT’s proprietary enumeration, resource allocation and device detection algorithms to allow PCIe endpoint devices and switches to connect to or disconnect from the system during operation, and Will not affect the operating status of the system. Other PCIe devices in the system. Taking Figure 1 as a reference, if the dual GigE controller is disconnected from the system and then reconnected to the ports on other switches, the operating status of the Fibre Channel controller and SATA/SAS controller will remain unchanged. This is different from the Windows Vista implementation of PCIe hot-plugging, which must be disabled, re-enumerate and reallocate resources to all devices that share the same root port, and then the newly connected device can operate. By writing files located in the /sysfs directory, you can simply connect or disconnect the device to or from the system in any way or more comfortably. Writing 0 to the /sysfs/bus/pci/slots/0000:bb:dd.f/power file (where bb specifies the bus number, dd specifies the device number, and f specifies the function number of the downstream port) will disable the specified downstream port, and at the same time Writing 1 to the same file will enable downstream ports. When the downstream port is disabled, all devices connected to the port will be deleted from the Linux kernel, and the PCI driver remove() function will be called to perform the required cleanup operation. Similarly, after enabling the downstream port, all devices connected to it will be discovered, and the PCI driver probe() function added to the Linux kernel will be called to perform device initialization. Downstream ports can also be enabled or disabled by third-party kernel-mode drivers using the hot-plug driver API described in the title “Enable and Disable Downstream Ports”.
Figure 2 below depicts an example of the types of devices that can be connected or disconnected from a system running the IDT PCIe hot-plug driver.
The key to realizing this hot-swappable function is to be able to allocate other unused bus numbers and memory resources for each downstream port in the system. When a new device is connected to the system, these additional resources will be used to bring the device into operation without compromising the operation state of other devices sharing the same root port.
System interconnection considerations
The IDT system interconnect topology shown in Figure 3 consists of an x86-based Root Joint Processor (RP), which is connected to one or more intelligent x86-based endpoint processors through the IDT system interconnect PCIe switch (EP). Each EP uses switches in the IDT domain to connect them to the downstream ports of the system interconnect PCIe switch. This topology is used in conjunction with the IDT system interconnection software to enable high-speed data transmission between any two peers in the system.
When using this type of topology to connect a new EP to or disconnect from the system, the hot-plug driver will treat the EP as the same as any ordinary endpoint device. However, when the RP is connected to or disconnected from the system, the way the RP is handled is different.
Generally, when the upstream port of a PCIe switch is disconnected, as shown in Figure 4, the switch will perform a warm reset on all its downstream ports. This feature is not desirable in the system interconnection topology, because when the RP is disconnected from the system, it will prevent EPs from communicating with each other. The hot plug driver solves this problem by detecting the IDT PCIe system interconnect switch and disabling the ability to generate a hot reset when its upstream port is disconnected. This function can be disabled in the hot plug driver by setting the linkDownHotReset module parameter to a non-zero value.
The hot-swappable driver also has a passive resource allocation algorithm. This algorithm is mainly used with the IDT system interconnection topology, which allows the RP to connect it to the operating system with minimal system interruption. Generally, when an endpoint device is connected to the root complex, it is the root complex that allocates memory resources to the device. For passive resource allocation, the root complex allocates memory resources to devices. In the case of passive resource allocation, the “root complex” of which memory resources should be used. By default, this feature is enabled in the hot swap driver, but it can be disabled by setting the passiveAllocationDisable module parameter.