2nd generation AMD EPYC processors (“Rome”) implement advanced Dynamic Power Management (DPM) in the I/O logic to optimize the balance between power consumption and performance. Certain I/O workloads may benefit from disabling this power management. This utility disables DPM for all PCI-e root complexes in the system and locks the logic into the highest performance operational mode.
Disabling I/O DPM will reduce the latency and/or improve the throughput of low-bandwidth messages for PCI-e Infiniband NICs and GPUs. Other workloads with low-bandwidth bursty PCI-e I/O characteristics may benefit as well if multiple such PCI-e devices are installed in the system. For a single GPU or PCI-e I/O device in the system, the UEFI Enhanced Preferred I/O setting should be utilized. Contact your system vendor for details on how to use this setting in the system setup.
For any PCI-e root complex that is unused (and therefore idle) or in the lower power state of Dynamic Power Management, this utility will increase the power consumption for that logic block by 350-500 mW. On a completely idle 2P server, this may result in an increase in idle power of 3-4 watts. If this utility is used on a server where it is not needed, this increased idle power consumption may reduce the headroom for a higher core performance boost. Therefore, the use of this utility is only recommended for very specific use cases as described above.
The actions of the utility do not persist across reboots and no UEFI variables are modified. There is no need to change any existing UEFI firmware settings when using this utility. For customers using Preferred I/O or Enhanced Preferred I/O, this setting should remain unchanged.
The recommended method to use the utility is either to create a system start-up script (e.g. one-shot System D service unit) or run the utility when starting up a job scheduler on the system. The installer packages in the download section below will create and enable a System D service unit for you. The service unit is configured to run in one-shot mode. This means that even when the service unit runs as expected, the status of the service unit will show inactive. This is the expected behavior when the utility runs normally. If the service unit shows failed, the utility did not run as expected. The output in either case can be shown with the systemctl status command. Note stopping the service unit has no effect since the utility does not leave anything running. To undo the effects of the utility, disable the service unit with the systemctl disable command and reboot the system.
To measure the impact of the utility you must run a before/after performance test such as the ROCm bandwidth test. If before/after tests do not show a performance improvement, you may have a use case where this utility is not expected to provide any meaningful performance gain.
The utility does not have any command-line options, and it must be run with super-user permissions.
To verify the performance benefits from the utility, you may run the AMD ROCm bandwidth tests on your GPUs, or you may run standard Infiniband latency tests on your NICs.