NVIDIA GPUs have many power-saving mechanisms. Some of them willreduce clocks and voltages to different parts of the chip, and insome cases turn off clocks or power to parts of the chip entirely,without affecting functionality or while continuing to function,just at a slower speed.
However, the lowest power states for NVIDIA GPUs require turningpower off to the entire chip, often through ACPI calls. Obviously,this impacts functionality. Nothing can run on the GPU while it ispowered off. Care has to be taken to only enter this state whenthere are no workloads running on the GPU and any attempts to startwork or any memory mapped I/O (MMIO) access must be preceded with asequence to first turn the GPU back on and restore any necessarystate.
The NVIDIA GPU may have one, two or four PCI functions:
Function 0: VGA controller / 3D controller
Function 1: Audio device
Function 2: USB xHCI Host controller
Function 3: USB Type-C UCSI controller
Out of the four PCI functions, the NVIDIA driver directlymanages the VGA controller / 3D Controller PCI function. Other PCIfunctions are managed by the device drivers provided with the Linuxkernel. The NVIDIA driver is capable of handling entry into andexit from these low power states, for the PCI function 0. Theremaining PCI functions are also powered down along with function 0when entering these low power states. As a result, the devicedrivers for the other three functions also need to be taken intoaccount to:
- prevent entering the lowest-power state when the device is inuse,
- trigger exiting the lowest-power state when the device isneeded,
- save and restore any hardware state around power-offevents.
The NVIDIA Linux driver includes initial experimental supportfor dynamically managing power to the NVIDIA GPU. It depends on theruntime power management framework within the Linux kernel toarbitrate power needs of various PCI functions. In order to havemaximum power saving from this feature, two conditions must bemet:
1. Runtime power management for all the PCI functions of the GPUshould be enabled.
2. The device drivers for all the PCI functions should supportruntime power management.
If these conditions are satisfied and if all the PCI functionsare idle, then The NVIDIA GPU will go to the lowest power stateresulting into maximum power savings.
This feature is available only when the following conditions aresatisfied:
This feature is supported only on notebooks.
This feature requires system hardware as well as ACPI support(ACPI "_PR0" and "_PR3" methods are needed to control PCIe power).The necessary hardware and ACPI support was first added in IntelCoffeelake chipset series. Hence, this feature is supported fromIntel Coffeelake chipset series.
This feature requires a Turing or newer GPU.
This feature is supported with Linux kernel versions 4.18 andnewer. With older kernel versions, it may not work as intended.
This feature is supported when Linux kernel defines CONFIG_PM(CONFIG_PM=y). Typically, if the system supports S3(suspend-to-RAM), then CONFIG_PM would be defined.
This feature is disabled by default by the NVIDIA driver. It canbe enabled via an option NVreg_DynamicPowerManagement.
Option"NVreg_DynamicPowerManagement=0x00"With this setting, the NVIDIA driver will only use the GPU'sbuilt-in power management so it always is powered on andfunctional. This is the default option, since this feature is a newand highly experimental feature. Actual power usage will vary withthe GPU's workload.
Option"NVreg_DynamicPowerManagement=0x01"With this setting, the NVIDIA GPU driver will allow the GPU togo into its lowest power state when no applications are runningthat use the nvidia driver stack. Whenever an application requiringNVIDIA GPU access is started, the GPU is put into an active state.When the application exits, the GPU is put into a low powerstate.
Option"NVreg_DynamicPowerManagement=0x02"With this setting, the NVIDIA GPU driver will allow the GPU togo into its lowest power state when no applications are runningthat use the nvidia driver stack. Whenever an application requiringNVIDIA GPU access is started, the GPU is put into an active state.When the application exits, the GPU is put into a low powerstate.
Additionally, the NVIDIA driver will actively monitor GPU usagewhile applications using the GPU are running. When the applicationshave not used the GPU for a short period, the driver will allow theGPU to be powered down. As soon as the application starts using theGPU, the GPU is reactivated.
It is important to note that the NVIDIA GPU will remain in anactive state if it is driving a display. In this case, the NVIDIAGPU will go to a low power state only when the X configurationoption HardDPMS isenabled and the display is turned off by some means - eitherautomatically due to an OS setting or manually using commands likexset.
Similarly, the NVIDIA GPU will remain in an active state if aCUDA application is running.
Option NVreg_DynamicPowerManagement can be set onthe command line while loading the NVIDIA Linux kernel module. Forexample,
modprobe nvidia"NVreg_DynamicPowerManagement=0x02"
This section describes automated ways to perform the manualsteps mentioned above so that this feature works seamlessly afterboot.
Create a file named
80-nvidia-pm.rulesin/lib/udev/rules.d/directory.Add the content given below to
80-nvidia-pm.rulesfile. This would enableruntime power management for the VGA Controller / 3D Controller PCIfunction. It would also remove Audio device PCI function, USB xHCIHost Controller function as well as USB Type-C UCSI Controller PCIfunction.# Remove NVIDIA USB xHCI Host Controller devices, if presentACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x0c0330", ATTR{remove}="1"# Remove NVIDIA USB Type-C UCSI devices, if presentACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x0c8000", ATTR{remove}="1"# Remove NVIDIA Audio devices, if presentACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x040300", ATTR{remove}="1"# Enable runtime PM for NVIDIA VGA/3D controller devices on driver bindACTION=="bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", TEST=="power/control", ATTR{power/control}="auto"ACTION=="bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030200", TEST=="power/control", ATTR{power/control}="auto"# Disable runtime PM for NVIDIA VGA/3D controller devices on driver unbindACTION=="unbind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", TEST=="power/control", ATTR{power/control}="on"ACTION=="unbind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030200", TEST=="power/control", ATTR{power/control}="on"The driver option
NVreg_DynamicPowerManagementcan be set via thedistribution's kernel module configuration files (such as thoseunder/etc/modprobe.d). For example,the following line can be added to/etc/modprobe.d/nvidia.conffile to seamlesslyenable this feature.options nvidia"NVreg_DynamicPowerManagement=0x02"Reboot the system.
