Z2 G9: unable to reach C-state above 2 with GPU installed in PCIe5 slot

[wrobelda]

I have a Z2 G9 system, with an i5 13600k installed. Slot 1 (PCIE5) has AMD 6600 XT GPU, slot 4 has a Mellanox 4 LX card.

The system is running Proxmox hypervisor, using  6.5.13-3-pve kernel.

All of the hardware installed in the system supports ASPM, as reported by the lspci:

root@proxmox:~# lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )'
00:01.0 PCI bridge: Intel Corporation Device a70d (rev 01) (prog-if 00 [Normal decode])
		LnkCap:	Port #2, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <16us
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:1b.0 PCI bridge: Intel Corporation Device 7ac4 (rev 11) (prog-if 00 [Normal decode])
		LnkCap:	Port #21, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:1c.0 PCI bridge: Intel Corporation Alder Lake-S PCH PCI Express Root Port #1 (rev 11) (prog-if 00 [Normal decode])
		LnkCap:	Port #1, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:1d.0 PCI bridge: Intel Corporation Alder Lake-S PCH PCI Express Root Port #13 (rev 11) (prog-if 00 [Normal decode])
		LnkCap:	Port #13, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
01:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] 10 XL Upstream Port of PCI Express Switch (rev c1) (prog-if 00 [Normal decode])
		LnkCap:	Port #0, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <64us
		LnkCtl:	ASPM L1 Enabled; Disabled- CommClk+
02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] 10 XL Downstream Port of PCI Express Switch (prog-if 00 [Normal decode])
		LnkCap:	Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
		LnkCtl:	ASPM L0s L1 Enabled; Disabled- CommClk+
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] 23 [Radeon RX 6600/6600 XT/6600M] (rev c1) (prog-if 00 [VGA controller])
		LnkCap:	Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
		LnkCtl:	ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] 21/23 HDMI/DP Audio Controller
		LnkCap:	Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
		LnkCtl:	ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
04:00.0 Non-Volatile memory controller: Shenzhen Longsys Electronics Co., Ltd. Device 1602 (rev 01) (prog-if 02 [NVM Express])
		LnkCap:	Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
05:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
05:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
05:01.2 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
05:01.3 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
05:01.4 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
05:01.5 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
05:01.6 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
05:01.7 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
05:02.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
05:02.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
06:00.0 Non-Volatile memory controller: Shenzhen Longsys Electronics Co., Ltd. Device 1602 (rev 01) (prog-if 02 [NVM Express])
		LnkCap:	Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+

Unfortunately, with both PCI cards installed, the system won't reach C state lower than 2, which results in excessive power consumption — when idling, I see 48-65 Watt, which is way more than this configuration should yield. 

For comparison, with both PCI cards removed, I can reach C8 easily (although I can't reach C10, which is another story), and the power consumption drops to mere 4.5-7W when idling! For the record, the GPU consumes 4 W when idling, and the Mellanox NIC 12W, so with both cards installed, I should be seeing roughly 25-30W consumption, not the 45-65.

This sounds like a bug somewhere along BIOS/Linux support, or a hardware issue, considering the spec sheets report that a similar system should idle at around 24W, which is what I would expect to see here as well, as mentioned above.