High CPU Usage - System/ACPI.sys - GPE.L6F Storm [Omen 15/17/Zbook 17 G5/Other HP Devices]

[MaPo31415]

I performed extensive tests that involved:

• Analysis of IRP's using WinDbg
• Analysis of ACPI using AMLI extension for WinDbg
• Analysis of Ubuntu Gnu/Linux behavior

The problem was very easy to recreate on Ubuntu - I just had to wait until a display blanks, indicating ACPI-triggered power-off based on the internal timer - that's when Interrupt storm starts. Upon inspection, I noticed thousands of IRQ9 spawned on CPU 1 (counting from 0).

I got back to my Windows environment, I waited a little bit for Interrupt storm to begin, then I analysed the log of the ACPI (!amli dl):

0:41:08.273 [ffff860255d930c0] RestartContext        Context=ffff8602512e8010
    \_GPE._L6F
    QTh=0 QCt=ffff8602512e8010 QFg=00000000 pbOp=0:[]

0:41:08.312 [ffff860241e87040] RunContext            Context=ffff8602512e8010
    \_GPE._L6F
    QTh=0 QCt=ffff8602512e8010 QFg=00000000

0:41:08.313 [ffff860241e87040] AsyncCallBack         rc=0 pEvent=a2800d
    \_GPE._L6F
    QTh=ffff860241e87040 QCt=ffff8602512e8010 QFg=00000000

0:41:08.313 [ffff860241e87040] FinishedContext       Context=ffff8602512e8010 rc=8004
    QTh=0 QCt=0 QFg=00000000

0:41:08.334 [ffff860255d930c0] RestartContext        Context=2
    \_GPE._L6F
    QTh=0 QCt=0 QFg=00000000 pbOp=0:[]

I got back on Ubuntu and blacklisted this particular IRQ echo disable > /sys/firmware/acpi/interrupts/gpe6F

Lo and behold, CPU usage went down immediately. I would settle for that workaroud if it wasn't for a fact, that I use Windows 10. 

All of these led me to the conclusion that all of these problems are caused by erroneous ACPI tables that need to be fixed by BIOS/Firmware Upgrade - There seem to be no proper handler for General Purpose Event 6F implemented in the firmware.

I stumbled upon this thread One CPU thread constantly at 90% in Linux (acpi gpe6F interrupt storm) - THE SAME ISSUE, THE SAME WORKAROUD, THE SAME LAPTOP AS MOST AFFECTED USERS (HP OMEN 17), AND MOST LIKELY THE SAME CAUSE

[MaPo31415]

Contents of __L6F Method (Q70 01.07.01 Rev.A) extracted from DSDT Table

Method (_L6F, 0, NotSerialized)  // _Lxx: Level-Triggered GPE, xx=0x00-0xFF
{
    If (CondRefOf (\_SB.PCI0.RP01.GPEH))
    {
        \_SB.PCI0.RP01.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.RP02.GPEH))
    {
        \_SB.PCI0.RP02.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.RP03.GPEH))
    {
        \_SB.PCI0.RP03.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.RP04.GPEH))
    {
        \_SB.PCI0.RP04.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.RP05.GPEH))
    {
        \_SB.PCI0.RP05.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.RP06.GPEH))
    {
        \_SB.PCI0.RP06.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.RP07.GPEH))
    {
        \_SB.PCI0.RP07.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.RP08.GPEH))
    {
        \_SB.PCI0.RP08.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.RP09.GPEH))
    {
        \_SB.PCI0.RP09.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.RP10.GPEH))
    {
        \_SB.PCI0.RP10.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.RP11.GPEH))
    {
        \_SB.PCI0.RP11.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.RP12.GPEH))
    {
        \_SB.PCI0.RP12.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.RP13.GPEH))
    {
        \_SB.PCI0.RP13.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.RP14.GPEH))
    {
        \_SB.PCI0.RP14.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.RP15.GPEH))
    {
        \_SB.PCI0.RP15.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.RP16.GPEH))
    {
        \_SB.PCI0.RP16.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.RP17.GPEH))
    {
        \_SB.PCI0.RP17.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.RP18.GPEH))
    {
        \_SB.PCI0.RP18.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.RP19.GPEH))
    {
        \_SB.PCI0.RP19.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.RP20.GPEH))
    {
        \_SB.PCI0.RP20.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.SAT0.PRT0.GPEH))
    {
        \_SB.PCI0.SAT0.PRT0.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.SAT0.PRT1.GPEH))
    {
        \_SB.PCI0.SAT0.PRT1.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.SAT0.PRT2.GPEH))
    {
        \_SB.PCI0.SAT0.PRT2.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.SAT0.PRT3.GPEH))
    {
        \_SB.PCI0.SAT0.PRT3.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.SAT0.PRT4.GPEH))
    {
        \_SB.PCI0.SAT0.PRT4.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.SAT0.PRT5.GPEH))
    {
        \_SB.PCI0.SAT0.PRT5.GPEH ()
    }

    If (CondRefOf (\_SB.PCI0.PEG0.PEGP.HGGE))
    {
        \_SB.PCI0.PEG0.PEGP.HGGE ()
    }

    If (CondRefOf (\_SB.CFHP))
    {
        \_SB.CFHP ()
    }
}

[MaPo31415]

I found the root cause of the Issue - Firmware does not support ASPM, whereas Windows, despite being aware of that, does not prevent routines from being spawned.

Workaroud (tested on both OSs):

• Windows
  Disable ASPM - make sure to disable it on both battery and AC, and reboot the system - this is to make sure that routines are not enabled in the NT kernel - this setting does not apply immediately, you won't see a sudden CPU usage drop right away (thus I believe that many dismissed this as a valid workaroud).
  
• GNU/Linux
  Boot your operating system with pcie_aspm=off in GRUB

Solution (will mark this as solved once HP delivers):

• Proper implementation of ASPM in the Firmware (Q70 and the one HP Omen laptops use)

Unfortunately, that's not it.

[Jay889]

I'm so impressed with all your troubleshooting.

Good job man! Hp should offer you a job 😛

Regarding the ASPM, I did turn it off before creating this thread. I thought it was fixed but sadly I  didnt notice any cpu drop.

I did not try it on grub. I will now 

Thank you for your help and support.

Best of luck

[MaPo31415]

Hi Jay889,

Let me know if that helps (FYI: upon booting Ubuntu LiveCD, just press "e" on the first GRUB screen, and enter "pcie_aspm=off" before "quiet" statement, and press Ctrl+x. This way you don't have to install any OS nor modify any files.

If that doesn't help, I'll help you further - I have a few ideas left.

[MaPo31415]

Some problems I found in the ACPI DSDT:

Linux Decompiler:

Firmware Error (ACPI): Could not resolve [^PCI0.LPCB.EC0.ECMX], AE_NOT_FOUND (20181213/dswload-388)
Firmware Error (ACPI): Could not resolve [^PCI0.LPCB.EC0.ECRG], AE_NOT_FOUND (20181213/dswload-388)
Firmware Error (ACPI): Could not resolve [^PCI0.LPCB.EC0.PSCF], AE_NOT_FOUND (20181213/dswload-388)
Firmware Error (ACPI): Could not resolve [^PCI0.LPCB.EC0.BCSC], AE_NOT_FOUND (20181213/dswload-388)
Firmware Error (ACPI): Could not resolve [^^PEG0.PEGP], AE_NOT_FOUND (20181213/dswload-388)
Firmware Error (ACPI): Could not resolve [^^^IETM.DFFG], AE_NOT_FOUND (20181213/dswload-388)
Firmware Error (ACPI): Could not resolve [^PCI0.LPCB.EC0.ECMX], AE_NOT_FOUND (20181213/dswload2-369)
Firmware Error (ACPI): Could not resolve [^PCI0.LPCB.EC0.ECRG], AE_NOT_FOUND (20181213/dswload2-369)
Firmware Error (ACPI): Could not resolve [^PCI0.LPCB.EC0.PSCF], AE_NOT_FOUND (20181213/dswload2-369)
Firmware Error (ACPI): Could not resolve [^PCI0.LPCB.EC0.BCSC], AE_NOT_FOUND (20181213/dswload2-369)
Firmware Error (ACPI): Could not resolve [^^PEG0.PEGP], AE_NOT_FOUND (20181213/dswload2-369)
Firmware Error (ACPI): Could not resolve [^^^IETM.DFFG], AE_NOT_FOUND (20181213/dswload2-369)

Latest Windows Decompiler:

Firmware Error (ACPI): Could not resolve symbol [^PCI0.LPCB.EC0.ECMX], AE_NOT_FOUND (20190703/dswload-496)
Firmware Error (ACPI): Could not resolve symbol [^PCI0.LPCB.EC0.ECRG], AE_NOT_FOUND (20190703/dswload-496)
Firmware Error (ACPI): Could not resolve symbol [^PCI0.LPCB.EC0.PSCF], AE_NOT_FOUND (20190703/dswload-496)
Firmware Error (ACPI): Could not resolve symbol [^PCI0.LPCB.EC0.BCSC], AE_NOT_FOUND (20190703/dswload-496)
Firmware Error (ACPI): Could not resolve symbol [^^PEG0.PEGP], AE_NOT_FOUND (20190703/dswload-496)
Firmware Error (ACPI): Could not resolve symbol [^^^IETM.DFFG], AE_NOT_FOUND (20190703/dswload-496)
Firmware Error (ACPI): Could not resolve symbol [^PCI0.LPCB.EC0.ECMX], AE_NOT_FOUND (20190703/dswload2-477)
Firmware Error (ACPI): Could not resolve symbol [^PCI0.LPCB.EC0.ECRG], AE_NOT_FOUND (20190703/dswload2-477)
Firmware Error (ACPI): Could not resolve symbol [^PCI0.LPCB.EC0.PSCF], AE_NOT_FOUND (20190703/dswload2-477)
Firmware Error (ACPI): Could not resolve symbol [^PCI0.LPCB.EC0.BCSC], AE_NOT_FOUND (20190703/dswload2-477)
Firmware Error (ACPI): Could not resolve symbol [^^PEG0.PEGP], AE_NOT_FOUND (20190703/dswload2-477)
Firmware Error (ACPI): Could not resolve symbol [^^^IETM.DFFG], AE_NOT_FOUND (20190703/dswload2-477)

While method _L6F seems to depend on [^^PEG0.PEGP]

If (CondRefOf (\_SB.PCI0.PEG0.PEGP.HGGE))
    {
        \_SB.PCI0.PEG0.PEGP.HGGE ()
    }

Which oddly corresponds to my graphics card (I just updated drivers and disabled power-saving in the NVidia panel to check if that changes anything)

I'm waiting for HP Expert's input

[MaPo31415]

I captured events that were sent to the Kernel for evaluation from ACPI, before and during the issue:

logman start HPbug -ets -p Microsoft-Windows-Kernel-Acpi -o acpi201907210813.etl

Other than standard methods, methods below just kept repeating, increasing their assigned frequency (ACPI-wise)

\_SB.PCI0.LPCB.EC0.SEN1._TMP (reached frequency 73)

\_SB.PCI0.LPCB.EC0.SEN2._TMP (reached frequency 73)

\_SB.PCI0.B0D4._TMP (reached frequency 146)

Then it there was a single occurrence of:

\_SB.BAT0._BST

\_SB.AC._PSR

And then then it was just a \_GPE._L6F storm, which over minute reached frequency 34773 (each eval adds +1)

[Jay889]

If you remove the power from the laptop the cpu will drop down for 1 second and then go back to high usage on idle.

if you put the computer on sleep mode and wake it up the cpu will go back to normal for 5-10 mins.

On WINDOWS i can confirm that there's NO WORKING SOLUTION FOR NOW.

[MaPo31415]

---

@Jay889 wrote:

If you remove the power from the laptop the cpu will drop down for 1 second and then go back to high usage on idle.

if you put the computer on sleep mode and wake it up the cpu will go back to normal for 5-10 mins.

---

Exactly, this proves that our problem is the same, despite different laptop brands.

Could you run "logman start HPbug -ets -p Microsoft-Windows-Kernel-Acpi -o ACPIlog.etl" and then stop it with "logman stop HPbug –ets" once the storm starts? This way we'll be able to pinpoint faulty ACPI method on HP Omen. If it is also L6F, it would make things a lot simpler.

Your output file will be located in C:\Windows\System32\ACPIlog.etl, you can open it with Windows Event Viewer, the output should be similiar to the one below (look for warnings related to any \_GPE._ method with high frequency).

[Jay889]

I was testing the issue on linux. The high cpu usage started after 20 mins of idle, screen was always on.

I did change the grub settings as you mentioned earlier and i switched to capture the logs on windows.

I will test later on to see if the solution worked on linux and i will post back here.