[RESPONDED] AMD FW13 -- s2idle premature wake

sheern · December 8, 2023, 4:03am

Hi all,

I received my laptop earlier this week in Batch 8 and I’ve enjoyed getting everything set up and running with Fedora 39.

I’ve had an issue with putting the laptop to sleep where the display will turn off and then immediately turn back on as if interrupted by something. The display appears to stay on with the lid closed and as a result the battery will be severely depleted if I leave it for a couple of hours.

I haven’t done much scientific testing of this issue, but I have run the amd_s2idle.py script and the same premature wake up happens with the following output. I can provide the full log file if that is of use.

I’m running with these additional kernel parameters based on what I’ve read so far: rtc_cmos.use_acpi_alarm=1 "acpi_osi=!Windows 2020".

Thanks for your help in advance!

Debugging script for s2idle on AMD systems
💻 Framework Laptop 13 (AMD Ryzen 7040Series) (Laptop) running BIOS 3.3 (03.03) released 10/17/2023 and EC unknown
🐧 Fedora Linux 39 (KDE Plasma)
🐧 Kernel 6.6.4-200.fc39.x86_64
🔋 Battery BAT1 ( ) is operating at 99.02% of design
Checking prerequisites for s2idle
✅ Logs are provided via systemd
✅ AMD Ryzen 5 7640U w/ Radeon 760M Graphics (family 19 model 74)
✅ LPS0 _DSM enabled
✅ ACPI FADT supports Low-power S0 idle
✅ HSMP driver `amd_hsmp` not detected (blocked: False)
✅ PMC driver `amd_pmc` loaded (Program 0 Firmware 76.70.0)
✅ USB4 driver `thunderbolt` loaded
✅ GPU driver `amdgpu` available
✅ System is configured for s2idle
✅ NVME Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 (SSD 970 EVO/PRO) is configured for s2idle in BIOS
✅ GPIO driver `pinctrl_amd` available
How long should suspend cycles last in seconds (default 10)? 
How long to wait in between suspend cycles in seconds (default 4)? 
How many suspend cycles to run (default 1)? 
Suspending system in 0:00:02
Started at 2023-12-07 21:44:13.546985 (cycle finish expected @ 2023-12-07 21:44:27.547002)
Results from last s2idle cycle
○ Suspend count: 1
○ GPIOs active: ['5', '5', '5', '5', '5', '5', '5']
○ Wakeups triggered from IRQs: [9, 1]
○ Woke up from IRQ 9 (IR-IO-APIC 9-fasteoi acpi)
○ gpe0B increased from 150 to 168
○ gpe1A increased from 580456 to 586844
❌ Userspace suspended for 0:00:04.522984 (< minimum expected 0:00:09)
❌ Did not reach hardware sleep state
🔋 Battery BAT1 ( ) is operating at 99.02% of design
Explanations for your system
🚦 Userspace wasn't asleep at least 0:00:10
        The system was programmed to sleep for 0:00:10, but woke up prematurely.
        This typically happens when the system was woken up from a non-timer based source.

        If you didn't intentionally wake it up, then there may be a kernel or firmware bug

Mario_Limonciello · December 8, 2023, 5:01am

Drop the osi modification and please rerrun and attach full log.

sheern · December 8, 2023, 6:04am

Trying an image as pasting the full text logs is giving me an error when I post

EDIT: this link should allow you to download the text file

Mario_Limonciello · December 8, 2023, 6:47am

Can up please try again with no modules connected?

sheern · December 8, 2023, 1:59pm

Sure thing:

Mario_Limonciello · December 8, 2023, 2:00pm

This is very odd. Is it a kernel regression perhaps? Or can you tell me what sort of things you’ve changed in Fedora?

sheern · December 8, 2023, 2:09pm

I’ll try again soon with kernel 6.5.something that I started with and report if that changes anything.

The only things that stick out to me as different than a typical install of Fedora is I’m running the KDE Spin and dual booting with Windows. I installed Fedora first (and for some reason 2 boot loader entries show up for Fedora).

sheern · December 8, 2023, 2:27pm

No difference in behavior with Kernel 6.5.6-300.fc39.x86_64

EDIT: also haven’t been using the Windows install at all really but just tested sleep on that for the first time and the display stays off with the power button blinking – which seems like desired behavior

Mario_Limonciello · December 8, 2023, 3:17pm

OK well it’s at least a relief it’s not a kernel regression. Are you using anything like TLP? Or any addon software in KDE that might be trying to access sensors?

If possible, can you double check a Fedora 39 USB live image? It should “just work” there so we can confirm it’s something done in software in your install causing it.

Mario_Limonciello · December 8, 2023, 4:21pm

I double checked on my side using the latest 6.6.y kernel (6.6.5 right now) but can’t reproduce there. I only have the network card connected.
Framework 13 6.6.5 s2idle report (on Ubuntu 22.04 + GNOME) (github.com)

sheern · December 8, 2023, 4:25pm

Not using TLP, I did install powertop and tried the auto-tune a couple of times. Nothing that sticks out in terms of add-on software, from what I remember just installed a few normal GUI apps like VS Code.

Running off a live USB stick of the default (GNOME) Fedora Workstation 39 has the same behavior with the following logs:

Debugging script for s2idle on AMD systems
💻 Framework Laptop 13 (AMD Ryzen 7040Series) (Laptop) running BIOS 3.3 (03.03) released 10/17/2023 and EC unknown
🐧 Fedora Linux 39 (Workstation Edition)
🐧 Kernel 6.5.6-300.fc39.x86_64
🔋 Battery BAT1 ( ) is operating at 100.70% of design
Checking prerequisites for s2idle
✅ Logs are provided via systemd
✅ AMD Ryzen 5 7640U w/ Radeon 760M Graphics (family 19 model 74)
✅ LPS0 _DSM enabled
✅ ACPI FADT supports Low-power S0 idle
✅ HSMP driver `amd_hsmp` not detected (blocked: False)
✅ PMC driver `amd_pmc` loaded (Program 0 Firmware 76.70.0)
✅ USB4 driver `thunderbolt` loaded
✅ GPU driver `amdgpu` available
✅ System is configured for s2idle
✅ NVME Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 (SSD 970 EVO) is configured for s2idle in BIOS
✅ GPIO driver `pinctrl_amd` available
How long should suspend cycles last in seconds (default 10)? 
How long to wait in between suspend cycles in seconds (default 4)? 
How many suspend cycles to run (default 1)? 
Started at 2023-12-08 11:05:07.926579 (cycle finish expected @ 2023-12-08 11:05:21.926596)
Results from last s2idle cycle
○ Suspend count: 2
○ Wakeups triggered from IRQs: [9, 1]
○ Woke up from IRQ 9 (IR-IO-APIC 9-fasteoi acpi)
○ gpe0B increased from 177 to 185
○ gpe1A increased from 365456 to 368150
❌ Userspace suspended for 0:00:02.950189 (< minimum expected 0:00:09)
❌ Did not reach hardware sleep state
🔋 Battery BAT1 ( ) is operating at 100.70% of design
🚦 RTC driver `rtc_cmos` configured to use ACPI alarm
Explanations for your system
🚦 Userspace wasn't asleep at least 0:00:10
	The system was programmed to sleep for 0:00:10, but woke up prematurely.
	This typically happens when the system was woken up from a non-timer based source.

	If you didn't intentionally wake it up, then there may be a kernel or firmware bug

🚦 rtc_cmos is not configured to use ACPI alarm
	Some problems can occur during wakeup cycles if the HPET RTC emulation is used to
	wake systems. This can manifest in unexpected wakeups or high power consumption.

For this run, I was on battery and I had the modules back in (2 usb-c in the back, 1 usb-c front left, 1 usb-a front right) with the live usb stick in the A port. Note that the rtc kernel param isn’t touched.

(small side note: the amd_s2idle.py script does not refer to the correct package to get iasl for dnf, I’ve manually installed it whenever I’ve run it)

sheern · December 8, 2023, 4:25pm

Sorry for the summary logs, the automatic flagger is now not allowing pastebin links.

I can try to match your setup with Ubuntu and see how the plays out.

Mario_Limonciello · December 8, 2023, 4:34pm

Have you changed any BIOS settings?

The big difference between our tests is that you have TONS of GPE 1A firing constantly:

○ gpe1A increased from 365456 to 368150

Comparing my system this GPE never triggers at runtime or suspend:

$ cat /sys/firmware/acpi/interrupts/gpe1A
       0  EN     enabled      unmasked

sheern · December 8, 2023, 4:41pm

Interesting, I have made a few BIOS changes so I can try restoring the defaults. The ones I’m remembering right now are UMA_GAME_OPTIMIZED as I was getting white flicker issues, and secure boot disabled to use Ventoy.

This is what I have for the same command on the live install I just ran from

$ cat /sys/firmware/acpi/interrupts/gpe1A
  502458     STS disabled     unmasked

sheern · December 8, 2023, 4:58pm

Unfortunately same behavior with optimal defaults in the BIOS and secure boot enabled

Mario_Limonciello · December 8, 2023, 5:02pm

Does it help if you just disable the 1A interrupt?

echo "disable" | sudo tee /sys/firmware/acpi/interrupts/gpe1A

sheern · December 8, 2023, 5:53pm

Testing in the live USB environment, since the interrupt was already disabled I instead tried ‘mask’. On log out/back in, this resulted in the script running to completion with the following summary:

Debugging script for s2idle on AMD systems
💻 Framework Laptop 13 (AMD Ryzen 7040Series) (Laptop) running BIOS 3.3 (03.03) released 10/17/2023 and EC unknown
🐧 Fedora Linux 39 (Workstation Edition)
🐧 Kernel 6.5.6-300.fc39.x86_64
🔋 Battery BAT1 ( ) is operating at 100.70% of design
Checking prerequisites for s2idle
✅ Logs are provided via systemd
✅ AMD Ryzen 5 7640U w/ Radeon 760M Graphics (family 19 model 74)
✅ LPS0 _DSM enabled
✅ ACPI FADT supports Low-power S0 idle
✅ HSMP driver `amd_hsmp` not detected (blocked: False)
✅ PMC driver `amd_pmc` loaded (Program 0 Firmware 76.70.0)
✅ USB4 driver `thunderbolt` loaded
✅ GPU driver `amdgpu` available
✅ System is configured for s2idle
✅ NVME Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 (SSD 970 EVO) is configured for s2idle in BIOS
✅ GPIO driver `pinctrl_amd` available
How long should suspend cycles last in seconds (default 10)? 
How long to wait in between suspend cycles in seconds (default 4)? 
How many suspend cycles to run (default 1)? 
Started at 2023-12-08 12:37:08.159791 (cycle finish expected @ 2023-12-08 12:37:22.159811)
Results from last s2idle cycle
○ Suspend count: 2
○ Hardware sleep cycle count: 1
○ GPIOs active: ['5', '5']
○ Wakeups triggered from IRQs: [9, 1]
○ Woke up from IRQ 9 (IR-IO-APIC 9-fasteoi acpi)
○ gpe0B increased from 219 to 227
✅ Userspace suspended for 0:00:12.511410
✅ In a hardware sleep state for 0:00:08.714446 (69.65%)
🔋 Battery BAT1 ( ) is operating at 100.70% of design
🚦 RTC driver `rtc_cmos` configured to use ACPI alarm
Explanations for your system
🚦 rtc_cmos is not configured to use ACPI alarm
	Some problems can occur during wakeup cycles if the HPET RTC emulation is used to
	wake systems. This can manifest in unexpected wakeups or high power consumption.
 
For more information on this failure see:
	https://github.com/systemd/systemd/issues/24279

After that I tried a normal GNOME suspend 2 different ways

typing out suspend and pressing enter in GNOME search – this resulted in an immediate wake up like before
clicking the button in the system tray for suspend – the laptop stayed asleep with the display off, but if I slowly close the lid I see it turn on at the very end

Quite confusing behavior. I’m figuring things out as I go, so I was curious what gpe1A is referring to? A specific device?

Mario_Limonciello · December 8, 2023, 6:11pm

It’s a level triggered general purpose event from the BIOS.
Decoding the ACPI table this is what it does:

    Scope (\_GPE)
    {
        Method (_L1A, 0, NotSerialized)  // _Lxx: Level-Triggered GPE, xx=0x00-0xFF
        {
            M460 ("  OEM-ASL-\\_GPE._L1A\n", Zero, Zero, Zero, Zero, Zero, Zero)
            If (CondRefOf (\_SB.PCI0.GP19))
            {
                M460 ("    Notify (\\_SB.PCI0.GP19, 0x2)\n", Zero, Zero, Zero, Zero, Zero, Zero)
                Notify (\_SB.PCI0.GP19, 0x02) // Device Wake
            }

            If (CondRefOf (\_SB.PCI0.GP19.NHI0))
            {
                M460 ("    Notify (\\_SB.PCI0.GP19.NHI0, 0x2)\n", Zero, Zero, Zero, Zero, Zero, Zero)
                Notify (\_SB.PCI0.GP19.NHI0, 0x02) // Device Wake
            }

            If (CondRefOf (\_SB.PCI0.GP19.NHI1))
            {
                M460 ("    Notify (\\_SB.PCI0.GP19.NHI1, 0x2)\n", Zero, Zero, Zero, Zero, Zero, Zero)
                Notify (\_SB.PCI0.GP19.NHI1, 0x02) // Device Wake
            }

            If (CondRefOf (\_SB.PCI0.GP19.XHC2))
            {
                M460 ("    Notify (\\_SB.PCI0.GP19.XHC2, 0x2)\n", Zero, Zero, Zero, Zero, Zero, Zero)
                Notify (\_SB.PCI0.GP19.XHC2, 0x02) // Device Wake
            }

            If (CondRefOf (\_SB.PCI0.GP19.XHC3))
            {
                M460 ("    Notify (\\_SB.PCI0.GP19.XHC3, 0x2)\n", Zero, Zero, Zero, Zero, Zero, Zero)
                Notify (\_SB.PCI0.GP19.XHC3, 0x02) // Device Wake
            }

            If (CondRefOf (\_SB.PCI0.GP19.XHC4))
            {
                M460 ("    Notify (\\_SB.PCI0.GP19.XHC4, 0x2)\n", Zero, Zero, Zero, Zero, Zero, Zero)
                Notify (\_SB.PCI0.GP19.XHC4, 0x02) // Device Wake
            }

            If (CondRefOf (\_SB.PCI0.GP11))
            {
                M460 ("    Notify (\\_SB.PCI0.GP11, 0x2)\n", Zero, Zero, Zero, Zero, Zero, Zero)
                Notify (\_SB.PCI0.GP11, 0x02) // Device Wake
            }

            If (CondRefOf (\_SB.PCI0.GP11.SWUS))
            {
                M460 ("    Notify (\\_SB.PCI0.GP11.SWUS, 0x2)\n", Zero, Zero, Zero, Zero, Zero, Zero)
                Notify (\_SB.PCI0.GP11.SWUS, 0x02) // Device Wake
            }

            If (CondRefOf (\_SB.PCI0.GP12))
            {
                M460 ("    Notify (\\_SB.PCI0.GP12, 0x2)\n", Zero, Zero, Zero, Zero, Zero, Zero)
                Notify (\_SB.PCI0.GP12, 0x02) // Device Wake
            }

            If (CondRefOf (\_SB.PCI0.GP12.SWUS))
            {
                M460 ("    Notify (\\_SB.PCI0.GP12.SWUS, 0x2)\n", Zero, Zero, Zero, Zero, Zero, Zero)
                Notify (\_SB.PCI0.GP12.SWUS, 0x02) // Device Wake
            }
        }
    }

You can see basically when the event happens it notifies a ton of stuff to wake up. I don’t know what causes that GPE to trigger. That would be something I think Framework might need to comment on as it’s their BIOS.

However as we have the same BIOS binary on both our systems and reset BIOS defaults doesn’t change things what else can be different? Does it only happen with certain power adapters maybe?

I looked again comparing ours and an interesting point (as you said) is it’s disabled by default for you but enabled by default for me.

sheern · December 8, 2023, 9:00pm

Gotcha, I will plan on making a support ticket to notify Framework of my issue then maybe there’ll be more illumination as to why this interrupt triggers so much for my machine.

On the power adapter note, I am using a third-party one (Anker) – my untrained eye tells me it would be strange for that to make a difference when I perform the tests on battery but of course anything is possible.

Thanks for your time so far with this!

Matt_Hartley · December 8, 2023, 9:40pm

Thanks as always @Mario_Limonciello , we appreciate the help!

@sheern when creating the ticket, please do include this thread - will save us a lot of time by avoiding any replication.