[Solved]: instant wakeup from hibernate with linux 6.10 when bluetooth is turned on/when XHC0 listens to wakeup events

:computer: System information

  • OS: Arch Linux
  • Kernel version: 6.10.7-arch1-1 (though I’ve also tested with the LTS kernel (6.6.48) and linux-mainline (6.11rc5), see below)
  • Model: AMD 13th gen (AMD Ryzen 7 7840U)
  • BIOS/firmware version: Gnome settings and dmidecode -t bios report version 3.05

:bug: Problem description

I’ve been having trouble getting my laptop to hibernate consistently with the most recent stable kernel. The LTS kernel works fine, but the linux-mainline kernel (6.11 rc5, currently) seems to exhibit the same behavior as 6.10.

Kernel 6.10.x (faulty kernel)

In short, I can get hibernation to work (i.e. systemctl hibernate), but only under certain conditions. From what I can gather from dmesg, the hibernation process gets (nearly) 100% of the way there, but is then interupted by some kind of wakeup event/kernel misbehavior. At that point, the system will just wake up again. Suspending (i.e. systemctl suspend) works absolutely fine, however.

So far, I’ve been able to narrow down this immediate wakeup after hibernation to two possibly related causes. Specially, hibernation works completely fine, unless…

  1. Bluetooth is turned on (i.e. I run rfkill unblock bluetooth)
  2. The kernel listens to wakeup triggers from the xhc0 device, which lspci tells me corresponds to USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b9.

To be more specific, hibernation works 100% of the time just in case bluetooth is turned off, or, alternatively, if the xhc0 device is told to ignore wakeup triggers. In other words, if bluetooth is on, and xhc0 is told to ignore wakeup events, hibernation will still work. Hibernation will also always work if bluetooth is turned off.

With respect to bluetooth, I should add that I can reproduce the problem even when no bluetooth devices are connected. Hence, it’s not as if some rogue bluetooth device is secretly sending inputs, so preventing the system from shutting down completely.

The “USB controller”/xhc0 device in question can be told to ignore wakeup triggers by running echo "XHC0" | sudo tee /proc/acpi/wakeup (actually, this acts as a toggle; to check the current value, simply run cat /proc/acpi/wakeup).

From what I can see, xhc0 will in fact respond to wakeup triggers by default.

Similar problems have been reported here and here.

Other kernels

The LTS kernel does not exhibit the above behavior: that is, hibernation works fine. Unfortunately, with linux 6.11 rc5, my system will immediately wakeup from hibernation if the conditions outlined above apply.

:page_with_curl: Interesting information from dmesg

dmesg contains some interesting information in case either of the conditions above obtains. All information provided here stems from the 6.10.7 kernel.

Because dmesg outputs a lot whenever hibernation, I had to put it in a gist. This log reflects what happens when bluetooth is turned on.

Given the fact that the problem is dependend on the hardware’s state, I’m betting that this a problem that sits below userspace. However, I’m not seeing any similar reports, except this thread. There, the conversation stopped after a question about the size of the swapfile. But as my logs indicate, the size of the swapfile is clearly not at fault here (a contention further corrabarated by the fact that hibernation will still work in some cases, e.g. by just simply turning my bluetooth off). FTR: I have 16GB of RAM, and a swapfile that can hold 28GB.

The above log also shows the kernel freaking out (see the call trace). I thought this happened exclusively in instances where hibernation failed, but I’ve now noticed similar logs in instances where hibernation does in fact work.

Finally, the dmesg logs also suggest that the laptop gets really far in the hibernation process — the image gets written to swapfile, all CPUS go offline, etc. In fact, it kind of seems like the system actually hibernates briefly, and then is interupted somehow, though it is important to note that the light on the power button never turns off. The latter is a sign of hibernation working as expected.


Here’s a log of hibernation working fine (with bluetooth turned on, and the XHC0 wakeup trigger disabled):

succesful hibernation with bluetooth on, and XHC0 triggers off
[31151.152569] wlan0: deauthenticating from c0:06:c3:0b:84:e6 by local choice (Reason: 3=DEAUTH_LEAVING)
[31152.124764] PM: Image not found (code -16)
[31152.133232] PM: hibernation: hibernation entry
[31152.138200] Filesystems sync: 0.003 seconds
[31152.138360] Freezing user space processes
[31152.139907] Freezing user space processes completed (elapsed 0.001 seconds)
[31152.139912] OOM killer disabled.
[31152.140028] PM: hibernation: Marking nosave pages: [mem 0x00000000-0x00000fff]
[31152.140031] PM: hibernation: Marking nosave pages: [mem 0x0009f000-0x000fffff]
[31152.140034] PM: hibernation: Marking nosave pages: [mem 0x09b00000-0x09dfffff]
[31152.140044] PM: hibernation: Marking nosave pages: [mem 0x09f00000-0x09f3bfff]
[31152.140046] PM: hibernation: Marking nosave pages: [mem 0x49b60000-0x4bd5ffff]
[31152.140143] PM: hibernation: Marking nosave pages: [mem 0x4bd69000-0x4bd6cfff]
[31152.140144] PM: hibernation: Marking nosave pages: [mem 0x4bd6f000-0x4bd6ffff]
[31152.140145] PM: hibernation: Marking nosave pages: [mem 0x4bd78000-0x4bd78fff]
[31152.140147] PM: hibernation: Marking nosave pages: [mem 0x4bd7c000-0x4bd7cfff]
[31152.140148] PM: hibernation: Marking nosave pages: [mem 0x52ea8000-0x52ef6fff]
[31152.140150] PM: hibernation: Marking nosave pages: [mem 0x5491d000-0x5491dfff]
[31152.140151] PM: hibernation: Marking nosave pages: [mem 0x57f7f000-0x5affefff]
[31152.140301] PM: hibernation: Marking nosave pages: [mem 0x5b000000-0xffffffff]
[31152.141542] PM: hibernation: Basic memory bitmaps created
[31152.142493] PM: hibernation: Preallocating image memory
[31162.613944] PM: hibernation: Allocated 806025 pages for snapshot
[31162.613950] PM: hibernation: Allocated 3224100 kbytes in 10.47 seconds (307.93 MB/s)
[31162.613953] Freezing remaining freezable tasks
[31162.905110] Freezing remaining freezable tasks completed (elapsed 0.291 seconds)
[31162.906701] printk: Suspending console(s) (use no_console_suspend to debug)
[31162.912842] btusb 1-5:1.1: suspend error -16
[31162.912845] btusb 1-5:1.0: suspend error -16
[31163.544507] ACPI: EC: interrupt blocked
[31163.574013] ACPI: PM: Preparing to enter system sleep state S4
[31163.576404] ACPI: EC: event blocked
[31163.576405] ACPI: EC: EC stopped
[31163.576406] ACPI: PM: Saving platform NVS memory
[31163.579597] Disabling non-boot CPUs ...
[31163.581678] smpboot: CPU 1 is now offline
[31163.584064] smpboot: CPU 2 is now offline
[31163.586343] smpboot: CPU 3 is now offline
[31163.588661] smpboot: CPU 4 is now offline
[31163.590915] smpboot: CPU 5 is now offline
[31163.593218] smpboot: CPU 6 is now offline
[31163.595570] smpboot: CPU 7 is now offline
[31163.597884] smpboot: CPU 8 is now offline
[31163.599898] smpboot: CPU 9 is now offline
[31163.602027] smpboot: CPU 10 is now offline
[31163.604369] smpboot: CPU 11 is now offline
[31163.606564] smpboot: CPU 12 is now offline
[31163.608809] smpboot: CPU 13 is now offline
[31163.610943] smpboot: CPU 14 is now offline
[31163.611636] Spectre V2 : Update user space SMT mitigation: STIBP off
[31163.613181] smpboot: CPU 15 is now offline
[31163.614268] PM: hibernation: Creating image:
[31163.919251] PM: hibernation: Need to copy 1245557 pages
[31163.919256] PM: hibernation: Normal pages needed: 1245557 + 1024, available pages: 2373579
[31163.614756] ACPI: PM: Restoring platform NVS memory
[31163.615578] ACPI: EC: EC started
[31163.616274] LVT offset 0 assigned for vector 0x400
[31163.616883] Enabling non-boot CPUs ...
[31163.617053] smpboot: Booting Node 0 Processor 1 APIC 0x1
[31163.619807] ACPI: \_SB_.PLTF.C001: Found 3 idle states
[31163.619983] Spectre V2 : Update user space SMT mitigation: STIBP always-on
[31163.620002] CPU1 is up
[31163.620105] smpboot: Booting Node 0 Processor 2 APIC 0x2
[31163.622639] ACPI: \_SB_.PLTF.C002: Found 3 idle states
[31163.622796] CPU2 is up
[31163.622902] smpboot: Booting Node 0 Processor 3 APIC 0x3
[31163.625379] ACPI: \_SB_.PLTF.C003: Found 3 idle states
[31163.625542] CPU3 is up
[31163.625664] smpboot: Booting Node 0 Processor 4 APIC 0x4
[31163.628288] ACPI: \_SB_.PLTF.C004: Found 3 idle states
[31163.628431] CPU4 is up
[31163.628548] smpboot: Booting Node 0 Processor 5 APIC 0x5
[31163.631035] ACPI: \_SB_.PLTF.C005: Found 3 idle states
[31163.631222] CPU5 is up
[31163.631370] smpboot: Booting Node 0 Processor 6 APIC 0x6
[31163.633916] ACPI: \_SB_.PLTF.C006: Found 3 idle states
[31163.634124] CPU6 is up
[31163.634240] smpboot: Booting Node 0 Processor 7 APIC 0x7
[31163.636719] ACPI: \_SB_.PLTF.C007: Found 3 idle states
[31163.636932] CPU7 is up
[31163.637037] smpboot: Booting Node 0 Processor 8 APIC 0x8
[31163.639555] ACPI: \_SB_.PLTF.C008: Found 3 idle states
[31163.639791] CPU8 is up
[31163.639949] smpboot: Booting Node 0 Processor 9 APIC 0x9
[31163.642504] ACPI: \_SB_.PLTF.C009: Found 3 idle states
[31163.642749] CPU9 is up
[31163.642848] smpboot: Booting Node 0 Processor 10 APIC 0xa
[31163.645338] ACPI: \_SB_.PLTF.C00A: Found 3 idle states
[31163.645555] CPU10 is up
[31163.645668] smpboot: Booting Node 0 Processor 11 APIC 0xb
[31163.648124] ACPI: \_SB_.PLTF.C00B: Found 3 idle states
[31163.648404] CPU11 is up
[31163.648591] smpboot: Booting Node 0 Processor 12 APIC 0xc
[31163.651128] ACPI: \_SB_.PLTF.C00C: Found 3 idle states
[31163.651374] CPU12 is up
[31163.651471] smpboot: Booting Node 0 Processor 13 APIC 0xd
[31163.653954] ACPI: \_SB_.PLTF.C00D: Found 3 idle states
[31163.654264] CPU13 is up
[31163.654390] smpboot: Booting Node 0 Processor 14 APIC 0xe
[31163.656881] ACPI: \_SB_.PLTF.C00E: Found 3 idle states
[31163.657163] CPU14 is up
[31163.657281] smpboot: Booting Node 0 Processor 15 APIC 0xf
[31163.659772] ACPI: \_SB_.PLTF.C00F: Found 3 idle states
[31163.660128] CPU15 is up
[31163.663035] ACPI: PM: Waking up from system sleep state S4
[31163.668513] ACPI: EC: interrupt unblocked
[31163.804466] ACPI: EC: event unblocked
[31163.804482] usb usb1: root hub lost power or was reset
[31163.804483] usb usb3: root hub lost power or was reset
[31163.804486] usb usb2: root hub lost power or was reset
[31163.804487] usb usb4: root hub lost power or was reset
[31163.804844] usb usb5: root hub lost power or was reset
[31163.804847] usb usb6: root hub lost power or was reset
[31163.804939] usb usb7: root hub lost power or was reset
[31163.804941] usb usb8: root hub lost power or was reset
[31163.805141] [drm] PCIE GART of 512M enabled (table at 0x000000807FD00000).
[31163.805179] amdgpu 0000:c1:00.0: amdgpu: PSP is resuming...
[31163.818226] nvme nvme0: 16/0/0 default/read/poll queues
[31163.829479] amdgpu 0000:c1:00.0: amdgpu: reserve 0x4000000 from 0x8078000000 for PSP TMR
[31164.077736] usb 3-1: reset high-speed USB device number 2 using xhci_hcd
[31164.087764] usb 1-5: reset high-speed USB device number 3 using xhci_hcd
[31164.351041] usb 1-1: reset high-speed USB device number 11 using xhci_hcd
[31164.382805] amdgpu 0000:c1:00.0: amdgpu: RAS: optional ras ta ucode is not available
[31164.391264] amdgpu 0000:c1:00.0: amdgpu: RAP: optional rap ta ucode is not available
[31164.391267] amdgpu 0000:c1:00.0: amdgpu: SECUREDISPLAY: securedisplay ta ucode is not available
[31164.391270] amdgpu 0000:c1:00.0: amdgpu: SMU is resuming...
[31164.423038] amdgpu 0000:c1:00.0: amdgpu: SMU is resumed successfully!
[31164.425637] [drm] DMUB hardware initialized: version=0x08004000
[31164.575221] [drm] kiq ring mec 3 pipe 1 q 0
[31164.578590] amdgpu 0000:c1:00.0: [drm:jpeg_v4_0_hw_init [amdgpu]] JPEG decode initialized successfully.
[31164.579150] amdgpu 0000:c1:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
[31164.579152] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0
[31164.579153] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0
[31164.579154] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 6 on hub 0
[31164.579155] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 7 on hub 0
[31164.579156] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 8 on hub 0
[31164.579157] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 9 on hub 0
[31164.579157] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 10 on hub 0
[31164.579159] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 11 on hub 0
[31164.579159] amdgpu 0000:c1:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0
[31164.579160] amdgpu 0000:c1:00.0: amdgpu: ring vcn_unified_0 uses VM inv eng 0 on hub 8
[31164.579162] amdgpu 0000:c1:00.0: amdgpu: ring jpeg_dec uses VM inv eng 1 on hub 8
[31164.579163] amdgpu 0000:c1:00.0: amdgpu: ring mes_kiq_3.1.0 uses VM inv eng 13 on hub 0
[31164.582853] [drm] ring gfx_32775.1.1 was added
[31164.583512] [drm] ring compute_32775.2.2 was added
[31164.584137] [drm] ring sdma_32775.3.3 was added
[31164.584162] [drm] ring gfx_32775.1.1 ib test pass
[31164.584188] [drm] ring compute_32775.2.2 ib test pass
[31164.584296] [drm] ring sdma_32775.3.3 ib test pass
[31164.618047] usb 1-4: reset full-speed USB device number 2 using xhci_hcd
[31164.882079] usb 2-1: reset SuperSpeed Plus Gen 2x1 USB device number 5 using xhci_hcd
[31164.971854] usb 1-1.4: reset high-speed USB device number 12 using xhci_hcd
[31165.141760] usb 1-1.5: reset high-speed USB device number 13 using xhci_hcd
[31165.861591] usb 1-1.4.5: reset high-speed USB device number 14 using xhci_hcd
[31166.994680] mt7921e 0000:01:00.0: Message 00020007 (seq 4) timeout
[31166.994694] mt7921e 0000:01:00.0: PM: dpm_run_callback(): pci_pm_restore returns -110
[31166.994707] mt7921e 0000:01:00.0: PM: failed to restore async: error -110
... pretty sure this is the moment my laptop wakes up again
[31166.999473] PM: hibernation: Basic memory bitmaps freed
[31166.999719] OOM killer enabled.
[31166.999720] Restarting tasks ... done.
[31167.001208] Bluetooth: hci0: HW/SW Version: 0x008a008a, Build Time: 20240716163633
[31167.001912] PM: hibernation: hibernation exit
[31167.002283] usb 2-1.4: USB disconnect, device number 6
[31167.002287] r8152-cfgselector 2-1.4.3: USB disconnect, device number 7
[31167.077349] mt7921e 0000:01:00.0: HW/SW Version: 0x8a108a10, Build Time: 20240716163242a
[31167.354505] usb 2-1.4: new SuperSpeed Plus Gen 2x1 USB device number 8 using xhci_hcd
[31167.371486] usb 2-1.4: New USB device found, idVendor=0424, idProduct=7216, bcdDevice= 1.91
[31167.371491] usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[31167.371493] usb 2-1.4: Product: USB7216 Smart Hub

:zzz: Hibernation setup

To hibernate, I simply use a swapfile at /swapfile. Since systemd version v255, hibernating to said file works automagically, without fiddling with kernel parameters (see the Arch Wiki), provided that your initramfs generator also supports the relevant systemd mechanism.

Again, I would say that the problem lies not with how I’ve setup hibernation — I can get it to work reliably under certian conditions, and dmesg suggests that something below userspace is at play.

Really curious to know if others are also seeing this issue!

:hammer_and_wrench: Workaround

There is a pretty simple workaround for the issue, other than not using bluetooth at all :slightly_smiling_face:

  1. Disable wakeups from XHC0:
$ echo "XHC0" | sudo tee /proc/acpi/wakeup
  1. Run sudo systemctl hibernate to check if that indeed fixed your hibernation issue.

  2. Values in /proc/acpi/wakeup are not persistent across boots. To re-apply the workaround at every boot, run:

$ echo 'w    /proc/acpi/wakeup     -    -    -    -   XHC0' | sudo tee /etc/tmpfiles.d/disable-xhc0-wakeup.conf

The above creates a file that tells systemd to write the string XHC0 to /proc/acpi/wakeup at boot.

When/if this issue gets fixed, simply delete /etc/tmpfiles.d/disable-xhc0-wakeup.conf again.

A few thoughts:

  1. If you can’t reproduce in LTS kernel it’s a good idea to bisect it to figure out what broke it.
  2. The thunderbolt problems reported I suspect will be fixed by this series:
    [PATCH v5 0/5] Verify devices transition from D3cold to D0 - Mario Limonciello

Can you try that and see? It would be a bonus if it fixed your instant wakeup, but I don’t expect it.

Thanks for the quick response!

EDIT: the patch series you mentioned applies to 6.11 just fine, sorry for the noise.
Just to save myself some time: I probably need to apply that thunderbold patch to 6.11 rc5, right?

It would be a bonus if it fixed your instant wakeup, but I don’t expect it.

It does seem to help and/or change things, quite suprinsgly! At the very least, I’m not seeing instant wakeups anymore. Instead, compared to vanilla 6.11, kernel 6.11 + your patch will lead to either of two outcomes: either my laptop shutsdown normally after systemctl hibernate, or it will seem to hang (where “hanging” means blank screen+power LED still on, while nothing else seems to happen). This is all with the xhc0 device listening to wakeup triggers (i.e. set to enabled)

Somewhat confusingly, I’m having trouble coming back from hibernation with your patch though, even if I explicilty specify resume= and resume_offset= on the kernel commandline (something I never have to do with 6.10). More specifically, I’m being greeted by systemd, and then the boot doesn’t proceed after the message “Starting resume from hibernation” (no errors are shown). This behavior seems strange enough to file it under “user error”, but then again, 6.10 and 6.11 vanilla seem to be able to resume from hibernate just fine.

Moreover, I’m still seeing a call trace in dmesg with your patch, including in situations where hibernation actually succeeds. Probably not something you aimed to fix, since it doesn’t seem to relate to thunderbolt , but here you go:

dmesg during a successful round of hibernation, on 6.11 with the patch you linked
  [   50.380944] wlan0: deauthenticating from c0:06:c3:0b:84:e7 by local choice (Reason: 3=  DEAUTH_LEAVING)
  [   51.464048] PM: Image not found (code -16)
  [   51.471738] PM: hibernation: hibernation entry
  [   51.474563] Filesystems sync: 0.000 seconds
  [   51.475066] Freezing user space processes
  [   51.476424] Freezing user space processes completed (elapsed 0.001 seconds)
  [   51.476427] OOM killer disabled.
  [   51.476542] PM: hibernation: Marking nosave pages: [mem 0x00000000-0x00000fff]
  [   51.476544] PM: hibernation: Marking nosave pages: [mem 0x0009f000-0x000fffff]
  [   51.476547] PM: hibernation: Marking nosave pages: [mem 0x09b00000-0x09dfffff]
  [   51.476558] PM: hibernation: Marking nosave pages: [mem 0x09f00000-0x09f3bfff]
  [   51.476560] PM: hibernation: Marking nosave pages: [mem 0x49b60000-0x4bd5ffff]
  [   51.476670] PM: hibernation: Marking nosave pages: [mem 0x4bd69000-0x4bd6cfff]
  [   51.476671] PM: hibernation: Marking nosave pages: [mem 0x4bd6f000-0x4bd6ffff]
  [   51.476672] PM: hibernation: Marking nosave pages: [mem 0x4bd78000-0x4bd78fff]
  [   51.476673] PM: hibernation: Marking nosave pages: [mem 0x4bd7c000-0x4bd7cfff]
  [   51.476674] PM: hibernation: Marking nosave pages: [mem 0x52ea5000-0x52ef3fff]
  [   51.476675] PM: hibernation: Marking nosave pages: [mem 0x5491d000-0x5491dfff]
  [   51.476676] PM: hibernation: Marking nosave pages: [mem 0x57f7f000-0x5affefff]
  [   51.476833] PM: hibernation: Marking nosave pages: [mem 0x5b000000-0xffffffff]
  [   51.481359] PM: hibernation: Basic memory bitmaps created
  [   51.482319] PM: hibernation: Preallocating image memory
  [   52.054123] PM: hibernation: Allocated 499573 pages for snapshot
  [   52.054132] PM: hibernation: Allocated 1998292 kbytes in 0.57 seconds (3505.77 MB/s)
  [   52.054135] Freezing remaining freezable tasks
  [   52.056533] Freezing remaining freezable tasks completed (elapsed 0.002 seconds)
  [   52.058294] printk: Suspending console(s) (use no_console_suspend to debug)
  [   52.066628] atkbd serio0: Disabling IRQ1 wakeup source to avoid platform firmware bug
  [   52.066633] ------------[ cut here ]------------
  [   52.066634] Unbalanced IRQ 1 wake disable
  [   52.066639] WARNING: CPU: 2 PID: 1542 at kernel/irq/manage.c:923 irq_set_irq_wake+0x15  c/0x1b0
  [   52.066647] Modules linked in: snd_seq_dummy snd_hrtimer rfcomm snd_seq snd_seq_device   uhid ccm algif_aead crypto_null des3_ede_x86_64 cbc des_generic libdes algif_skcipher cm  ac md4 algif_hash af_alg bnep amd_atl vfat intel_rapl_msr fat intel_rapl_common mousedev   amdgpu snd_sof_amd_acp63 snd_sof_amd_vangogh snd_sof_amd_rembrandt snd_sof_amd_renoir snd  _sof_amd_acp snd_sof_pci snd_sof_xtensa_dsp snd_sof kvm_amd snd_sof_utils mt7921e mt7921_  common snd_pci_ps snd_amd_sdw_acpi mt792x_lib soundwire_amd kvm snd_hda_codec_realtek mt7  6_connac_lib soundwire_generic_allocation soundwire_bus mt76 snd_hda_codec_generic cros_u  sbpd_charger leds_cros_ec snd_hda_scodec_component snd_hda_codec_hdmi snd_soc_core snd_hd  a_intel led_class_multicolor cros_kbd_led_backlight cros_ec_sysfs cros_ec_chardev cros_ch  arge_control cros_ec_debugfs cros_usbpd_notify cros_usbpd_logger cros_ec_hwmon gpio_cros_  ec hid_sensor_als crct10dif_pclmul amdxcp snd_compress snd_intel_dspcfg drm_exec crc32_pc  lmul snd_intel_sdw_acpi ac97_bus hid_sensor_trigger
  [   52.066714]  cros_ec_dev mac80211 polyval_clmulni gpu_sched snd_pcm_dmaengine industri  alio_triggered_buffer snd_hda_codec polyval_generic snd_rpl_pci_acp6x drm_buddy kfifo_buf   uvcvideo ghash_clmulni_intel snd_acp_pci hid_sensor_iio_common snd_hda_core i2c_algo_bit   sha512_ssse3 videobuf2_vmalloc snd_acp_legacy_common btusb industrialio drm_suballoc_hel  per sha256_ssse3 snd_pci_acp6x uvc snd_hwdep videobuf2_memops drm_ttm_helper libarc4 btrt  l sha1_ssse3 snd_pcm videobuf2_v4l2 snd_pci_acp5x ttm btintel serio_raw aesni_intel snd_r  n_pci_acp3x joydev atkbd gf128mul videodev drm_display_helper snd_timer hid_multitouch uc  si_acpi cros_ec_lpcs btbcm hid_sensor_hub amd_pmf crypto_simd libps2 snd_acp_config cfg80  211 hid_generic vivaldi_fmap videobuf2_common wmi_bmof cros_ec snd cec i2c_piix4 btmtk am  dtee typec_ucsi snd_soc_acpi cryptd bluetooth mc rapl typec thunderbolt pcspkr amd_sfh so  undcore rfkill snd_pci_acp3x k10temp video ccp i2c_smbus roles i2c_hid_acpi platform_prof  ile amd_pmc i8042 wmi tee mac_hid i2c_hid serio
  [   52.066794]  pkcs8_key_parser i2c_dev crypto_user loop dm_mod nfnetlink ip_tables x_ta  bles ext4 crc32c_generic crc16 mbcache jbd2 nvme nvme_core xhci_pci crc32c_intel xhci_pci  _renesas nvme_auth
  [   52.066814] CPU: 2 UID: 0 PID: 1542 Comm: systemd-sleep Not tainted 6.11.0-rc5-1-mainl  ine-dirty #1 8f4e6e1b02693d249fc50573191ab4d4e50c592d
  [   52.066818] Hardware name: Framework Laptop 13 (AMD Ryzen 7040Series)/FRANMDCP07, BIOS   03.05 03/29/2024
  [   52.066820] RIP: 0010:irq_set_irq_wake+0x15c/0x1b0
  [   52.066824] Code: f6 ff d2 0f 1f 00 89 c5 85 c0 74 38 c7 83 c4 00 00 00 01 00 00 00 e9   32 ff ff ff 44 89 e6 48 c7 c7 18 d6 28 9e e8 64 fe f5 ff <0f> 0b 31 ed e9 1a ff ff ff 48   8b 43 78 31 ed 81 08 00 40 00 00 e9
  [   52.066825] RSP: 0018:ffffb703062cfb98 EFLAGS: 00010086
  [   52.066828] RAX: 0000000000000000 RBX: ffff98a5c005a400 RCX: 0000000000000027
  [   52.066829] RDX: ffff98a8cfb21a48 RSI: 0000000000000001 RDI: ffff98a8cfb21a40
  [   52.066831] RBP: 00000000ffffffea R08: 0000000000000000 R09: ffffb703062cf960
  [   52.066832] R10: ffffffff9eab3fa8 R11: 0000000000000003 R12: 0000000000000001
  [   52.066833] R13: 0000000000000000 R14: 0000000000000000 R15: ffff98a5c1965010
  [   52.066834] FS:  00007f20c8a68880(0000) GS:ffff98a8cfb00000(0000) knlGS:00000000000000  00
  [   52.066836] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [   52.066837] CR2: 00007f373bc00988 CR3: 0000000123d52000 CR4: 0000000000f50ef0
  [   52.066839] PKRU: 55555554
  [   52.066840] Call Trace:
  [   52.066843]  <TASK>
  [   52.066844]  ? irq_set_irq_wake+0x15c/0x1b0
  [   52.066847]  ? __warn.cold+0x8e/0xe8
  [   52.066851]  ? irq_set_irq_wake+0x15c/0x1b0
  [   52.066859]  ? report_bug+0xff/0x140
  [   52.066863]  ? console_unlock+0xd7/0x130
  [   52.066868]  ? handle_bug+0x3c/0x80
  [   52.066871]  ? exc_invalid_op+0x17/0x70
  [   52.066873]  ? asm_exc_invalid_op+0x1a/0x20
  [   52.066878]  ? irq_set_irq_wake+0x15c/0x1b0
  [   52.066881]  ? irq_set_irq_wake+0x15c/0x1b0
  [   52.066884]  ? __pfx_acpi_subsys_freeze+0x10/0x10
  [   52.066890]  amd_pmc_suspend_handler+0x9d/0xd0 [amd_pmc 620659fded5742e65b9386e8c02c85  0312b99bdc]
  [   52.066895]  ? acpi_subsys_freeze+0x14/0x20
  [   52.066897]  dpm_run_callback+0x47/0x150
  [   52.066903]  device_suspend+0x141/0x510
  [   52.066905]  ? srso_alias_return_thunk+0x5/0xfbef5
  [   52.066909]  dpm_suspend+0x105/0x240
  [   52.066911]  hibernation_snapshot+0xcc/0x520
  [   52.066915]  hibernate.cold+0xcd/0x333
  [   52.066919]  state_store+0xc3/0xd0
  [   52.066922]  kernfs_fop_write_iter+0x13e/0x1f0
  [   52.066927]  vfs_write+0x291/0x460
  [   52.066933]  ksys_write+0x6d/0xf0
  [   52.066937]  do_syscall_64+0x82/0x190
  [   52.066941]  ? srso_alias_return_thunk+0x5/0xfbef5
  [   52.066943]  ? __x64_sys_fcntl+0x98/0xd0
  [   52.066946]  ? srso_alias_return_thunk+0x5/0xfbef5
  [   52.066948]  ? syscall_exit_to_user_mode+0x10/0x200
  [   52.066951]  ? srso_alias_return_thunk+0x5/0xfbef5
  [   52.066952]  ? do_syscall_64+0x8e/0x190
  [   52.066955]  ? srso_alias_return_thunk+0x5/0xfbef5
  [   52.066957]  ? kmem_cache_free+0x3d5/0x400
  [   52.066961]  ? srso_alias_return_thunk+0x5/0xfbef5
  [   52.066963]  ? srso_alias_return_thunk+0x5/0xfbef5
  [   52.066965]  ? syscall_exit_to_user_mode+0x10/0x200
  [   52.066967]  ? srso_alias_return_thunk+0x5/0xfbef5
  [   52.066969]  ? do_syscall_64+0x8e/0x190
  [   52.066971]  ? exc_page_fault+0x81/0x190
  [   52.066973]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
  [   52.066975] RIP: 0033:0x7f20c851b7a4
  [   52.067022] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3   0f 1e fa 80 3d c5 28 0e 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f   1f 00 55 48 89 e5 48 83 ec 20 48 89
  [   52.067023] RSP: 002b:00007ffc4cf39ec8 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
  [   52.067025] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007f20c851b7a4
  [   52.067026] RDX: 0000000000000005 RSI: 0000556210fa8d00 RDI: 0000000000000007
  [   52.067028] RBP: 00007ffc4cf39ef0 R08: 0000000000000000 R09: 0000000000000001
  [   52.067029] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000005
  [   52.067030] R13: 0000556210fa8d00 R14: 0000556210f9f2a0 R15: 00007f20c85f4ea0
  [   52.067034]  </TASK>
  [   52.067035] ---[ end trace 0000000000000000 ]---
  [   52.439872] ACPI: EC: interrupt blocked
  [   52.499751] ACPI: PM: Preparing to enter system sleep state S4
  [   52.502445] ACPI: EC: event blocked
  [   52.502447] ACPI: EC: EC stopped
  [   52.502447] ACPI: PM: Saving platform NVS memory
  [   52.507909] Disabling non-boot CPUs ...
  [   52.509938] smpboot: CPU 15 is now offline
  [   52.513295] smpboot: CPU 14 is now offline
  [   52.516352] smpboot: CPU 13 is now offline
  [   52.519256] smpboot: CPU 12 is now offline
  [   52.522149] smpboot: CPU 11 is now offline
  [   52.525013] smpboot: CPU 10 is now offline
  [   52.527675] smpboot: CPU 9 is now offline
  [   52.530307] smpboot: CPU 8 is now offline
  [   52.533325] smpboot: CPU 7 is now offline
  [   52.536262] smpboot: CPU 6 is now offline
  [   52.539397] smpboot: CPU 5 is now offline
  [   52.541883] smpboot: CPU 4 is now offline
  [   52.544614] smpboot: CPU 3 is now offline
  [   52.547083] smpboot: CPU 2 is now offline
  [   52.550054] smpboot: CPU 1 is now offline
  [   52.551104] Spectre V2 : Update user space SMT mitigation: STIBP off
  [   52.551588] PM: hibernation: Creating image:
  [   52.896094] PM: hibernation: Need to copy 548260 pages
  [   52.896102] PM: hibernation: Normal pages needed: 548260 + 1024, available pages: 3070  808
  [   52.552484] ACPI: PM: Restoring platform NVS memory
  [   52.554184] ACPI: EC: EC started
  [   52.554897] LVT offset 0 assigned for vector 0x400
  [   52.555654] Enabling non-boot CPUs ...
  [   52.556024] smpboot: Booting Node 0 Processor 1 APIC 0x1
  [   52.560172] ACPI: \_SB_.PLTF.C001: Found 3 idle states
  [   52.560437] Spectre V2 : Update user space SMT mitigation: STIBP always-on
  [   52.560458] CPU1 is up
  [   52.560658] smpboot: Booting Node 0 Processor 2 APIC 0x2
  [   52.563548] ACPI: \_SB_.PLTF.C002: Found 3 idle states
  [   52.563786] CPU2 is up
  [   52.563988] smpboot: Booting Node 0 Processor 3 APIC 0x3
  [   52.566833] ACPI: \_SB_.PLTF.C003: Found 3 idle states
  [   52.567091] CPU3 is up
  [   52.567298] smpboot: Booting Node 0 Processor 4 APIC 0x4
  [   52.570322] ACPI: \_SB_.PLTF.C004: Found 3 idle states
  [   52.570607] CPU4 is up
  [   52.570826] smpboot: Booting Node 0 Processor 5 APIC 0x5
  [   52.573723] ACPI: \_SB_.PLTF.C005: Found 3 idle states
  [   52.574019] CPU5 is up
  [   52.574223] smpboot: Booting Node 0 Processor 6 APIC 0x6
  [   52.577267] ACPI: \_SB_.PLTF.C006: Found 3 idle states
  [   52.577584] CPU6 is up
  [   52.577791] smpboot: Booting Node 0 Processor 7 APIC 0x7
  [   52.580743] ACPI: \_SB_.PLTF.C007: Found 3 idle states
  [   52.581056] CPU7 is up
  [   52.581277] smpboot: Booting Node 0 Processor 8 APIC 0x8
  [   52.584213] ACPI: \_SB_.PLTF.C008: Found 3 idle states
  [   52.584570] CPU8 is up
  [   52.584819] smpboot: Booting Node 0 Processor 9 APIC 0x9
  [   52.587644] ACPI: \_SB_.PLTF.C009: Found 3 idle states
  [   52.588007] CPU9 is up
  [   52.588205] smpboot: Booting Node 0 Processor 10 APIC 0xa
  [   52.591092] ACPI: \_SB_.PLTF.C00A: Found 3 idle states
  [   52.591408] CPU10 is up
  [   52.591606] smpboot: Booting Node 0 Processor 11 APIC 0xb
  [   52.594461] ACPI: \_SB_.PLTF.C00B: Found 3 idle states
  [   52.594882] CPU11 is up
  [   52.595146] smpboot: Booting Node 0 Processor 12 APIC 0xc
  [   52.597978] ACPI: \_SB_.PLTF.C00C: Found 3 idle states
  [   52.598335] CPU12 is up
  [   52.598528] smpboot: Booting Node 0 Processor 13 APIC 0xd
  [   52.601524] ACPI: \_SB_.PLTF.C00D: Found 3 idle states
  [   52.601980] CPU13 is up
  [   52.602178] smpboot: Booting Node 0 Processor 14 APIC 0xe
  [   52.605084] ACPI: \_SB_.PLTF.C00E: Found 3 idle states
  [   52.605463] CPU14 is up
  [   52.605666] smpboot: Booting Node 0 Processor 15 APIC 0xf
  [   52.608591] ACPI: \_SB_.PLTF.C00F: Found 3 idle states
  [   52.609079] CPU15 is up
  [   52.613445] ACPI: PM: Waking up from system sleep state S4
  [   52.619849] ACPI: EC: interrupt unblocked
  [   52.718337] usb usb1: root hub lost power or was reset
  [   52.718338] usb usb3: root hub lost power or was reset
  [   52.718345] usb usb2: root hub lost power or was reset
  [   52.718345] usb usb4: root hub lost power or was reset
  [   52.718366] ACPI: EC: event unblocked
  [   52.718948] [drm] PCIE GART of 512M enabled (table at 0x000000807FD00000).
  [   52.719006] usb usb5: root hub lost power or was reset
  [   52.719006] amdgpu 0000:c1:00.0: amdgpu: PSP is resuming...
  [   52.719010] usb usb6: root hub lost power or was reset
  [   52.719133] usb usb7: root hub lost power or was reset
  [   52.719137] usb usb8: root hub lost power or was reset
  [   52.732878] nvme nvme0: 16/0/0 default/read/poll queues
  [   52.743276] amdgpu 0000:c1:00.0: amdgpu: reserve 0x4000000 from 0x8078000000 for PSP T  MR
  [   52.991368] usb 3-1: reset high-speed USB device number 2 using xhci_hcd
  [   52.991387] usb 1-5: reset high-speed USB device number 3 using xhci_hcd
  [   53.297911] amdgpu 0000:c1:00.0: amdgpu: RAS: optional ras ta ucode is not available
  [   53.306173] amdgpu 0000:c1:00.0: amdgpu: RAP: optional rap ta ucode is not available
  [   53.306176] amdgpu 0000:c1:00.0: amdgpu: SECUREDISPLAY: securedisplay ta ucode is not   available
  [   53.306179] amdgpu 0000:c1:00.0: amdgpu: SMU is resuming...
  [   53.337230] amdgpu 0000:c1:00.0: amdgpu: SMU is resumed successfully!
  [   53.339561] [drm] DMUB hardware initialized: version=0x08004000
  [   53.347649] amdgpu 0000:c1:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
  [   53.347652] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0
  [   53.347654] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0
  [   53.347656] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 6 on hub 0
  [   53.347657] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 7 on hub 0
  [   53.347658] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 8 on hub 0
  [   53.347659] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 9 on hub 0
  [   53.347660] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 10 on hub 0
  [   53.347662] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 11 on hub 0
  [   53.347663] amdgpu 0000:c1:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0
  [   53.347664] amdgpu 0000:c1:00.0: amdgpu: ring vcn_unified_0 uses VM inv eng 0 on hub 8  [   53.347666] amdgpu 0000:c1:00.0: amdgpu: ring jpeg_dec uses VM inv eng 1 on hub 8
  [   53.347667] amdgpu 0000:c1:00.0: amdgpu: ring mes_kiq_3.1.0 uses VM inv eng 13 on hub   0
  [   53.351765] [drm] ring gfx_32773.1.1 was added
  [   53.352540] [drm] ring compute_32773.2.2 was added
  [   53.353288] [drm] ring sdma_32773.3.3 was added
  [   53.353338] [drm] ring gfx_32773.1.1 ib test pass
  [   53.353384] [drm] ring compute_32773.2.2 ib test pass
  [   53.353471] [drm] ring sdma_32773.3.3 ib test pass
  [   55.785151] mt7921e 0000:01:00.0: Message 00020007 (seq 13) timeout
  [   55.785169] mt7921e 0000:01:00.0: PM: dpm_run_callback(): pci_pm_restore returns -110
  [   55.785180] mt7921e 0000:01:00.0: PM: failed to restore async: error -110
  [   55.789788] PM: hibernation: Basic memory bitmaps freed
  [   55.790001] OOM killer enabled.
  [   55.790003] Restarting tasks ... 
  [   55.790081] usb 1-4: USB disconnect, device number 2
  [   55.790534] done.
  [   55.794508] PM: hibernation: hibernation exit
  [   55.870567] mt7921e 0000:01:00.0: HW/SW Version: 0x8a108a10, Build Time: 2024071616324  2a
  
  [   55.901503] Bluetooth: hci0: HW/SW Version: 0x008a008a, Build Time: 20240716163633
  [   55.914695] usb 1-4: new full-speed USB device number 4 using xhci_hcd
  [   56.083420] usb 1-4: New USB device found, idVendor=27c6, idProduct=609c, bcdDevice= 1  .00
  [   56.083429] usb 1-4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
  [   56.083432] usb 1-4: Product: Goodix Fingerprint USB Device
  [   56.083434] usb 1-4: Manufacturer: Goodix Technology Co., Ltd.
  [   56.083436] usb 1-4: SerialNumber: UID27A025ED_XXXX_MOC_B0
  [   56.246541] mt7921e 0000:01:00.0: WM Firmware Version: ____000000, Build Time: 2024071  6163327
  [   57.001447] usb 1-4: reset full-speed USB device number 4 using xhci_hcd
  [   58.158602] Bluetooth: hci0: Device setup in 2313005 usecs
  [   58.158612] Bluetooth: hci0: HCI Enhanced Setup Synchronous Connection command is adve  rtised, but not supported.
  [   58.235216] Bluetooth: hci0: AOSP extensions version v1.00
  [   58.235226] Bluetooth: hci0: AOSP quality report is supported
  [   58.240887] Bluetooth: MGMT ver 1.23
  [   58.672457] wlan0: Ignoring ECSA in probe response - was considered stuck!
  [   58.713925] wlan0: authenticate with c0:06:c3:0b:84:e6 (local address=14:ac:60:29:83:d  b)
  [   59.199971] wlan0: send auth to c0:06:c3:0b:84:e6 (try 1/3)
  [   59.228765] wlan0: authenticated
  [   59.228831] wlan0: Ignoring ECSA in probe response - was considered stuck!
  [   59.231527] wlan0: associate with c0:06:c3:0b:84:e6 (try 1/3)
  [   59.261328] wlan0: RX AssocResp from c0:06:c3:0b:84:e6 (capab=0x1911 status=0 aid=4)
  [   59.286586] wlan0: associated
  [   59.289226] wlan0: Limiting TX power to 20 (20 - 0) dBm as advertised by c0:06:c3:0b:8  4:e6


edit:

Instead, compared to vanilla 6.11, kernel 6.11 + your patch will lead to either of two outcomes: either my laptop shutsdown normally after systemctl hibernate, or it will seem to hang (where “hanging” means blank screen+power LED still on, while nothing else seems to happen).

Okay, apperently this is not true. Instant wakeups might still happen, so there are three possibilities. Feels like some kind of race condition. In case of instant wakeups, dmesg fails as follows (sometimes including the call trace from the log above):

[   94.530469] PM: hibernation: hibernation entry
[   94.542306] Filesystems sync: 0.010 seconds
[   94.542583] Freezing user space processes
[   94.543597] Freezing user space processes completed (elapsed 0.001 seconds)
[   94.543601] OOM killer disabled.
[   94.543737] PM: hibernation: Marking nosave pages: [mem 0x00000000-0x00000fff]
[   94.543740] PM: hibernation: Marking nosave pages: [mem 0x0009f000-0x000fffff]
...
[   94.544028] PM: hibernation: Marking nosave pages: [mem 0x5b000000-0xffffffff]
[   94.548611] PM: hibernation: Basic memory bitmaps created
[   94.549671] PM: hibernation: Preallocating image memory
[   95.166889] PM: hibernation: Allocated 584913 pages for snapshot
[   95.166898] PM: hibernation: Allocated 2339652 kbytes in 0.61 seconds (3835.49 MB/s)
[   95.166902] Freezing remaining freezable tasks
[   95.168425] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
[   95.170045] printk: Suspending console(s) (use no_console_suspend to debug)

[   95.204550] xhci_hcd 0000:c1:00.3: PM: pci_pm_freeze(): hcd_pci_suspend returns -16
[   95.204611] xhci_hcd 0000:c1:00.3: PM: dpm_run_callback(): pci_pm_freeze returns -16
[   95.204627] xhci_hcd 0000:c1:00.3: PM: failed to freeze async: error -16
[   95.637739] [drm] PCIE GART of 512M enabled (table at 0x000000807FD00000).
[   95.637809] amdgpu 0000:c1:00.0: amdgpu: PSP is resuming...
...

The above is with xhc0 device listening to wakeup triggers. Noteworthy is that this device corresponds to 0000:c1:00.3 according to /proc/acpi/wakeup, and that whole pci_pm_freeze(): hcd_pci_suspend returns -16 error doesn’t show up in my 6.10 logs on failed attempts.

I will try a bisection next weekend, if I find the time!

Well it’s good that series does at least help a bit with what’s in your logs, but it does sound like multiple things going on. Will be interested to hear the bisect results.

This seems to have been fixed now. I was going to do some bisecting, but today, hibernate works just fine, even with wakeup triggers from the xhc0 device enabled and/or bluetooth being turned on.

No idea what fixed it though, but perhaps it was linux 6.10.8 (i was on 6.10.7 last week when I opened this).

Marking this thread as solved for now, and thanks for you continued work Mario_Limonciello :slightly_smiling_face: