[SOLVED] Using Linux QEMU/KVM causes s2idle hard freeze on Arch - Linux 6.10.8

David_Markey · September 7, 2024, 12:13am

Which Linux distro are you using?

Arch

Which release version?
7th September

Which kernel are you using?
6.10.8

Which BIOS version are you using?
03.05

Which Framework Laptop 13 model are you using? (AMD Ryzen™)

I’ve been trying to track this down… A s2idle suspend hard freezes my system when using QEMU with KVM acceleration.

Easy to repro… start qemu-system-x86_64 --accel kvm and try to suspend… Hard freeze… 10 second salute.

Nothing exotic… No extra kernel params.

Mario_Limonciello · September 7, 2024, 4:36am

How do you reproduce?

❯ uname -r
6.10.8-2-cachyos
❯ qemu-system-x86_64 --accel kvm

And then in another tab I ran a suspend sequence test and hit no problem.

❯ sudo ~/src/amd/scripts/amd_s2idle.py
Location of log file (default s2idle_report-2024-09-06.txt)? 
Debugging script for s2idle on AMD systems
💻 Framework Laptop 13 (AMD Ryzen 7040Series) (Laptop) running BIOS 3.5 (03.05) released 03/29/2024 and EC unknown
🐧 CachyOS
🐧 Kernel 6.10.8-2-cachyos
🔋 Battery BAT1 (NVT Framewo) is operating at 100.17% of design
Checking prerequisites for s2idle
✅ Logs are provided via systemd
✅ AMD Ryzen 5 7640U w/ Radeon 760M Graphics (family 19 model 74)
✅ SMT enabled
✅ LPS0 _DSM enabled
✅ ACPI FADT supports Low-power S0 idle
✅ HSMP driver `amd_hsmp` not detected (blocked: False)
✅ PMC driver `amd_pmc` loaded (Program 0 Firmware 76.82.0)
✅ USB4 driver `thunderbolt` bound to 0000:c3:00.5
✅ USB4 driver `thunderbolt` bound to 0000:c3:00.6
✅ GPU driver `amdgpu` bound to 0000:c1:00.0
✅ System is configured for s2idle
✅ NVME Sandisk Corp WD Black SN770 / PC SN740 256GB / PC SN560 (DRAM-less) NVMe SSD is configured for s2idle in BIOS
✅ GPIO driver `pinctrl_amd` available
How long should suspend cycles last in seconds (default 10)? 
How long to wait in between suspend cycles in seconds (default 4)? 
How many suspend cycles to run (default 1)? 
Started at 2024-09-06 23:36:00.277079 (cycle finish expected @ 2024-09-06 23:36:14.277092)
Results from last s2idle cycle
○ Suspend count: 1
○ Hardware sleep cycle count: 1
○ Wakeup triggered from IRQ 9: ACPI SCI
○ Woke up from IRQ 9: ACPI SCI
○ gpe0B increased from 146 to 160
✅ Userspace suspended for 0:00:12.503340
✅ In a hardware sleep state for 0:00:08.396301 (67.15%)
🔋 Battery BAT1 (NVT Framewo) is operating at 100.17% of design

parawizard · September 7, 2024, 6:15am

You’ve got no guests running? I’m guessing maybe there might be more to reproductio in steps

David_Markey · September 7, 2024, 9:52am

Curious, it’s not reproducible with that tool, but does happen with systemctl suspend

Here’s the tool output anyway.

gist.github.com

https://gist.github.com/dmarkey/248faea95e98fe19dce0f79fda9e179e

gistfile1.txt

2024-09-07 10:49:07,816 INFO:   Debugging script for s2idle on AMD systems
2024-09-07 10:49:07,816 INFO:   💻 Framework Laptop 13 (AMD Ryzen 7040Series) (Laptop) running BIOS 3.5 (03.05) released 03/29/2024 and EC unknown
2024-09-07 10:49:07,816 INFO:   🐧 Arch Linux
2024-09-07 10:49:07,816 INFO:   🐧 Kernel 6.10.8-arch1-1
2024-09-07 10:49:07,817 DEBUG:  BAT1 charge level is 3294000 µAh
2024-09-07 10:49:07,817 INFO:   🔋 Battery BAT1 (NVT FRANGWA) is operating at 101.38% of design
2024-09-07 10:49:07,817 INFO:   Checking prerequisites for s2idle
2024-09-07 10:49:07,817 INFO:   ✅ Logs are provided via systemd
2024-09-07 10:49:07,818 INFO:   ✅ AMD Ryzen 5 7640U w/ Radeon 760M Graphics (family 19 model 74)
2024-09-07 10:49:07,818 DEBUG:  SMT control: on

This file has been truncated. show original

David_Markey · September 7, 2024, 10:28am

Aha!

Some systemd change… this is the fix [SOLVED] System freezes after sleep since update / Newbie Corner / Arch Linux Forums
Not sure what side-effect this has on battery life.

Not hardware related at all, sorry!

Mario_Limonciello · September 7, 2024, 11:36am

github.com/systemd/systemd

Since 256, suspend fails over 50% of the time: Freezing user space processes failed

opened 04:54PM - 04 Jul 24 UTC

mahkoh

sleep kernel-bug

### systemd version the issue has been seen with 256 ### Used distribution Ar…ch ### Linux kernel version used 6.10.0-rc6 ### CPU architectures issue was seen on x86_64 ### Component _No response_ ### Expected behaviour you didn't see `systemctl suspend` suspends the system ### Unexpected behaviour you saw Suspending fails with ``` Jul 03 17:47:54 desk kernel: Freezing user space processes failed after 20.009 seconds (1 tasks refusing to freeze, wq_busy=0): Jul 03 17:47:54 desk kernel: task:jay state:R stack:0 pid:1550 tgid:1550 ppid:1533 flags:0x00024002 ``` I originally reported this issue here: https://bugzilla.kernel.org/show_bug.cgi?id=219004 In there I already suspected that systemd might be involved since the issue started appearing immediately after upgrading systemd from 255 to 256 and knowing that systemd restarts itself without requiring a system reboot. I had not rebooted the system for around two weeks at that point. I have now performed the following experiment which makes it look more likely that some behavior in systemd changed that has unintended consequences: - I downgraded systemd, systemd-libs, and systemd-sysvcompat to 255 and tried to suspend 4 times in a row. They all succeeded. - I upgraded to 256 and tried the same. 2 out of 4 attempts failed with the error above. - I again downgraded to 255 and again all attempts succeeded. The issue seems to happen with at least two stable 6.9.x kernels and also 6.10.0-rc6. I've added logging to the kernel to see why the process cannot be frozen and it shows that the condition https://github.com/torvalds/linux/blob/795c58e4c7fc6163d8fb9f2baa86cfe898fa4b19/kernel/freezer.c#L112 is always true. More precisely, `p->on_rq == TASK_ON_RQ_QUEUED`. Is it possible that systemd 256 has started doing something that would prevent a task with `p->on_rq == TASK_ON_RQ_QUEUED` during suspend to leave that state? ### Steps to reproduce the problem You can try to install https://github.com/mahkoh/jay to reproduce. But the issue might be specific to my system. ### Additional program output to the terminal or log subsystem illustrating the issue _No response_

github.com/systemd/systemd

user session fails to resume from suspend when user is using NFS or KVM

opened 05:46PM - 29 May 24 UTC

dkg

bug 🐛 pid1 sleep kernel-bug

### systemd version the issue has been seen with 256~rc3-2 ### Used distributi…on debian unstable ### Linux kernel version used 6.7.12-amd64 ### CPU architectures issue was seen on x86_64 ### Component systemd ### Expected behaviour you didn't see I put my machine to sleep, while a user session was running, with a graphical wayland environment (`sway`, locked with `swaylock`). Then i opened the lid to resume use. The machine woke up, but `swaylock` wasn't active any more, and i couldn't even `ctrl+alt+F2` to a different VT. However, pressing the power button did start the machine's shutdown process, though it also hung. ### Unexpected behaviour you saw New to me (since upgrading to systemd 256~rc3) was the "Failed to freeze user sessions" in the journal: ``` May 28 18:51:12 alice systemd[1]: Starting systemd-suspend.service - System Suspend... May 28 18:51:14 alice systemd-sleep[1242496]: Failed to freeze user sessions, ignoring: Connection timed out May 28 18:51:14 alice systemd-sleep[1242496]: Performing sleep operation 'suspend'... May 28 18:51:14 alice kernel: PM: suspend entry (deep) May 28 18:51:14 alice kernel: Filesystems sync: 0.048 seconds May 28 18:51:14 alice kernel: (NULL device *): firmware: direct-loading firmware regulatory.db ``` and then when trying to shut down, messages about "cannot stop frozen unit": ``` May 29 00:38:41 alice systemd-logind[1280]: Power key pressed short. May 29 00:38:41 alice systemd-logind[1280]: Powering off... May 29 00:38:41 alice systemd-logind[1280]: System is powering down. May 29 00:38:41 alice systemd[1]: Cannot stop frozen unit session-1.scope - Session 1 of User dkg. May 29 00:38:41 alice systemd[1]: Removed slice system-cron\x2dmail.slice - Slice /system/cron-mail. May 29 00:38:41 alice systemd[1]: Removed slice system-modprobe.slice - Slice /system/modprobe. May 29 00:38:41 alice systemd[1]: Stopped target blockdev@dev-mapper-sda4_crypt.target - Block Device Preparation for /dev/mapper/sda4_crypt. May 29 00:38:41 alice systemd[1]: Stopped target graphical.target - Graphical Interface. -- May 29 00:38:41 alice systemd[1]: tor.service: Deactivated successfully. May 29 00:38:41 alice systemd[1]: Stopped tor.service - Anonymizing overlay network for TCP (multi-instance-master). May 29 00:38:41 alice systemd[1]: Stopping tor@default.service - Anonymizing overlay network for TCP... May 29 00:38:41 alice systemd[1]: Stopping upower.service - Daemon for power management... May 29 00:38:41 alice systemd[1]: Cannot stop frozen unit user@1000.service - User Manager for UID 1000. May 29 00:38:41 alice systemd[1]: Cannot stop frozen unit user-runtime-dir@1000.service - User Runtime Directory /run/user/1000. May 29 00:38:41 alice systemd[1]: Cannot stop frozen unit user-1000.slice - User Slice of UID 1000. May 29 00:38:41 alice systemd[1]: run-credentials-systemd\x2dudev\x2dload\x2dcredentials.service.mount: Deactivated successfully. May 29 00:38:41 alice systemd[1]: atd.service: Deactivated successfully. May 29 00:38:41 alice postmulti[1242861]: postfix: Postfix is using backwards-compatible default settings May 29 00:38:41 alice postmulti[1242861]: postfix: See http://www.postfix.org/COMPATIBILITY_README.html for details -- May 29 00:38:46 alice systemd[1]: Stopped target paths.target - Path Units. May 29 00:38:46 alice systemd[1]: cron-update.path: Deactivated successfully. May 29 00:38:46 alice systemd[1]: Stopped cron-update.path - systemd-cron path monitor. May 29 00:38:46 alice systemd[1]: Stopped target slices.target - Slice Units. May 29 00:38:46 alice systemd[1]: Cannot stop frozen unit user.slice - User and Session Slice. May 29 00:38:46 alice systemd[1]: Stopped target sockets.target - Socket Units. May 29 00:38:46 alice systemd[1]: avahi-daemon.socket: Deactivated successfully. May 29 00:38:46 alice systemd[1]: Closed avahi-daemon.socket - Avahi mDNS/DNS-SD Stack Activation Socket. May 29 00:38:46 alice systemd[1]: dbus.socket: Deactivated successfully. ``` Eventually i hard-powered down the machine (power button for ~15 seconds). The user journal doesn't say anything about being frozen or woken back up -- it just sees normal session log action, and then a reboot. ### Steps to reproduce the problem I have not tried to replicate it in a dedicated VM or anything like that. I don't know whether there is anything special about what's happening my particular session that would make make it fail to freeze or be unfrozen. ### Additional program output to the terminal or log subsystem illustrating the issue _No response_

Mario_Limonciello · September 7, 2024, 1:17pm

I think this really boils down to systemd exposed a kernel bug. There will be tension for a while until the kernel is fixed or systemd withdraws this change.

David_Markey · September 7, 2024, 2:11pm

Do you think the workaround has any adverse effects?

David_Markey · September 7, 2024, 2:59pm

FYI also needed

[Service]
Environment="SYSTEMD_SLEEP_FREEZE_USER_SESSIONS=false"

in

/etc/systemd/system/systemd-hibernate.service.d/disable_freeze_user_session.conf

or hibernate has the same behavior.

David_Markey · September 7, 2024, 4:43pm

and


cp -rv /etc/systemd/system/systemd-suspend.service.d/ /etc/systemd/system/systemd-suspend-then-hibernate.service.d/

Annoying whackamole

Mario_Limonciello · September 9, 2024, 1:33am

On the systemd bug they say it should be fixed in 6.11. Can you upgrade to 6.11-rc7 and confirm?

David_Markey · September 9, 2024, 9:06am

Should get time later today UTC+1

David_Markey · September 9, 2024, 6:21pm

6.11.0-rc7 still has the problem(without the workaround)

Mario_Limonciello · September 9, 2024, 6:38pm

Can you share to the systemd ticket? They seem to think it would be fixed.

Szabi · November 8, 2024, 9:42am

The work around suggested by @David_Markey still works today
Just to summarize it in a single place:

# as root
mkdir /etc/systemd/system/systemd-suspend.service.d \
      /etc/systemd/system/systemd-hibernate.service.d \
      /etc/systemd/system/systemd-suspend-then-hibernate.service.d \
      /etc/systemd/system/systemd-hybrid-sleep.service.d

cat <<EOF > /etc/systemd/system/systemd-suspend.service.d/disable_freeze_user_session.conf
[Service]
Environment="SYSTEMD_SLEEP_FREEZE_USER_SESSIONS=false"
EOF

cp /etc/systemd/system/systemd-suspend.service.d/disable_freeze_user_session.conf /etc/systemd/system/systemd-hibernate.service.d/
cp /etc/systemd/system/systemd-suspend.service.d/disable_freeze_user_session.conf /etc/systemd/system/systemd-suspend-then-hibernate.service.d/
cp /etc/systemd/system/systemd-suspend.service.d/disable_freeze_user_session.conf /etc/systemd/system/systemd-hybrid-sleep.service.d/

This is a very annoying bug, that has plagued me for a long time, at times the suspend worked and at times it would freeze my system. I’m running VMs in Qemu using gnome-boxes and by the time I’ve noticed that that VM has something to do with the freezes… Anyway thanks to this thread I managed to apply the workaround and can now again suspend and hibernate my system. Thank you!

David_Markey · November 8, 2024, 1:37pm

You’re welcome.