[TRACKING] Fedora 35 kernel 5.16.5 s2idle and maybe wifi issues

Anil_Kulkarni · February 11, 2022, 12:18pm

Yeah and holy smokes does this take a long time!

Adam_Strauch · February 11, 2022, 1:46pm

I am experiencing the same problem and same time I am a little bit concerned about Fedora releasing kernel where this kind of bugs is reported. It’s not just our laptop.

@Anil_Kulkarni Thanks for your time trying to bisect it!

junaruga · February 11, 2022, 4:43pm

Nice. So, could you create a small reproducing script to fail on the kernel 5.16.5, and pass the kernel 5.15? The test case is not included in the unit tests in kernel? I am curious to know which git repo in https://git.kernel.org/ is related to this issue.

Anil_Kulkarni · February 12, 2022, 12:21am

Somewhat shockingly… the bisect points to: ACPICA: Add support for Windows 2020 _OSI string

And indeed, reverting this patch on-top of 5.16 fixes backlight at least. @ dimitris, if you wanted to try compiling a kernel with git revert 3bf70bd2538f0515ce17b1c067889ff0e4fec842 and see if it fixes your issues.

Since it’s the EOW, I’ll wait until Monday to create a bugzilla report over this

dimitris · February 12, 2022, 3:04am

@Anil_Kulkarni nice work! I’ll try to build a Fedora kernel from source with that revert somehow included, but it’s something I’ve never done before so it may take me a while to get it done.

FWIW I tried to quickly play with the acpi_osi= kernel command line parameter (Linux, Windows, Windows 2019) but it didn’t make a difference.

I also reproduced the difference in behavior with lid vs power button: The lid does seem to put the laptop into s2idle sleep successfully. I still would prefer to use the power button primarily because there isn’t (yet) a way to visually confirm the laptop has in fact entered sleep. I believe this has been discussed in the EC thread(s) re: modifying LED behavior.

@Kieran_Levin this combination of “behavior deltas” (kernel adding a “reported capability” to ACPI and difference between power button vs lid sensor) point to some underlying EC/BIOS issue?

dimitris · February 12, 2022, 4:12am

Update: Apparently it’s possible to tell the kernel to mask _OSI entries; no git revert needed.

After adding "acpi_osi=!Windows 2020" to the kernel cmdline args:

sudo grubby --update-kernel=vmlinuz-5.16.8-200.fc35.x86_64 --args='"acpi_osi=!Windows 2020"'

(edit: or sudo grubby --update-kernel=ALL --args='"acpi_osi=!Windows 2020"' to make it the default going forward)

and rebooting, I can now get into s2idle sleep with the power button. So it seems that telling the Insyde BIOS (3.07 anyway) that we “support/are” Windows 2020 makes it do Bad Things.

Background:
kernel docs: ACPI _OSI and _REV methods — The Linux Kernel documentation
and this for the “negate entry” hint: Linux: ACPI: Fix problems with Suspend, Resume, and Missing devices using acpi_osi=

dimitris · February 12, 2022, 4:32am

And one quick followup on the wifi issues: Those do persist, so they don’t seem to be a side effect of S3 sleep but rather an actual regression with the kernel/Intel wireless driver. I believe I’ve seen something in a thread here about that being worked on in 5.17 or 5.18, but I’m not sure… (edit: 5.17)

Anyway, the wireless network still does come back after a momentary/mildly annying freeze on resume.

Anil_Kulkarni · February 12, 2022, 5:43am

Ah thanks for digging in - this makes more sense. Just from reading the patch I was very confused how Windows was affecting standby. But now it makes sense because it’s telling the bios “Yes, we’re windows 2020”

Here’s the key snippet:

Linux had no choice but to also return TRUE to _OSI(“Windows 2001”) and its successors. To do otherwise would virtually guarantee breaking a BIOS that has been tested only with that _OSI returning TRUE.
This strategy is problematic, as Linux is never completely compatible with the latest version of Windows, and sometimes it takes more than a year to iron out incompatibilities.

so I take it we’ll need yet-another-bios update to fix this eventually. In the mean time we probably should get those kernel args added to the linux guides on this site

D.H · February 14, 2022, 5:26pm

Hi @dimitris

I had a bunch of other things to focus on but I finally can devote a few minutes to helping troubleshoot this. In EndeavourOS (arch based), with 5.16.8:

Seeing the same keyboard backlight remaining on behavior for s2idle. It does take a power button push to wake it up, not a keyboard keypress, so I know it made it to some kind of “sleep”. I prefer deep by far though, so I’m going back to that.
No issues resuming wifi from s2idle or from deep
None of your iwlwifi errors show up in my dmesg.

Here’s dmesg at boot:

$ sudo dmesg | grep -i wifi
[    3.469737] Intel(R) Wireless WiFi driver for Linux
[    3.470023] iwlwifi 0000:aa:00.0: enabling device (0000 -> 0002)
[    3.512379] iwlwifi 0000:aa:00.0: api flags index 2 larger than supported by driver
[    3.512396] iwlwifi 0000:aa:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 0.0.2.34
[    3.512635] iwlwifi 0000:aa:00.0: loaded firmware version 67.8f59b80b.0 ty-a0-gf-a0-67.ucode op_mode iwlmvm
[    3.654393] iwlwifi 0000:aa:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[    3.660897] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 1, ret=-1
[    3.660900] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 2, ret=-1
[    3.660901] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 3, ret=-1
[    3.828038] iwlwifi 0000:aa:00.0: loaded PNVM version dda57f4f
[    3.843264] iwlwifi 0000:aa:00.0: Detected RF GF, rfid=0x10d000
[    3.912446] iwlwifi 0000:aa:00.0: base HW address: f4:46:37:ca:8b:bd
[    4.004239] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 1, ret=-1
[    4.004242] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 2, ret=-1
[    4.004243] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 3, ret=-1
[    4.297471] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 1, ret=-1
[    4.297478] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 2, ret=-1
[    4.297480] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 3, ret=-1

and I get this after every resume from either s2idle or from deep:

$ sudo dmesg | grep -i wifi
[ 3493.683355] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 1, ret=-1
[ 3493.683362] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 2, ret=-1
[ 3493.683364] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 3, ret=-1

cannot reproduce your usb issues / messages. Is there a particular device that inserting/removing changes your behavior? Have you checked internall for EMI sticker issues, melted covers, etc?

dimitris · February 14, 2022, 9:31pm

@D.H I’ve checked inside the laptop, no melt/overhead issues, and the EMI stickers seem ok (it’s a batch 3 build, the KB article implies this was an issue only in batches 1 and 2).

About the iwlwifi stack traces: I’ve opened a fedora issue with pretty much the same issue, separate from the now worked-around s2idle one.

The Intel wireless firmware version in your dmesg is the same I have. This could be a difference in the patches that EndeavourOS carries in the kernel vs Fedora, or even the particular wireless network. Since it doesn’t reproduce 100% for me, it’s worth keeping an eye on it. In my case it’s roughly one in three attempts.

Roberto_Vernina · February 15, 2022, 5:35am

In an ideal world, it should be the company itself submitting patches/updates Upstream for its own hardware. I guess, but I may be wrong, that in this case the regression is due to Intel rather than to Framework, and that can be proved by replacing the wi-fi card with something else (not Intel, possibly) and see how it goes.
As per the USB error logs, it might as well be an hardware issue (possibly in the expansion card), but I hope I’m wrong.

D.H · February 18, 2022, 3:09pm

The plot sickens. It seems plasma shell is unable to make the laptop sleep in s2idle nor in deep mode lately. Suspending via systemctl call works, but not clicking “sleep” in the GUI, closing the lid, etc.

Hibernate works from GUI and from command line as expected.

Anybody else seeing similar?

5.16.8 in arch derivative, not Fedora, but still…

Adam_Strauch · February 18, 2022, 3:35pm

Suspend is working for me with the acpi_osi workaround but while I am cycling suspend/resume then sooner or later something happens to Pipewire or something underlying, all sound devices vanish from the system and sound related processes including pipewrite block shutting the machine down. I cannot even kill them with SIGKILL. It could be related to my USB-C dock.

EDIT: It’s probably this bug:
https://patchwork.kernel.org/project/alsa-devel/patch/20220214125711.20531-1-tiwai@suse.de/
and it should be fixed in the maintree soon.

dimitris · February 18, 2022, 4:55pm

@Adam_Strauch the USB audio issues sound like what I encountered with an AMD Thinkpad: 2051038 – Thinkpad T495 fails to enter S3 sleep after first sleep/resume; audio tasks refusing to freeze - unrelated issue. BTW in latest 5.16 kernels, at least on Fedora the behavior morphed to what you describe. Sleep happens, but on resume audio is messed up. I’ve blacklisted the USB audio module on that machine for now.

Glad to see it’s already being dealt with upstream.

Adam_Strauch · February 18, 2022, 5:15pm

@dimitris Thanks, I blacklisted the module and when I reboot I will test it with the latest Fedora kernel. It would be nice to stop drifting from the upstream.

junaruga · February 19, 2022, 10:25am

I reported the upstream patch here.
https://bugzilla.redhat.com/show_bug.cgi?id=2050036#c11

TARDIS42 · February 21, 2022, 5:03am

Got my Framework Laptop last Thursday and installed Fedora 35 (5.16.9-200.fc35.x86_64 kernel). Everything is working well… except for the Wifi card (AX210) which is connecting to my home WiFi network (on the latest generation eero AP’s with WiFi6 support).

The main issues are:

It doesn’t reattach to the network after any sustained period (more than a hour) of suspend state (laptop closed)
Randomly craps out, sometimes in short succession, sometimes it then runs fine for 8+ hours.

It’s somewhat similar to Fedora 34: Wifi card crashing randomly - #13 by Andrew_Marshall but that was Fedora 34 instead of 35. In both cases, turning WiFi off and on successfully re-attaches to the network and all is well again.

Here are the relevent kernel messages:

[84883.920623] iwlwifi 0000:00:14.3: Unhandled alg: 0x71b
[85798.920967] iwlwifi 0000:00:14.3: Unhandled alg: 0x71b
[86107.927391] iwlwifi 0000:00:14.3: Unhandled alg: 0x71b
[88561.509854] iwlwifi 0000:00:14.3: Microcode SW error detected. Restarting 0x0.
[88561.510770] iwlwifi 0000:00:14.3: Start IWL Error Log Dump:
[88561.510773] iwlwifi 0000:00:14.3: Transport status: 0x0000004A, valid: 6
[88561.510777] iwlwifi 0000:00:14.3: Loaded firmware version: 67.8f59b80b.0 QuZ-a0-hr-b0-67.ucode
[88561.510781] iwlwifi 0000:00:14.3: 0x00000071 | NMI_INTERRUPT_UMAC_FATAL    
[88561.510785] iwlwifi 0000:00:14.3: 0x0000A210 | trm_hw_status0
[88561.510787] iwlwifi 0000:00:14.3: 0x00000000 | trm_hw_status1
[88561.510790] iwlwifi 0000:00:14.3: 0x004CB2DE | branchlink2
[88561.510792] iwlwifi 0000:00:14.3: 0x004C1A5A | interruptlink1
[88561.510795] iwlwifi 0000:00:14.3: 0x004C1A5A | interruptlink2
[88561.510797] iwlwifi 0000:00:14.3: 0x0000B2C8 | data1
[88561.510799] iwlwifi 0000:00:14.3: 0x00001000 | data2
[88561.510801] iwlwifi 0000:00:14.3: 0x00000000 | data3
[88561.510803] iwlwifi 0000:00:14.3: 0xE0002D36 | beacon time
[88561.510806] iwlwifi 0000:00:14.3: 0x0E723725 | tsf low
[88561.510808] iwlwifi 0000:00:14.3: 0x000002BD | tsf hi
[88561.510810] iwlwifi 0000:00:14.3: 0x00000000 | time gp1
[88561.510812] iwlwifi 0000:00:14.3: 0x000590CA | time gp2
[88561.510815] iwlwifi 0000:00:14.3: 0x00000001 | uCode revision type
[88561.510817] iwlwifi 0000:00:14.3: 0x00000043 | uCode version major
[88561.510819] iwlwifi 0000:00:14.3: 0x8F59B80B | uCode version minor
[88561.510822] iwlwifi 0000:00:14.3: 0x00000351 | hw version
[88561.510824] iwlwifi 0000:00:14.3: 0x00C89004 | board version
[88561.510826] iwlwifi 0000:00:14.3: 0x80BBFC03 | hcmd
[88561.510828] iwlwifi 0000:00:14.3: 0x24020000 | isr0
[88561.510830] iwlwifi 0000:00:14.3: 0x00000000 | isr1
[88561.510833] iwlwifi 0000:00:14.3: 0x08F00002 | isr2
[88561.510835] iwlwifi 0000:00:14.3: 0x00C36C0C | isr3
[88561.510837] iwlwifi 0000:00:14.3: 0x00000000 | isr4
[88561.510839] iwlwifi 0000:00:14.3: 0x04AA001C | last cmd Id
[88561.510841] iwlwifi 0000:00:14.3: 0x0000B2C8 | wait_event
[88561.510843] iwlwifi 0000:00:14.3: 0x00000080 | l2p_control
[88561.510846] iwlwifi 0000:00:14.3: 0x00010034 | l2p_duration
[88561.510848] iwlwifi 0000:00:14.3: 0x0000003F | l2p_mhvalid
[88561.510850] iwlwifi 0000:00:14.3: 0x00000080 | l2p_addr_match
[88561.510852] iwlwifi 0000:00:14.3: 0x00000009 | lmpm_pmg_sel
[88561.510854] iwlwifi 0000:00:14.3: 0x00000000 | timestamp
[88561.510856] iwlwifi 0000:00:14.3: 0x00004030 | flow_handler
[88561.511319] iwlwifi 0000:00:14.3: Start IWL Error Log Dump:
[88561.511321] iwlwifi 0000:00:14.3: Transport status: 0x0000004A, valid: 7
[88561.511324] iwlwifi 0000:00:14.3: 0x20003463 | ADVANCED_SYSASSERT
[88561.511327] iwlwifi 0000:00:14.3: 0x00000000 | umac branchlink1
[88561.511330] iwlwifi 0000:00:14.3: 0x80455E52 | umac branchlink2
[88561.511332] iwlwifi 0000:00:14.3: 0xC00811A4 | umac interruptlink1
[88561.511334] iwlwifi 0000:00:14.3: 0x00000000 | umac interruptlink2
[88561.511336] iwlwifi 0000:00:14.3: 0x0E723718 | umac data1
[88561.511338] iwlwifi 0000:00:14.3: 0x000590BB | umac data2
[88561.511340] iwlwifi 0000:00:14.3: 0x0F98C813 | umac data3
[88561.511343] iwlwifi 0000:00:14.3: 0x00000043 | umac major
[88561.511345] iwlwifi 0000:00:14.3: 0x8F59B80B | umac minor
[88561.511347] iwlwifi 0000:00:14.3: 0x000590C4 | frame pointer
[88561.511349] iwlwifi 0000:00:14.3: 0xC0885E00 | stack pointer
[88561.511351] iwlwifi 0000:00:14.3: 0x00B7010D | last host cmd
[88561.511353] iwlwifi 0000:00:14.3: 0x00000000 | isr status reg
[88561.511514] iwlwifi 0000:00:14.3: IML/ROM dump:
[88561.511516] iwlwifi 0000:00:14.3: 0x00000003 | IML/ROM error/state
[88561.511602] iwlwifi 0000:00:14.3: 0x00005DDF | IML/ROM data1
[88561.511690] iwlwifi 0000:00:14.3: 0x00000080 | IML/ROM WFPM_AUTH_KEY_0
[88561.511707] iwlwifi 0000:00:14.3: Fseq Registers:
[88561.511745] iwlwifi 0000:00:14.3: 0x60000000 | FSEQ_ERROR_CODE
[88561.511778] iwlwifi 0000:00:14.3: 0x80290033 | FSEQ_TOP_INIT_VERSION
[88561.511811] iwlwifi 0000:00:14.3: 0x00090006 | FSEQ_CNVIO_INIT_VERSION
[88561.511844] iwlwifi 0000:00:14.3: 0x0000A482 | FSEQ_OTP_VERSION
[88561.511881] iwlwifi 0000:00:14.3: 0x00000003 | FSEQ_TOP_CONTENT_VERSION
[88561.511915] iwlwifi 0000:00:14.3: 0x4552414E | FSEQ_ALIVE_TOKEN
[88561.511956] iwlwifi 0000:00:14.3: 0x20000302 | FSEQ_CNVI_ID
[88561.511996] iwlwifi 0000:00:14.3: 0x01300504 | FSEQ_CNVR_ID
[88561.512033] iwlwifi 0000:00:14.3: 0x20000302 | CNVI_AUX_MISC_CHIP
[88561.512072] iwlwifi 0000:00:14.3: 0x01300504 | CNVR_AUX_MISC_CHIP
[88561.512112] iwlwifi 0000:00:14.3: 0x05B0905B | CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM
[88561.512156] iwlwifi 0000:00:14.3: 0x0000025B | CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR

Firmware version:

[ 18.164769] iwlwifi 0000:00:14.3: loaded firmware version 67.8f59b80b.0 QuZ-a0-hr-b0-67.ucode op_mode iwlmvm

Any ideas on any fixes? Thanks in advance.

Senhara · February 21, 2022, 7:41am

Interesting. I don’t have these issues myself, openSUSE Tumbleweed.

I’ll post my logs in case if you find it useful.

[    5.081334] iwlwifi 0000:aa:00.0: enabling device (0000 -> 0002)
[    5.127504] iwlwifi 0000:aa:00.0: api flags index 2 larger than supported by driver
[    5.127523] iwlwifi 0000:aa:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 0.0.2.34
[    5.127710] iwlwifi 0000:aa:00.0: loaded firmware version 67.8f59b80b.0 ty-a0-gf-a0-67.ucode op_mode iwlmvm
[    5.281998] iwlwifi 0000:aa:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[    5.297832] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 1, ret=-1
[    5.297836] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 2, ret=-1
[    5.297837] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 3, ret=-1
[    5.463635] iwlwifi 0000:aa:00.0: loaded PNVM version dda57f4f
[    5.478965] iwlwifi 0000:aa:00.0: Detected RF GF, rfid=0x10d000
[    5.548130] iwlwifi 0000:aa:00.0: base HW address: (OMITTED)
[    5.885410] iwlwifi 0000:aa:00.0 wlp170s0: renamed from wlan0
[    5.956659] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 1, ret=-1
[    5.956669] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 2, ret=-1
[    5.956671] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 3, ret=-1
[    6.244642] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 1, ret=-1
[    6.244647] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 2, ret=-1
[    6.244647] iwlwifi 0000:aa:00.0: WRT: Failed to set DRAM buffer for alloc id 3, ret=-1

Kernel is 5.16.8-1-default

Spotted something, our ucodes are different. Perhaps your ucode version is breaking things?

dimitris · February 23, 2022, 2:52am

@junaruga I think you meant to report that USB audio issue to the fedora bug titled “Thinkpad …”? That’s the one about USB audio. The one about framework is not related to USB audio.

junaruga · February 23, 2022, 7:00am

@dimitris I see the ticket “Thinkpad …” is one written at 2050036 – Framework laptop: 5.16.5 breaks s2idle sleep . Sorry I might post my comment without understanding the issue. I see you posted the upstream patch at 2051038 – Thinkpad T495 fails to enter S3 sleep after first sleep/resume; audio tasks refusing to freeze .