I also see this problem sporadically after resuming from a suspend. Suspending again, and re-resuming often fixes it. This is on Ubuntu 22.04.2 LTS.
For Ubuntu 22.04 and up, please try this:
echo "blacklist cros_ec_lpcs" | sudo tee -a /etc/modprobe.d/no_cros_ec.conf
Reboot.
Then report back. It blacklists this: LKML: "Dustin L. Howett": [PATCH 0/2] platform/chrome: Add support for the Framework Laptop Won’t affect anything in a negative way, but it going to likely solve the issue for you on 11th gen.
2 Likes
By the way, I just want to precise that this happens on 12th gen.
I didn’t know that this thread was 11th gen only:
I switched from 11th-gen to 12th-gen some time last year, and the issue still happens from time to time.
Looks to be different. 11th gen users were seeing this happen with some suspend/resume activity and it looks like your seeing this with Fn lock after unplugging a TB dock.
@Matt_Hartley thanks for the tip about blacklisting the cros_ec_lpcs module, I’m trying that now. It still seems to get loaded even with the modprobe.d file:
chris@vega:~ ☸ home in 112ms @ 20:34:32 $ cat /etc/modprobe.d/no_cros_ec.conf
# Blacklists a kernel module that causes the Fn key to stop working
# https://community.frame.work/t/tracking-fn-key-stops-working-on-popos-after-a-while/21208/32
blacklist cros_ec_lpcs
chris@vega:~ ☸ home in 109ms @ 20:35:42 $ lsmod | grep cros_
cros_usbpd_charger 20480 0
cros_usbpd_logger 20480 0
cros_usbpd_notify 20480 1 cros_usbpd_charger
cros_ec_debugfs 16384 0
cros_ec_chardev 16384 0
cros_ec_sysfs 16384 0
cros_ec_dev 16384 0
cros_ec_lpcs 16384 0
cros_ec 20480 1 cros_ec_lpcs
Do you have any advice on a stronger way to blacklist it?
No not always, I had it the last week-end, and it was not after touching the docks nor the ports (I think… 90% sure…).
Next time it happens, I’ll make sure to memo it and report back here.
1 Like
Ah update on the modprobe.d solution: after adding a modprobe configuration file, you also have to run sudo update-initramfs -u to make it take effect on the next boot. I’m now running without those cros_ modules and will report back after a week or so.
3 Likes
I’m experiencing this error on 11th and 12th gen. I created this topic when I was already at 12th as far as I remember.
So far I didn’t find a solution as well, however the error happens not as often as before within the last few weeks.
1 Like
Was suspend involved and which distro/desktop? I have to date, only seen this on 11th gen and even at that have not reproduced the issue.
Ok so now I just had the problem (on the 12-th gen). It was not after manipulating the TB dock.
The good news: I discovered that I just had to do a s2ram and then when waking it back up the problem was gone! So, no more need to do a full reboot! (and no need to loose the state of the laptop, no more need to re-open all the files etc.)
I’m having the same problem with my 12th gen on Ubuntu 23.04.
It only happens to me after waking from sleep, and only every 5th time or so. Putting it back to sleep does not fix it for me, only a reboot fixes it.
The Fn functions stop working, only F1-F12 works, no matter if a hold down the Fn key or not. Fn lock also doesn’t work. Fn+space bar for the keyboard backlight also stops working. The keyboard backlight stays on the level it was before going to sleep.
I usually have nothing connected to the laptop except for the charger when putting it to sleep by closing the lid. I have two USB-C modules, one USB-A and one HDMI, if that makes any difference.
I got my Framework in March '23 and it’s been running Ubuntu from the start. I don’t think I’ve noticed this problem in the beginning. It seems to me it only started recently.
1 Like
Have you tried s2ram specifically? There are several flavors of “sleep”, so maybe it depends on which you use…
Reporting back in after 2 weeks and I’m pretty sure blocking cros_ec_lpcs as @Matt_Hartley suggested here ([TRACKING] Fn key stops working on PopOS after a while - #32 by Matt_Hartley) has completely solved the problem for me!
Just be sure to run update-initramfs -u afterwards to ensure it takes.
5 Likes
Fantastic! Glad to hear it.
As I understand it, that cros_ec_lpcs is expected to fix things only on the 11th gen, is that right? Are there any similar workarounds for the same issue on 12th gen?
On the support ticket that I have open for this issue, support have asked me to try this workaround on my 12th gen, which I’m currently doing. It’s a bit too early to tell for sure, but so far it looks very promising - no re-occurrences yet, no more dmesg errors from the module, and I haven’t noticed any other ill effects. The main downside is that blacklisting the module means no fan reporting/control via ectool (though it might be possible by allowing ectool to do raw port IO?). Overall I think it’s well worth attempting the workaround on 12th gen and seeing how you go with it.
EDIT: I also cheated and did an rmmod of the module, without actually rebooting or blacklisting (yet). The rmmod apparently worked despite throwing an error. YMMV.
1 Like
Hi, new owner of a 13th Gen (intel) with Ubuntu 22.04. On wake after a night in sleep mode, the fn key doesn’t work at all. Undetected. fn+esc does not switch state, and all the keys available in fn mode don’t work.
I tried the 11th gen fix, no success. Will reboot and report back.
Update: fixed after reboot. Will follow up if it restarts after a while in sleep mode as this seemed to be the trigger: I had small periods of sleep mode (<30 min) yesterday with no issues. Seems it’s the “long” sleep that triggered this.
I have not had this reoccur. However, my laptop crashed (unrelated) after ~25 days of uptime. But prior to that I had used it several times for numerous hours in situations that were previously highly likely to cause the problem (unplugged and moving around indoors).
So I think it’s reasonably safe to conclude that the problem is indeed caused by something the cros_ec_lpcs module (possibly related to all the dmesg warnings/errors that it also emits).
However, I only consider blacklisting the module to be at most a workaround. Without the module it’s not possible to monitor the fan speed from userspace, or do other ectool-related actions. (Even if giving ectool raw port IO access works, that seems like a risky approach that could potentially cause even more problems.)
The module ought to be able to work, and IIRC there were some Framework-specific patches to it, which I think may possibly be where the problem is. So now that the likely cause has been more-or-less narrowed down, I’d really value if a Framework engineer could perhaps have a closer look into what exactly is going wrong with these EC-host comms.
(Also, my use case for wanting real time fan-speed monitoring is that I often use the laptop with headphones, which means I can’t hear if the fans spin up loudly, ie. I don’t notice if something is causing heavy cooling. This can sometimes happen without generating very high system load averages, eg. there seems to be a minor bug that sometimes causes interrupt storms to/from snd_hda_intel. The fan monitoring allows me to visually notice when the fan is working hard, and then if that’s unexpected, look into the problem.)
[With the appropriate disclaimer that I’m not an engineer at Framework Computer,] we now know what’s going on thanks to all the reports here and the case you’ve built up! 
The cros_ec_lpcs driver is generic for any laptop that has a ChromeOS EC on the LPC bus. The patches to add support for the Framework Laptop really just add a device identifier and fix port allocation, but don’t themselves cause the issue.
At the end of the day, it’s the same root cause as this equivalent issue filed by Kieran in the (my) CrosEC Windows driver: EC access is sometimes corrupted. · Issue #3 · DHowett/FrameworkWindowsUtils · GitHub; it will likely also reproduce with coolstar’s crosecbus driver.
The power and battery state of the machine are managed by ACPI, and the ACPI methods for querying those things call the EC directly³. When it does so, it uses a mutex that can’t be shared with the OS². There’s also a couple ACPI-driven exchanges that occur during wake from sleep. Now, because cros_ec_lpcs (Linux) and CrosEC (Windows) use the LPC bus directly, an inflight request from ACPI can collide with an inflight request from one of these drivers.
Since the cros_ec_debugfs driver (not _lpcs, mind!) seems to query the EC console repeatedly to surface it via the debugfs interface, it causes a lot of traffic–especially around system startup–that runs a chance of stomping the one ACPI exchange that clears the preOS bit⁴.
Letting ectool do raw port I/O will “fix” it only because it reduces the incidence of host command exchanges. If you run it in a tight loop starting from the moment the machine wakes, you’ll still encounter some corruption of inflight packets.
I wonder… if you put cros_ec_debugfs on the disallowlist instead of cros_ec_lpcs, does it do anything for this issue⁵? (@Matt_Hartley, I would love if you had some cycles spare to help figure out with the community if ..._debugfs is an effective workaround; if so, people could still use ectool!)
¹ I’m comfortable saying “we” only because I’m the person who caused this issue 
² There’s another generic method (FWMI) that would allow for the OS to communicate with the EC via ACPI instead of using the I/O ports directly, but using it would need a solid chunk of driver work.
³ Beware, this file is huge. DSDT from 11th gen v3.17 > EC0.M001
⁴ The one that you noted earlier and is used to determine whether to ignore/respect Fn
⁵ This might help all users, minus the subset of people who really are using ectool during early boot. It would be a more hit-or-miss fix for those folks. It will unequivocally reduce host command traffic!
8 Likes
@DHowett Splendid diagnostic work!!! Really really appreciate!
Why do you say we cannot use ectool when blacklisting cros_ec_lpcs?
I have it blacklisted but I still can e.g. set the battery charge limit using ectool.