Function (Fn) Keys Sticking + Fix for Linux

Michael_Wu · October 29, 2021, 8:26pm

Seems like BIOS 3.09 has fixed the issue

For your convenience, here’s how to uninstall this post’s fix now that it’s no longer needed:

To uninstall the udev rules see here.

To uninstall the custom driver:

Below is the original post for archival purposes/if needed, etc.

Thanks @carlos1001 for bringing this to the spotlight and the initial thread. I’ve had this issue since day 1/batch 1 but thought it was a bug in Sway until re-investigating. If e.g. your cursor keeps moving, page keeps jumping/scrolling up, volume up keeps increasing, etc. this is probably why. It’s not until that key/combo is pressed again, (I believe twice) that it releases.

Thankfully, this isn’t a physical/hardware issue

I did a thorough dive into the issue so I thought I’d create a new thread detailing the problem, my fixes, and hopefully get @Framework’s idea on how to proceed with this. I’m fairly certain this should be fixable in firmware, without needing to apply these fixes. If not, there’s always the less ideal possibility of pushing to systemd/udev and/or the Linux kernel.

Issue: Any key that has a `Function (Fn)` combo can get stuck except for `Airplane Mode (F10)`

This issue affects Linux, though it can be fixed by adding a synthetic force release event, where then it behaves similarly to Windows.

The Brightness Down/Up (F7/F8) keys, however, seem to go through a separate input, and can stick on both Linux and Windows.

The Airplane Mode (F10) key seems to go through the same input as the Brightness keys, but has no sticking issue to begin with.

Straight to the repo/fix

https://github.com/miXwui/framework-laptop-fn-keys-fix/

Or in this post scroll down to the Quick Fix for Most Keys section, and the Brightness Keys Fix section.

To reproduce

Terms used:

Function lock on = F1 functions as F1. Hold Fn + F1 for Mute.
Function lock off = press F1 for Mute, no need to hold Fn.
Fn + Esc (fn lock) toggles function lock .

Primary function = on F1 key: F1; on Left Arrow: Left Arrow; on Delete: Delete, etc.
Secondary function = on F1 key: Mute; on Left Arrow: Home; on Delete: Insert, etc.

`F1`-`F12` primary / secondary functions sticking

When function lock is on

Hold Fn
Hold one of the F1-F12
Release Fn
Release the key held in (2.)

The secondary function of the key held in (2.) will stick, repeating e.g. Mute (for F1).

Hold one of the F1-F12
Hold Fn
Release Fn
Release the key held in (1.)

The primary function of the key held in (1.) will stick, repeating e.g. F1.

When function lock is off

Hold Fn
Hold one of the F1-F12 keys
Release Fn
Release the key held in (2.)

The primary function of the key held in (2.) will stick, repeating e.g. F1.

Hold one of the F1-F12 keys
Hold Fn
Release Fn
Release the key held in (1.)

The secondary function of the key held in (1.) will stick, repeating e.g. Mute (for F1).

`Delete` / (`Insert`) key sticking

Same behavior with function lock off or on:

Hold `Fn
Hold Delete
Release Fn
Release Delete

Insert will stick.

Hold Delete
Hold Fn
Release Fn
Release Delete

Delete will stick.

Arrow keys primary / secondary function sticking

Same behavior with function lock off or on:

Hold one of the Arrow keys
Hold Fn
Release Fn
Release the Arrow key held in (1.)

The Arrow key will stuck.

Hold Fn
Hold one of the Arrow keys
Release Fn
Release the Arrow key held in (2.)

The secondary function of the Arrow key will stick (Home/End/PageDown/PageUp).

Quick Fix for Most Keys

The quickest and easiest current fix that should satisfy most people (this forces a synthetic release event in systemd/udev):

Create the file /etc/udev/hwdb.d/70-keyboard-framework-fix.hwdb

# Fix for Fn keys sticking on Framework laptop
evdev:atkbd:dmi:bvn*:bvr*:bd*:svnFramework:pnLaptop:pvr*:*
evdev:input:b0011v0001p0001*
# Primary function fixes
 KEYBOARD_KEY_3b=!F1           # F1
 KEYBOARD_KEY_3c=!F2           # F2
 KEYBOARD_KEY_3d=!F3           # F3
 KEYBOARD_KEY_3e=!F4           # F4
 KEYBOARD_KEY_3f=!F5           # F5
 KEYBOARD_KEY_40=!F6           # F6
 KEYBOARD_KEY_41=!F7           # F7
 KEYBOARD_KEY_42=!F8           # F8
 KEYBOARD_KEY_43=!F9           # F9
 KEYBOARD_KEY_44=!F10          # F10
 KEYBOARD_KEY_57=!F11          # F11
 KEYBOARD_KEY_58=!F12          # F12
 KEYBOARD_KEY_d3=!delete       # Delete key
# Secondary function fixes
 KEYBOARD_KEY_a0=!mute         # On F1 key
 KEYBOARD_KEY_ae=!volumedown   # On F2 key
 KEYBOARD_KEY_b0=!volumeup     # On F3 key
 KEYBOARD_KEY_90=!previoussong # On F4 key
 KEYBOARD_KEY_a2=!playpause    # On F5 key
 KEYBOARD_KEY_99=!nextsong     # On F6 key
 KEYBOARD_KEY_b7=!sysrq        # On F11 key
 KEYBOARD_KEY_ed=!media        # On F12 key
 KEYBOARD_KEY_d2=!insert       # On Delete key
# Arrow keys
 KEYBOARD_KEY_cb=!left         # Left Arrow key
 KEYBOARD_KEY_cd=!right        # Right Arrow key
 KEYBOARD_KEY_d0=!down         # Down Arrow
 KEYBOARD_KEY_c8=!up           # Up Arrow
 KEYBOARD_KEY_c7=!home         # On Left Arrow key
 KEYBOARD_KEY_cf=!end          # On Right Arrow key
 KEYBOARD_KEY_d1=!pagedown     # On Down Arrow key
 KEYBOARD_KEY_c9=!pageup       # On Up Arrow key

Run
```
sudo systemd-hwdb update`
sudo udevadm trigger --verbose --sysname-match="event*"`
```
(This may need to be run more than once, due to a possible bug, to actually change.)
Edit: may also need to restart your computer

Verify changes with: udevadm info /sys/class/input/event2

E: KEYBOARD_KEY_3b=!F1
E: KEYBOARD_KEY_3c=!F2
E: KEYBOARD_KEY_3d=!F3
...

More info here:
https://github.com/miXwui/framework-laptop-fn-keys-fix/

And original post with explanation on the bottom for anyone interested.

`Projector` key (`F9`'s secondary function`)

This seems to trigger the letter KEY_P + KEY_LEFTMETA (P + Windows Key). I haven’t figured out how to force a synthetic release event on Linux, so unfortunately this isn’t fixed.

KEYBOARD_KEY_19=!p
KEYBOARD_KEY_db=!leftmeta

Didn’t work, it would just disable the LeftMeta/Windows Key. Maybe someone can figure this out!

`Brightness Down` / `Up` (`F7`'s / `F8`'s secondary function) keys:

The BrightnessDown and BrightnessUp key, seem like they can’t be fixed like in the way above since they seem to go through a different input than the rest of the keyboard.

This happens on both Linux and Windows.

When function lock is on:

Hold Fn
Hold F7 or F8
Release Fn
Release key held in (2.)

Brightness Down (F7) or Brightness Up (F8) will stick, continuing to decrease/increase brightness.

When function lock is off:

Hold F7 or F8
Hold Fn
Release Fn
Release the key held in (1.)

Brightness Down (F7) or Brightness Up (F8) will stick, continuing to decrease/increase brightness.

Some debugging notes on Linux

It seems like because it’s going through i2c and isn’t a serial keyboard? So it doesn’t send a release signal if keys are pressed in the specific orders specified above.

It seems like it’s because only the atkbd driver supports force_release of keys via !.

If we add udev.log_level=debug to our kernel parameters, and during loading of the drivers for “Brightness Down / Brightness Up / Airplane Mode” controller? This error message appears: Failed to get serio parent: No such file or directory which is from https://github.com/systemd/systemd/blob/42ffc40ce382dfda5f2a100698673bc252a72194/src/udev/udev-builtin-keyboard.c#L34 where install_force_release fails. Seems like there’s a force_release attribute for the atkbd/serio driver? I’m not sure exactly how it works. May also be an edge case bug in systemd/udev, not sure.

This controller doesn’t seem to use that, I think it’s cause it’s an I2C_HID_ACPI device thing (the embedded controller/EC?), not a keyboard or doesn’t run over serio or something something I’m not sure.

So it seems like udev additions to generate a synthetic release doesn’t work, although reassigning a brightness key to e.g. mute does work. This makes sense, I guess. Keys can be reassigned, but my assumption is since there is no force_release option, the ! in front of !brightnessdown doesn’t do anything. I tried force using the atkbd/serio driver, but gave up.

On Linux, FRMW0001:00 32AC:0006 Consumer Control / /devices/pci0000:00/0000:00:15.1/i2c_designware.1/i2c-1/i2c-FRMW0001:00/0018:32AC:0006.0002/ uses the hid-generic driver. I wrote a custom quirks driver that intercepts the raw HID events and add a key release right after key down. This prevents keys from being repeated, but unfortunately also loses the long-press ability to keep decreasing/increasing brightness. Here’s the code (this is the first time I’ve written anything in C, so take that with a bucket of salt). Thankfully this is fairly simple, here’s the code:

Brightness Keys Fix

https://github.com/miXwui/framework-laptop-fn-keys-fix/tree/main/driver

`Airplane Mode` (`F10`'s secondary function) key

The Airplane Mode key seems to go through the same input as the brightness keys, but doesn’t exhibit the issue since both a short or a long-will trigger the action just once.

Conclusion

I think the real solution is fixing how the Function (Fn) key operates on the BIOS/ACPI/EC controller firmware side, if I were to make an educated guess.

Edited:

I accidentally switched the standard definition of function lock; fixed
Clarified/re-ordered /etc/udev/hwdb.d/70-keyboard-framework-fix.hwdb

Michael_Wu · October 29, 2021, 8:45pm

Thank you @RandomUser I fixed it. And nice catch! (pretty quick haha)

Also whoops, I forgot to add the rest of the steps under the Quick Fix for Most Keys section. Fixed as well.

Charles_Daniels · November 15, 2021, 5:17pm

I have found that this solution does not work for me with Sway. Even after following the instructions in the git repository, it still seems like the End key gets stuck easily in XWayland applications. I have confirmed this by running xev. If I move the mouse into the xev window, it will see thousands of End press and release events per second (according to pv, about 14kb/s worth. If I press any other key while in the xev window, it stops until the mouse leaves and returns to the window. I do not observe this behavior with Wayland native applications, or with wev. Below is a sample of the xev output.

...
KeyPress event, serial 34, synthetic NO, window 0x1c00001,
    root 0x2a1, subw 0x0, time 48445327, (89,100), root:(1085,556),
    state 0x10, keycode 115 (keysym 0xff57, End), same_screen YES,
    XLookupString gives 0 bytes: 
    XmbLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyRelease event, serial 34, synthetic NO, window 0x1c00001,
    root 0x2a1, subw 0x0, time 48445367, (89,100), root:(1085,556),
    state 0x10, keycode 115 (keysym 0xff57, End), same_screen YES,
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyPress event, serial 34, synthetic NO, window 0x1c00001,
    root 0x2a1, subw 0x0, time 48445367, (89,100), root:(1085,556),
    state 0x10, keycode 115 (keysym 0xff57, End), same_screen YES,
    XLookupString gives 0 bytes: 
    XmbLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyRelease event, serial 34, synthetic NO, window 0x1c00001,
    root 0x2a1, subw 0x0, time 48445407, (89,100), root:(1085,556),
    state 0x10, keycode 115 (keysym 0xff57, End), same_screen YES,
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False
...

Here are some details on my environment:

$ uname -a
Linux <hostname censored> 5.14.14-zen1-1-zen #1 ZEN SMP PREEMPT Wed, 20 Oct 2021 21:35:17 +0000 x86_64 GNU/Linux
$ cat /etc/os-release 
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
LOGO=archlinux-logo
$ pacman -Q sway wayland wlroots
sway 1:1.6.1-2
wayland 1.19.0-2
wlroots 0.14.1-2

Edit: it’s worth noting that this seems not to happen all the time. Often time it will go for hours or days without exhibiting this problem, and it will also randomly stop for no apparent reason.

Michael_Wu · November 15, 2021, 5:47pm

@Charles_Daniels that sounds like the exact issue I had which resolved itself after applying the above fix (in Sway, random stuck key like End only in XWayland windows, press a key and it stops until mouse is moved back out then back into the window).

Are you sure the settings applied successfully? Assuming AT Translated Set 2 keyboard is /dev/input/event2 (check with evtest):

udevadm info /sys/class/input/event2 | grep KEYBOARD_KEY should show:

...
KEYBOARD_KEY_cf=!end
...

Can you reliably reproduce the issue with these steps? In e.g. an XWayland browser (issue should occur in Wayland as well), try this:

Hold Fn
Hold End
Release Fn
Release End

Does that result in the End key getting stuck for that window?

Can you verify if your other keys stick, like volume down/up?
For me, they did before the fix. Once applied successfully, the issues went away.

Edit: also perhaps the udev rules aren’t applying successfully in your Arch setup – they work with a base Fedora installation FWIW, so something to investigate.

Charles_Daniels · November 15, 2021, 6:29pm

I did verify with evtest and udevadm info. input2 is indeed the correct input.

After rebooting, I cannot get it to reproduce anymore using the steps you mentioned. It’s possible that since it was already in a “stuck” state at the time that I applied your fix, that it stayed in that state.

We’ll see if it comes back at some point. Until then I can’t really gather any more data.

Fingers crossed that it only didn’t work because I applied after it was already stuck, and that it will stay fixed

Michael_Wu · November 15, 2021, 7:45pm

@Charles_Daniels I’ve seen oddities where I had to run the commands twice and/or just reboot for the changes to work, but once it does, it stays. Edit: added note about possibly needing to restart.

Hope the reboot worked and that it stays fixed!

Jake_Bailey · December 6, 2021, 6:16pm

Thanks for this, I was pulling my hair out trying to figure out why switching workspaces in sway to Xwayland windows was causing my up arrow key to get stuck, though I’m not super thrilled to have to apply this hack on a brand new machine.

Kieran_Levin · December 6, 2021, 8:18pm

Jumping in here, we can reproduce this, and will work on a fix for 3.07 bios.

Jake_Bailey · December 6, 2021, 8:37pm

Wonderful, thank you!

Jake_Bailey · January 6, 2022, 9:39pm

I’m assuming that this won’t actually appear in 3.07 at this point, so will hopefully appear in some later version, correct? (I think “BETA” is just the status, and the 3.07 version is what it is.)

Zachary_Burnett · March 14, 2022, 1:37pm

Thank you very much for providing this fix. I use HOME and END a lot while typing and I thought I was going crazy when my cursor would intermittently jump to the end or beginning of a line.

Ygor_Dreyer · May 7, 2022, 5:20pm

I love you. No joke.
For some reason I was not able to find this post easily, so it took me hours of debugging and searching… almost gave up on Linux for this laptop. And you did it! Thank you so much!

Michael_Wu · May 7, 2022, 8:05pm

Haha thanks for the kind words! I sure had those same feelings @Ygor_Dreyer while I was going through this, so I’m glad I could help y’all!

Maybe this should be pinned @moderators so that others can more easily find this, until fixed in an upcoming BIOS version?

jukan · May 7, 2022, 11:51pm

Just to clarify for readers catching up on the thread, looks like the fix for this didn’t make it into the 3.07 bios beta release (as of writing this) since I can still repro so the solution in this thread is still necessary.

fearedbliss · May 18, 2022, 6:09pm

I haven’t noticed the particular function stickiness issue, but I have noticed what other people have been saying regarding certain keys (like Fn + F9 (Display Switching Logo), Fn + F10 (Airplane) not being able to be easily accessible or not generating any entries. I’m on FreeBSD 13.1-STABLE and used ‘xev’ to test on X11. (The same happened though when I was using sway / Wayland on FreeBSD as well - but using ‘wev’ instead).

Fn + F9 yields:

KeyPress event, serial 34, synthetic NO, window 0x1e00001,
root 0x593, subw 0x0, time 21470262, (132,427), root:(3260,451),
state 0x0, keycode 133 (keysym 0xffeb, Super_L), same_screen YES,
XLookupString gives 0 bytes:
XmbLookupString gives 0 bytes:
XFilterEvent returns: False

KeyPress event, serial 34, synthetic NO, window 0x1e00001,
root 0x593, subw 0x0, time 21470264, (132,427), root:(3260,451),
state 0x40, keycode 33 (keysym 0x70, p), same_screen YES,
XLookupString gives 1 bytes: (70) “p”
XmbLookupString gives 1 bytes: (70) “p”
XFilterEvent returns: False

KeyRelease event, serial 34, synthetic NO, window 0x1e00001,
root 0x593, subw 0x0, time 21470322, (132,427), root:(3260,451),
state 0x40, keycode 33 (keysym 0x70, p), same_screen YES,
XLookupString gives 1 bytes: (70) “p”
XFilterEvent returns: False

KeyRelease event, serial 34, synthetic NO, window 0x1e00001,
root 0x593, subw 0x0, time 21470322, (132,427), root:(3260,451),
state 0x40, keycode 133 (keysym 0xffeb, Super_L), same_screen YES,
XLookupString gives 0 bytes:
XFilterEvent returns: False

Fn + F10 doesn’t generate any output (Although I don’t believe my machine is intercepting that signal - I’m on i3).

Fn + F12 yields this (Seems like this might be incorrect given it references Audio but it’s the wheel key - I’ll need confirmation).

KeyPress event, serial 34, synthetic NO, window 0x1e00001,
root 0x593, subw 0x0, time 21473824, (132,427), root:(3260,451),
state 0x0, keycode 234 (keysym 0x1008ff32, XF86AudioMedia), same_screen YES,
XLookupString gives 0 bytes:
XmbLookupString gives 0 bytes:
XFilterEvent returns: False

KeyRelease event, serial 34, synthetic NO, window 0x1e00001,
root 0x593, subw 0x0, time 21473882, (132,427), root:(3260,451),
state 0x0, keycode 234 (keysym 0x1008ff32, XF86AudioMedia), same_screen YES,
XLookupString gives 0 bytes:
XFilterEvent returns: False

agentcrm · May 19, 2022, 7:34am

I just bought a HP Probook and I’m trying Ubuntu on it.
For me the issue was solved by using the Fn Lock option on the keyboard.

Not sure if the framework has that option as a workaround.

Michael_Wu · May 19, 2022, 7:54am

Yes, the Fn Lock option exists, but that’s not what this issue + its fix is about.

Chiraag_Nataraj · May 19, 2022, 7:30pm

Yes, the Framework logo function key (F12) maps to XF86AudioMedia. Others have mentioned that on Windows, it opens up the media center or something (can’t test, since I only run Linux on personal machines).

agentcrm · May 20, 2022, 1:25am

Appologies, I explained things poorly.

On the HP the default state of the keyboard is Fn keys & num pad.
When I engaged the Fn Lock it brings the default state back to the F row and standard keys.