[TRACKING] Touchpad interrupts & battery usage issues (idma64.2)

42jd · January 12, 2022, 8:02am

Issue background

So to preface I have been encountering this issue since I first got my framework laptop back in October but I finally seem to have narrowed down the cause (but not a fix). There is an issue where touchpad input causes seemingly abnormal amount of interrupts. This ends up causing extremely high number of context switches and atomic work creating high power usage no matter the CPU usage or performance settings (excerpt from powertop included below). It occurs in TTY, Wayland, and X11. I included all my observations but I am not entirely sure of the context around the data and how to fix.

I am running NixOS on kernel 5.15.2 but this issue has been around since I started with Framework at 5.12.5. I have never experienced it working. There are a few issues I have found that are seemingly related:

Interrupt benchmarks on X11

Moving the cursor causes extraordinarily high power usage as a result of interrupts. Only the terminal was open for these. As I continue to move the cursor, power usage continues to increase. Excerpt from dstat history showing total interrupts (int), context switches (csw), and specific interrupts (30, 186, 188) in single second intervals. When touching the touchpad it produces ~3000 interrupts/second, and drops to zero is when mouse isn’t moving. Note: the keyboard seems to be correct and produces exactly two interrupts per key press (so around 5-20 interrupts per second).

----system---- ---system-- ----interrupts---
     time     | int   csw |  30   186   188
12-01 02:24:00|4291  4948 | 699   376    45
12-01 02:24:01|1529   761 |   0    26     0
12-01 02:24:02|1381   696 |   0    20     0
12-01 02:24:03|4120  2941 |1298   229    60
12-01 02:24:04|9776    11k|3111   616   142
12-01 02:24:05|  11k   14k|2481   799   127
12-01 02:24:06|  12k   18k|2679  1040   142
12-01 02:24:07|  11k   14k|2735   949   142
12-01 02:24:08|  12k   18k|3112  1054   142
12-01 02:24:09|  11k   15k|2871  1030   142
12-01 02:24:10|9865    12k|3067   899   142
12-01 02:24:11|8589  8820 |3064   899   142
12-01 02:24:12|8383  8291 |3065   782   142
12-01 02:24:13|6050  3717 |2540   449   118
12-01 02:24:14|1435   687 |   0    22     0
12-01 02:24:15|1490   801 |   0    22     0
12-01 02:24:16|1329   599 |   0    19     0
12-01 02:24:17|1487   762 |   0    23     0
12-01 02:24:18|1695  1178 |   0    29     0

The detailed excerpt from /proc/interrupts. Included PIXA as that I believe is the touchpad and it occasionally hits high power usage.

            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
  30:          0          0    2010982          0          0          0          0          0  IR-IO-APIC   30-fasteoi   idma64.2, i2c_designware.2
 186:          0          0          0          0    1022896          0          0          0  IR-PCI-MSI 32768-edge      i915
 188:          0          0          0          0     122111          0          0          0  INT34C5:00    3  PIXA3854:00t

Some other less relevant ones, not sure what they mean:

 NMI:          0          0          0          0          0          0          0          0   Non-maskable interrupts
 LOC:     893261     740162     921499     767780    1388452     695251     670815     687432   Local timer interrupts
 SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
 PMI:          0          0          0          0          0          0          0          0   Performance monitoring interrupts
 IWI:      20585      16309      13147      11025     720474      16316      15925       9071   IRQ work interrupts
 RTR:          0          0          0          0          0          0          0          0   APIC ICR read retries
 RES:      12691      11043      21085      15608      25167      11877      12430      24970   Rescheduling interrupts
 CAL:     188599     139740     122271     141921      94140     108913     115163     108439   Function call interrupts
 TLB:      92443      79167      77591     104669      59372      72812      80577      74698   TLB shootdowns

Powertop benchmarks on Wayland (battery power)

Now for the fun powertop. For this I was on wayland. The first benchmark the only thing running was powertop (with a few background services) in a full screen terminal. I have extremely conservative power settings but notice all the top process power users when I am using the touchpad. Powertop isn’t the most accurate but it definitely wouldn’t be that wrong.

The battery reports a discharge rate of 8.90 W
The energy consumed was 202 J
The estimated remaining time is 3 hours, 59 minutes

Summary: 1412.0 wakeups/second,  0.0 GPU ops/seconds, 0.0 VFS ops/sec and 28.9% CPU use

Power est.              Usage       Events/s    Category       Description
  1.77 W     30.0%                      Device         Display backlight
  1.32 W      6.9 ms/s     454.0        Timer          tick_sched_timer
  782 mW      1.9 pkts/s                Device         Network interface: wlp170s0 (iwlwifi)
  582 mW      3.0 ms/s     199.7        Interrupt      [30] idma64.2
  354 mW      7.0 ms/s     119.0        Process        [PID 1418] [irq/188-PIXA385]
  332 mW    142.7 ms/s      50.7        kWork          intel_atomic_commit_work
  193 mW     35.2 ms/s      51.0        Process        [PID 9543] /nix/store/65ihhv3wk5npiccq4y9dapqw1mf70wfb-dwl-jd-0.1/bin/dwl -s /nix/store/d8ba2wsv5ppjgc0l3
  189 mW      1.3 ms/s      64.6        Process        [PID 1641] /nix/store/cw74cxb9jxssdvf5lqcyjvc2hlr9larj-keybase-5.8.1/bin/keybase service --auto-forked
  181 mW    539.2 µs/s      62.4        kWork          rps_work
  169 mW    561.1 µs/s      58.0        Process        [PID 14] [rcu_sched]
  148 mW      1.1 ms/s      50.8        kWork          intel_atomic_cleanup_work
  115 mW      0.9 ms/s      39.4        kWork          engine_retire

Here is a powertop when watching a 4K video on firefox (albeit not hardware accelerated).

The battery reports a discharge rate of 16.9 W
The energy consumed was 322 J
The estimated remaining time is 2 hours, 0 minutes

Summary: 1665.1 wakeups/second,  0.0 GPU ops/seconds, 0.0 VFS ops/sec and 302.5% CPU use

Power est.              Usage       Events/s    Category       Description
  3.48 W     2574 ms/s      13.9        kWork          intel_atomic_commit_work
  3.41 W     1294 pkts/s                Device         Network interface: wlp170s0 (iwlwifi)
  2.61 W      0.0 µs/s      0.00        Process        [PID 2087] /nix/store/9658in1mpbz3a7921sqgjfrplrbdbd64-pipewire-0.3.42/bin/pipewire
  1.62 W     30.0%                      Device         Display backlight
  1.57 W     16.0 ms/s     549.4        Timer          tick_sched_timer
  323 mW    186.4 ms/s      26.3        Process        [PID 10852] /nix/store/32xbijs1iz5hlzvx28mip7ix68dvhcj6-firefox-95.0.2/lib/firefox/firefox -contentproc -
  254 mW    682.7 µs/s      90.0        kWork          flush_to_ldisc
  226 mW     11.6 ms/s      74.7        Process        [PID 9874] /nix/store/qq2ik3vsyh3ag998zxl1gcngf07bc619-foot-1.10.3/bin/foot
  207 mW      3.8 ms/s      71.9        Process        [PID 11195] /nix/store/32xbijs1iz5hlzvx28mip7ix68dvhcj6-firefox-95.0.2/lib/firefox/firefox -contentproc -
  162 mW      0.9 ms/s      57.3        kWork          rps_work
  150 mW      2.3 ms/s      52.1        Interrupt      [30] idma64.2
  138 mW      4.0 ms/s      47.4        Process        [PID 11148] /nix/store/32xbijs1iz5hlzvx28mip7ix68dvhcj6-firefox-95.0.2/lib/firefox/firefox -contentproc -
  137 mW      2.4 ms/s      47.6        Process        [PID 10645] /nix/store/32xbijs1iz5hlzvx28mip7ix68dvhcj6-firefox-95.0.2/bin/.firefox-wrapped
  135 mW      0.8 ms/s      47.7        kWork          iwl_pcie_rx_allocator_work

Note on X11 this is way worse (20+ watt power usage) and a USB 2.0 Goodix device appears.

Finishing off with TTY

Same dstat as X11 test on the tty:

----system---- ---system-- ----interrupts---
     time     | int   csw |  30   186   188 
12-01 02:58:00|4997  5652 | 792   398    48 
12-01 02:58:01| 364   591 |   0     0     0 
12-01 02:58:02| 983   655 | 591     0    50 
12-01 02:58:03|2706  1252 |1825     0   132 
12-01 02:58:04|2221  1000 |1586     0   123 
12-01 02:58:05|2226  1061 |1569     0   123 
12-01 02:58:06|1076   703 | 651     0    50 
12-01 02:58:07| 355   563 |   0     0     0

Exact same issues appear with the touchpad. And the same power issues (albeit this time with around 5-6 watt power usage as there are no graphics). Still too high though.

Now that you have seen all the observations, I am hoping someone with the right knowledge can point me in the direction of a fix! Took me a few months to learn linux well enough to pinpoint the exact issue and know I wasn’t crazy so I am hoping there is a solution.

nrp · January 13, 2022, 1:22am

Would you be able to check if this also occurs on other kernel versions? One fast way to check that would be a Fedora 35 or Ubuntu 21.10 Live USB.

Kieran_Levin · January 13, 2022, 1:32am

Interesting. We would expect around 140 hardware interrupts/second from the touchpad.

For your note about the usb goodix device. This is the fingerprint reader.

42jd · January 13, 2022, 2:17am

Perfect (in terms of finding the issue )! Been just working around the high power consumption as it wasn’t too bad as I mainly do terminal work. I started Linux with the framework so didn’t have the capabilities to debug this until recently.

@nrp In terms of another distro (eg Ubuntu) I’ll try out a usb boot sometime in the next few days. If you have any specific Linux kernel versions you know that work, let me know as I can change my NixOS config to pin to it (only cost is having to compile from source).

2disbetter · January 13, 2022, 11:52am

My several times on Ubuntu with my Framework were with Fedora 34 and Ubuntu 20.04 / 21.04. IN Federa the only issue I had was occasional stops when using the touchpad. It was minor and didn’t bother me at all. I chalked it up to being Linux.

On Ubuntu I did not have these problems. Why that is the case, I can’t say.

Linen · January 14, 2022, 3:26am

Tested mine with the latest Arch 5.16 kernel and I’m only hitting ~5k at most in system int and csw, however in powertop I am getting almost the same Usage and Events/s in idma64.2.

Duane_Johnson · March 10, 2022, 4:44am

Has anyone identified the root cause on this?

I’m also seeing way too many interrupts, and I’ve noticed that it may be the cause of some mysterious audio stuttering that I had attributed to web browsing. The stuttering only occurs when I am moving the pointer (via touchpad) or swiping (gestures) on the touchpad, however.

Duane_Johnson · March 25, 2022, 4:29am

A possible clue:

The following procedure reduces IRQ/sec from thousands to about 100 or so:

rmmod intel_lpss_pci

suspend to ram, wait a few seconds, resume, reload the kernel module

modprobe intel_lpss_pci

(from this askubuntu q/a)

42jd · March 26, 2022, 7:59pm

@Duane_Johnson What kernel version are you on?

I tried that procedure (going to S3 sleep for a few seconds) but unfortunately had no change in IRQ/sec.

Duane_Johnson · March 26, 2022, 8:57pm

Kernel:

5.16.11-76051611-generic

I’m having doubts now about whether that procedure fixed anything, because I did a reboot and tested IRQ/sec and it’s still at a reasonable 100-or-so per second.

So perhaps something else fixed this issue for me, and I just didn’t know it at the time…

42jd · March 26, 2022, 10:13pm

Interesting, I pinned my kernel to 5.16.11 but still have the same interrupts. What distribution are you running? I am on NixOS so running the official release but your kernel is a distribution provided one. Fingers crossed they might be applying a patch that fixes the issue.

42jd · March 31, 2022, 9:00pm

Spent some more time investigating and I think it must be a hardware issue. On NixOS I started by disabling kernel modules. When disabling intel_lpss_pci IRQ 30 shuts off and it switches to IRQ 16 (i8042) with around 500 interrupts/sec. Multitouch gestures are disabled so I believe it is just switching to psmouse. Exact same thing occurs when disabling any of the i2c related kernel modules. 500 I believe is still too high based on Kieran’s response earlier so it isn’t a bug in the multitouch kernel drivers.

I finally got around to live booting Fedora and the same 2000 interrupts/sec occur on IRQ 30. Unfortunately I am unable to disable i2c_designware_core as it is builtin and don’t have the time to fully install fedora and blacklist the module but I think this is enough evidence to show it not being a software issue.

My final attempt was reconnecting the touchpad cables and it had no effect. Any thoughts on this from the framework team?

Mathias_Beig · April 5, 2022, 7:36pm

Any news on this topic? I also noticed the insane number of interrupts/second under archlinux and kde plasma. When you just let the laptop idle, open powertop and move the cursor constantly, the interrupts skyrocket and the power usage gets up really significant.

I also tested a new Ubuntu 22.04 beta gnome live image, because a reddit post suggested really good idle power management → same here as under archlinux and kde plasma (Powertop Description: idma64.2)

42jd · April 19, 2022, 9:18pm

I contacted framework support and they sent me a new input cover but unfortunately the issue persists. Behavior was identical as with the old touchpad. Thanks though to framework support for being prompt after I reached out! To sum up what I have ruled out so far:

touchpad hardware (from testing new input cover)
touchpad connnection (from testing new input cover)
multitouch kernel drivers (disabling intel_lpss_pci, issue persisted on i8042)
i2c_designware driver (disabling i2c_designware, issue persisted on i8042)
kernel version (tested countless versions since 5.13)
Linux distribution (tested NixOS and Fedora 35)

I’ll post any updates if I find a solution/learn more from framework support.

Sven_Hiller · April 24, 2022, 6:04am

DIY 1185G7 running Fedora 35.
Just wanted to confirm the outlined observations.

I noticed the very same behaviour when moving a wired USB mouse (IBM Travel Mouse) or a wireless BT mouse with dedicated USB adapter (Logitech V500) while leaving the touchpad completely unused.

dk505 · April 26, 2022, 3:02am

Similar findings on DIY i7-1165G7. Arch with kernels 5.16.{0…16} and 5.17.{1…4}, and Ubuntu 22.4 with 5.15.0.

Interrupt 30, idma64.2, is not triggered unless I touch the touchpad. If i just rest a finger on it, it gets up to ~1500/s. If i continuously move the cursor, up to ~2000/s.

The same is visible in powertop. The system idles at 2.7W, but gently moving the cursor about doubles that.

PS/2 emulation is disabled, multitouch is enabled.

PhilS · April 26, 2022, 4:00pm

I had problems with my touchpad’s hardware click button not registering unless you pressed real hard. framework support replaced my touchpad and that fixed the HW issue, but I found this thread while I was trying to troubleshoot.

I’m running Ubuntu 20.04 with Kernel 5.14.9-051409-generic, and I too see a shitton (metric) of interrupts from idma64.2, and i2c_designware.2 while interacting with the touchpad. I just ran dstat -t --top-int which displays the highest offender every second, and I see close to 2000 per second just with a finger on the TP.

I’m an embedded systems engineer, but not a Linux expert. However, I can’t understand why this many interrupts would be needed? I’ve spent hours and hours trying to get the power consumption tuned better so far, I’m still nowhere as good as my old Lenovo X1 6th gen, which had no special tuning. I’m running lightdm, and I can’t get suspend to work right either, so I have it disabled.

42jd · April 27, 2022, 4:20pm

It seems that framework is labeling this as a kernel driver issue that is out of scope of support. I might be wrong as support wasn’t super specific but thats what I got from the tone of the email. If anyone in the community is experienced in linux kernel driver programming this excerpt might be of help: “The touchpad is an i2c-hid device that is connected directly to the intel SOC.”

My key takeaway is that if its a kernel driver issue then everyone running linux should be affected. Thus I am extremely interested in knowing if any linux users don’t have this issue. I am currently running 5.17.3 and have had this issue since I got my framework laptop at 5.13.

42jd · April 27, 2022, 4:42pm

Adding on the one thing I haven’t tested is bios version. From dmidecode I am running version 3.02 with revision 3.2. When I get the chance I will upgrade to the latest bios and see if that fixes anything.

edit: updated to bios 3.07. Still no change. I replied to framework support to see if they can give any insight on what kernel/distro was being run where they achieved 150 interrupts/sec.

Sven_Hiller · April 28, 2022, 8:38am

BIOS 3.08 - high wake-up count unchanged.