Touchpad interrupts & battery usage issues (idma64.2)

Issue background

So to preface I have been encountering this issue since I first got my framework laptop back in October but I finally seem to have narrowed down the cause (but not a fix). There is an issue where touchpad input causes seemingly abnormal amount of interrupts. This ends up causing extremely high number of context switches and atomic work creating high power usage no matter the CPU usage or performance settings (excerpt from powertop included below). It occurs in TTY, Wayland, and X11. I included all my observations but I am not entirely sure of the context around the data and how to fix.

I am running NixOS on kernel 5.15.2 but this issue has been around since I started with Framework at 5.12.5. I have never experienced it working. There are a few issues I have found that are seemingly related:

Interrupt benchmarks on X11

Moving the cursor causes extraordinarily high power usage as a result of interrupts. Only the terminal was open for these. As I continue to move the cursor, power usage continues to increase. Excerpt from dstat history showing total interrupts (int), context switches (csw), and specific interrupts (30, 186, 188) in single second intervals. When touching the touchpad it produces ~3000 interrupts/second, and drops to zero is when mouse isn’t moving. Note: the keyboard seems to be correct and produces exactly two interrupts per key press (so around 5-20 interrupts per second).

----system---- ---system-- ----interrupts---
     time     | int   csw |  30   186   188
12-01 02:24:00|4291  4948 | 699   376    45
12-01 02:24:01|1529   761 |   0    26     0
12-01 02:24:02|1381   696 |   0    20     0
12-01 02:24:03|4120  2941 |1298   229    60
12-01 02:24:04|9776    11k|3111   616   142
12-01 02:24:05|  11k   14k|2481   799   127
12-01 02:24:06|  12k   18k|2679  1040   142
12-01 02:24:07|  11k   14k|2735   949   142
12-01 02:24:08|  12k   18k|3112  1054   142
12-01 02:24:09|  11k   15k|2871  1030   142
12-01 02:24:10|9865    12k|3067   899   142
12-01 02:24:11|8589  8820 |3064   899   142
12-01 02:24:12|8383  8291 |3065   782   142
12-01 02:24:13|6050  3717 |2540   449   118
12-01 02:24:14|1435   687 |   0    22     0
12-01 02:24:15|1490   801 |   0    22     0
12-01 02:24:16|1329   599 |   0    19     0
12-01 02:24:17|1487   762 |   0    23     0
12-01 02:24:18|1695  1178 |   0    29     0

The detailed excerpt from /proc/interrupts. Included PIXA as that I believe is the touchpad and it occasionally hits high power usage.

            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
  30:          0          0    2010982          0          0          0          0          0  IR-IO-APIC   30-fasteoi   idma64.2, i2c_designware.2
 186:          0          0          0          0    1022896          0          0          0  IR-PCI-MSI 32768-edge      i915
 188:          0          0          0          0     122111          0          0          0  INT34C5:00    3  PIXA3854:00t

Some other less relevant ones, not sure what they mean:

 NMI:          0          0          0          0          0          0          0          0   Non-maskable interrupts
 LOC:     893261     740162     921499     767780    1388452     695251     670815     687432   Local timer interrupts
 SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
 PMI:          0          0          0          0          0          0          0          0   Performance monitoring interrupts
 IWI:      20585      16309      13147      11025     720474      16316      15925       9071   IRQ work interrupts
 RTR:          0          0          0          0          0          0          0          0   APIC ICR read retries
 RES:      12691      11043      21085      15608      25167      11877      12430      24970   Rescheduling interrupts
 CAL:     188599     139740     122271     141921      94140     108913     115163     108439   Function call interrupts
 TLB:      92443      79167      77591     104669      59372      72812      80577      74698   TLB shootdowns

Powertop benchmarks on Wayland (battery power)

Now for the fun powertop. For this I was on wayland. The first benchmark the only thing running was powertop (with a few background services) in a full screen terminal. I have extremely conservative power settings but notice all the top process power users when I am using the touchpad. Powertop isn’t the most accurate but it definitely wouldn’t be that wrong.

The battery reports a discharge rate of 8.90 W
The energy consumed was 202 J
The estimated remaining time is 3 hours, 59 minutes

Summary: 1412.0 wakeups/second,  0.0 GPU ops/seconds, 0.0 VFS ops/sec and 28.9% CPU use

Power est.              Usage       Events/s    Category       Description
  1.77 W     30.0%                      Device         Display backlight
  1.32 W      6.9 ms/s     454.0        Timer          tick_sched_timer
  782 mW      1.9 pkts/s                Device         Network interface: wlp170s0 (iwlwifi)
  582 mW      3.0 ms/s     199.7        Interrupt      [30] idma64.2
  354 mW      7.0 ms/s     119.0        Process        [PID 1418] [irq/188-PIXA385]
  332 mW    142.7 ms/s      50.7        kWork          intel_atomic_commit_work
  193 mW     35.2 ms/s      51.0        Process        [PID 9543] /nix/store/65ihhv3wk5npiccq4y9dapqw1mf70wfb-dwl-jd-0.1/bin/dwl -s /nix/store/d8ba2wsv5ppjgc0l3
  189 mW      1.3 ms/s      64.6        Process        [PID 1641] /nix/store/cw74cxb9jxssdvf5lqcyjvc2hlr9larj-keybase-5.8.1/bin/keybase service --auto-forked
  181 mW    539.2 µs/s      62.4        kWork          rps_work
  169 mW    561.1 µs/s      58.0        Process        [PID 14] [rcu_sched]
  148 mW      1.1 ms/s      50.8        kWork          intel_atomic_cleanup_work
  115 mW      0.9 ms/s      39.4        kWork          engine_retire

Here is a powertop when watching a 4K video on firefox (albeit not hardware accelerated).

The battery reports a discharge rate of 16.9 W
The energy consumed was 322 J
The estimated remaining time is 2 hours, 0 minutes

Summary: 1665.1 wakeups/second,  0.0 GPU ops/seconds, 0.0 VFS ops/sec and 302.5% CPU use

Power est.              Usage       Events/s    Category       Description
  3.48 W     2574 ms/s      13.9        kWork          intel_atomic_commit_work
  3.41 W     1294 pkts/s                Device         Network interface: wlp170s0 (iwlwifi)
  2.61 W      0.0 µs/s      0.00        Process        [PID 2087] /nix/store/9658in1mpbz3a7921sqgjfrplrbdbd64-pipewire-0.3.42/bin/pipewire
  1.62 W     30.0%                      Device         Display backlight
  1.57 W     16.0 ms/s     549.4        Timer          tick_sched_timer
  323 mW    186.4 ms/s      26.3        Process        [PID 10852] /nix/store/32xbijs1iz5hlzvx28mip7ix68dvhcj6-firefox-95.0.2/lib/firefox/firefox -contentproc -
  254 mW    682.7 µs/s      90.0        kWork          flush_to_ldisc
  226 mW     11.6 ms/s      74.7        Process        [PID 9874] /nix/store/qq2ik3vsyh3ag998zxl1gcngf07bc619-foot-1.10.3/bin/foot
  207 mW      3.8 ms/s      71.9        Process        [PID 11195] /nix/store/32xbijs1iz5hlzvx28mip7ix68dvhcj6-firefox-95.0.2/lib/firefox/firefox -contentproc -
  162 mW      0.9 ms/s      57.3        kWork          rps_work
  150 mW      2.3 ms/s      52.1        Interrupt      [30] idma64.2
  138 mW      4.0 ms/s      47.4        Process        [PID 11148] /nix/store/32xbijs1iz5hlzvx28mip7ix68dvhcj6-firefox-95.0.2/lib/firefox/firefox -contentproc -
  137 mW      2.4 ms/s      47.6        Process        [PID 10645] /nix/store/32xbijs1iz5hlzvx28mip7ix68dvhcj6-firefox-95.0.2/bin/.firefox-wrapped
  135 mW      0.8 ms/s      47.7        kWork          iwl_pcie_rx_allocator_work

Note on X11 this is way worse (20+ watt power usage) and a USB 2.0 Goodix device appears. :confused:

Finishing off with TTY

Same dstat as X11 test on the tty:

----system---- ---system-- ----interrupts---
     time     | int   csw |  30   186   188 
12-01 02:58:00|4997  5652 | 792   398    48 
12-01 02:58:01| 364   591 |   0     0     0 
12-01 02:58:02| 983   655 | 591     0    50 
12-01 02:58:03|2706  1252 |1825     0   132 
12-01 02:58:04|2221  1000 |1586     0   123 
12-01 02:58:05|2226  1061 |1569     0   123 
12-01 02:58:06|1076   703 | 651     0    50 
12-01 02:58:07| 355   563 |   0     0     0 

Exact same issues appear with the touchpad. And the same power issues (albeit this time with around 5-6 watt power usage as there are no graphics). Still too high though.

Now that you have seen all the observations, I am hoping someone with the right knowledge can point me in the direction of a fix! Took me a few months to learn linux well enough to pinpoint the exact issue and know I wasn’t crazy so I am hoping there is a solution.

4 Likes

Would you be able to check if this also occurs on other kernel versions? One fast way to check that would be a Fedora 35 or Ubuntu 21.10 Live USB.

Interesting. We would expect around 140 hardware interrupts/second from the touchpad.

For your note about the usb goodix device. This is the fingerprint reader.

1 Like

Perfect (in terms of finding the issue :slight_smile: )! Been just working around the high power consumption as it wasn’t too bad as I mainly do terminal work. I started Linux with the framework so didn’t have the capabilities to debug this until recently.

@nrp In terms of another distro (eg Ubuntu) I’ll try out a usb boot sometime in the next few days. If you have any specific Linux kernel versions you know that work, let me know as I can change my NixOS config to pin to it (only cost is having to compile from source).

My several times on Ubuntu with my Framework were with Fedora 34 and Ubuntu 20.04 / 21.04. IN Federa the only issue I had was occasional stops when using the touchpad. It was minor and didn’t bother me at all. I chalked it up to being Linux.

On Ubuntu I did not have these problems. Why that is the case, I can’t say.

Tested mine with the latest Arch 5.16 kernel and I’m only hitting ~5k at most in system int and csw, however in powertop I am getting almost the same Usage and Events/s in idma64.2.