[TRACKING] Touchpad interrupts & battery usage issues (idma64.2)

Framework Laptop 13 Linux
37

I was curious to see if this was an issue with 12th gen as well. On Arch running Linux 6.0.7-arch1-1, I get the following on my system between doing nothing and holding my finger on the touchpad measuring with turbostat:

# turbostat -c package -S --show IRQ,PkgWatt -i 1

IRQ		PkgWatt
227		0.53
203		0.52
216		0.53
238		0.52
261		0.77
1979	1.44
2295	1.58
2686	1.67
2463	1.58
3634	1.82
3577	1.78
3239	1.71
2311	1.51
3559	1.79
3528	1.79
3576	1.81
3592	1.84
2348	1.60
3543	1.80
3232	1.70
2186	1.47
3595	1.79
3575	1.79
3632	1.81
2593	1.53
3521	1.80
3563	1.78
3789	1.80
2649	1.56
2616	1.61
3562	1.79
3157	1.69
2630	1.62
3567	1.77
3649	1.77
1727	1.03
237		0.63
223		0.52
225		0.53
245		0.56

And here’s the comparable dstat numbers for reference (turbostat gives accurate package power usage so more useful for looking at real world effect?)

# dstat -t --top-int
----system---- ---most-frequent----
     time     |     interrupt      
10-11 14:51:38|i915              7 
10-11 14:51:39|i915             17 
10-11 14:51:40|i915             14 
10-11 14:51:41|i915             13 
10-11 14:51:42|idma64.2, i2c_designware.2 1107 
10-11 14:51:43|idma64.2, i2c_designware.2 2898 
10-11 14:51:44|idma64.2, i2c_designware.2 2858 
10-11 14:51:45|idma64.2, i2c_designware.2 2646 
10-11 14:51:46|idma64.2, i2c_designware.2 2926 
10-11 14:51:47|idma64.2, i2c_designware.2 2870 
10-11 14:51:48|idma64.2, i2c_designware.2 2715 
10-11 14:51:49|idma64.2, i2c_designware.2 1784 
10-11 14:51:50|idma64.2, i2c_designware.2 1881 
10-11 14:51:51|idma64.2, i2c_designware.2 1722 
10-11 14:51:52|idma64.2, i2c_designware.2 1493 
10-11 14:51:53|idma64.2, i2c_designware.2 2913 
10-11 14:51:54|idma64.2, i2c_designware.2 2897 
10-11 14:51:55|idma64.2, i2c_designware.2 2629 
10-11 14:51:56|idma64.2, i2c_designware.2 2777 
10-11 14:51:57|idma64.2, i2c_designware.2 2817 
10-11 14:51:58|idma64.2, i2c_designware.2 1688 
10-11 14:51:59|idma64.2, i2c_designware.2 2458 
10-11 14:52:00|idma64.2, i2c_designware.2  86 
10-11 14:52:01|i915             13 
10-11 14:52:02|nvme0q8          15 
10-11 14:52:03|i915             13

So, maybe someone with a Ryzen 6000 laptop can chime, but with my curiousity piqued, I also decided to test with my old 4800H PF5NU1G (running the same 6.0.7-arch1-1 kernel):

# turbostat -c package -S --show IRQ,PkgWatt,C3% -i 1

IRQ		C3%	PkgWatt
791		99.53	2.68
815		99.56	2.66
698		99.57	2.69
738		99.54	2.68
3794	88.70	3.40
5542	83.45	3.79
5534	76.96	3.76
5542	79.23	3.85
5507	85.05	3.79
5569	85.37	3.77
5518	83.33	3.74
5425	83.98	3.73
5435	65.37	3.74
5534	80.22	3.77
5555	85.38	3.74
5463	78.40	3.74
1178	87.22	3.42
768		99.54	2.66
668		99.59	2.67
664		99.56	2.66
804		99.41	2.68

As you can see, this seems to have about the same impact (more IRQs and worse on C-state residency even) on a Ryzen 4000 laptop.

38

@lhl I did similar on my 11th gen and got this:

IRQ	PkgWatt
649	0.90
613	0.92
2254	1.78
553	1.11
1939	1.59
604	0.91
627	0.98
604	0.94
579	0.92
548	0.98
593	0.99
570	0.94
592	0.93
643	1.06
590	0.98
497	0.97
555	0.86
616	0.98
355	0.84
606	1.02
563	0.94
605	1.01
608	0.89
627	0.92
572	0.90
539	0.92
579	0.94
510	0.88
576	0.95
579	0.93
1773	1.40
2510	1.93
491	0.79
1935	1.55
665	0.98
553	0.92
531	0.88
575	0.90
559	0.92
544	0.81
594	0.93
579	0.96
530	0.92
546	0.98
655	0.97
494	0.88
586	0.91
600	1.04
472	0.87
635	0.97
359	0.87
485	0.94
950	1.13
564	0.94
630	0.92
633	0.96
557	0.86
584	0.91
704	0.99
658	0.97
550	0.83
650	0.96
618	0.92
1018	1.05
640	0.97
490	0.91
386	0.86
646	1.02
578	0.90
587	0.95
630	0.95
566	0.95
563	0.90
658	1.02
658	1.04
565	0.91
572	0.94
665	0.96
574	0.94
544	1.62
345	0.83
645	0.98
733	1.05

This is on Fedora 37 (beta) with 6.0.7-*

I’ll need to spend some time with dstat Monday.

1 Like
39

I noticed the increase in CPU pakage power is substantially different to the increase in total system draw however the psys power domain (which also adds the PCH and eDRAM consumption AFAIK) is much more reflective of the change. Perhaps this is more about the PCH than CPU package?

Power Domains
Power Domains330×502 31.5 KB

My results from i5 11th gen Fedora 36 kernel 6.07

Initially the system was drawing 3.0W resting a finger increased this to 4.5W with only a ~0.5W increase to the pkg power.

IRQ	PkgWatt
548	0.49
511	0.62
463	0.61
479	0.49
552	0.67
573	0.58
459	0.48
515	0.62
422	0.53
565	0.57
800	0.73
2272	0.99
2184	0.97
2500	1.07
2362	1.05
2341	0.98
2618	1.14
2426	1.02
2262	1.00
2502	1.08
2392	0.99
2558	1.05
2436	1.05

Now “drawing” circles on the touchpad again idle at 3.0W rising to ~6.75W with 1.5W increase to pkg pwr.

IRQ	PkgWatt
442	0.45
481	0.45
485	0.55
495	0.49
441	0.55
423	0.47
453	0.46
434	0.49
419	0.53
382	0.45
423	0.51
2226	1.03
5549	1.90
5816	2.02
5573	2.00
5412	2.01
5510	2.02
5330	2.00
4921	1.96
5487	2.05
4798	1.95
5449	1.98
5859	2.03
5625	2.05
5583	2.04
41

Not a framework laptop, but Googling led me here as I am having this issue in Arch on a Dell G5 with an i7-10750H cpu. Same symptoms as above, verified with turbostat. ~1.25w idle and ~2w just moving the mouse. ~1.5w just touching the touchpad…

Edit: Can confirm that I get better battery life and lower powertop system usage on both Fedora and NixOS compared to Arch on this specific laptop… So unfortunately I’ll have to use something other than Arch on this laptop for now :frowning:

in Arch ~13-15 idle - 16-20w browsing the web (even a live ISO)
in Fedora or NixOS ~7.5w idle and 10-13w browsing the web so the difference is significant.

Edit2: installed NixOS including nvidia drivers… crummy battery life persisted, and even though in Arch I had enabled udev rules to disable the nvidia card as per arch wiki, my nvidia card still wasn’t disabling. Turns out at least on NixOS I was able to get the dGPU suspend working using the nix config posted in How to use Nvidia PRIME offload to run the X server on the integrated board - #15 by Denommus - Help - NixOS Discourse

42

Did anybody create a diff of the TLP configuration?

43

I think i have part of the reason why this is happening. I looked at this using fedora 37, and it seems the gpio interrupt is pinned to one core, and the i2c interrupts are pinned to a separate core. This means every time the touchpad fires an interrupt, both a high performance and efficiency core cluster have to wake up to service the touchpad! And these are probably not even in the same cluster, so lots of cache evictions etc might be happening as both core clusters power on and off and caches are cleared and filled.

What core type are TP interrupts handled by?

The second idea was to investigate what core the interrupts are handled by:

First you can look at the Cores on the system to map them to E or P cores. The E cores will have lower frequencies when looked at using:

lscpu --all --extended

The second thing was to find out what cores the interrupts were handled by:

cat /proc/interrupts

watch -n 1 "cat /proc/interrupts | grep designware"  
watch -n 1 "cat /proc/interrupts | grep PIXA"

            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      CPU12      CPU13      CPU14      CPU15      CPU16      CPU17      CPU18      CPU19      
  43:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   43694516          0          0          0          0  IR-IO-APIC   43-fasteoi   i2c_designware.2, idma64.2
 135:          0          0    1711105          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  intel-gpio    3  PIXA3854:00

And move the touchpad, you will see which core the touchpad interrupts are increasing on both.
However the PIXA GPIO interrupt is pinned to a high performance core, but the i2c-designware interrupt is pinned to an efficiency core.

Lets pin them both to the same core CPU2:

# echo 00004 > /proc/irq/43/smp_affinity
# echo 00004 > /proc/irq/135/smp_affinity # DOESNT WORK, IO error?

Power consumption when using the touchpad seems to drop from about 7W to 5W! (Baseline is 4W)

I would like to pin the PIXA intel-gpio interrupt to an efficiency core, but I have not figured out how to do this.

9 Likes
44

Wow, that’s an awesome find!

45

I replicate this on EndeavourOS (Arch), Zorin (Ubuntu), and a Fedora live USB environment. My dstat readouts closely resemble those posted by others above. EndeavourOS is somewhat worse, and Fedore is somewhat less-terrible. On Endeavour, a USB mouse throws about half as many interrupts as the trackpad; I didn’t think to test this on the other OS’s.

46

I seem to be seeing the same thing - the PIXA interrupt is going HAM on my first (P) core.

Which identifier does one pass to SMP Affinity from the following? I understand that the e cores are the second block, however what is the identifier? Its currently set to 00c0 for btoh, and the smp_affinity_list set to 6-7

47

I tried to investigate this. I really don’t know some things written in this thread. I tried to find a clue…

1 Like
48

For anyone testing this, here’s a minimally nicer command:

watch -n 1 "cat /proc/interrupts | grep 'CPU\|designware\|PIXA'"
1 Like
49

Here’s a simple systemd service touchpad-smp-affinity.service that pins the affinity of the designware interrupt to CPU2:

[Unit]
Description=Pin touchpad interrupts to CPU2
Documentation=https://community.frame.work/t/tracking-touchpad-interrupts-battery-usage-issues-idma64-2/13630

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/sh -c 'echo 2-2 > /proc/irq/$(grep designware.2 /proc/interrupts | cut -d ":" -f1 | xargs)/smp_affinity_list'
ExecStop=/usr/bin/sh -c 'echo "0-$(nproc --all --ignore=1)" > /proc/irq/$(grep designware.2 /proc/interrupts | cut -d ":" -f1 | xargs)/smp_affinity_list'

[Install]
WantedBy=basic.target

Test this by running

watch -n 1 "grep 'CPU\|designware\|PIXA' /proc/interrupts"

and checking if the numbers increase in the same column if using the touchpad.

I don’t think this is the final solution. As I understand from the thread, the number of generated interrupts is still too high. But this workaround is apparently better than nothing.

the large number of interrupts generated are probably from the i2c controller on the SOC, as each physical interrupt requires the CPU to transfer multiple bytes. A more optimized Linux kernel driver may be able to fix this.

Is there anyone here who’s willing to report this to the kernel folks?

4 Likes
50

Thanks everybody for the large amount of research that has gone into this.

From reading I understand that pinning interrupts to a certain core is a workaround that some people test with, but the process raises a few questions for me. I hope someone can shed a light here

  1. Why only pinning the designware.2 interrupts? On my 11th Gen I see designware.{0,1,2} interrupts listed in /proc/interrupts.
    What are the other 2 for?

  2. Why pinning to CPU 2?
    When I execute lscpu --all --extended multiple times over a range of a few minutes I see different cores with a high frequency - including CPU2. According to @Kieran_Levin the high frequency is how to recognize a P core, but if the E vs P core is not constant, the pinning ends up to be a hit & miss, no?
    Is there a way to dedicate a CPU as E/P when on battery?

  3. Why not pinning the PIXA interrupts too? Just to be explicit that both interrupts are actually handled by the same CPU.

51

For some reasion, it is not possible to pin the PIXA interrupt, so only thing we can try is to pin the disignware interrupt and assign it to the same core as the PIXA.

52

True you can’t pin this interrupt during runtime, but using the kernel param irqaffinity one can tell the kernel which CPUs to use for hardware interrupts.

Using irqaffinity=0,1 I was able to move the PIXA interrupt from CPU 7 to CPU 1.
I haven’t fully tested the impact to power consumption and/ or performance (when charging). There’s also a question how reasonable it is to set irqaffinity to a single CPU core?

Pondering on how to resolve my questions above, the plan seems to look like

  1. Pin hardware IRQs to 2 (or 1 - this needs more research) CPU using kernels irqaffinity
  2. Pin designware CPU to the same as PIXA, if more than 1 core was listed in irqaffinity, otherwise skip this step
  3. Ensure the CPU used for the touchpad interrupts is an E core when on battery.

Step 3 is the one I was struggling with on my machine (11th Gen Framework), until I stumbled over
https://www.makeuseof.com/intel-cpus-explained-what-are-e-cores-and-p-cores/ :

E/P cores is a feature since Intels 12th Gen chips, which explains why the CPU frequency behaviour @Kieran_Levin explained did not match my observations.

Also found this during research: according to x86 64 - How to detect E-cores and P-cores in Linux alder lake system? - Stack Overflow E cores can be identified (if SMT is enabeld in BIOS) by looking at the CORE column for core unique core-numbers - they represent the E cores in the system.

1 Like
53

@dfh there is another i2c bus connected to the EC for i2c-hid events that are for some keyboard hotkeys and the ALS.
The other one should be the DDR interface to get the module information.

Both of these are pretty low frequency updates so have diminishing returns, and DDR will not have any data transactions happening on it after boot.

2 Likes
54

I don’t know what I’m doing and am in way over my head (hoping someone with more knowledge can chime in, or tell me if I’m totally wrong), but I think I may have found why there are so many designware interrupts.

Looking at what is happening in the interrupt handler (in linux/drivers/i2c/busses/i2c-designware-core.h in the funcion i2c_dw_isr with a focus on i2c_dw_read), the system reads 38 bytes from i2c. However it does so at a rate of 2 bytes/interrupt (rx_valid, which dictates the number of bytes to read, is 2). Tap/click events seem to be sent more reasonably with 38 bytes over 3 interrupts (rx_valid is between 7-17). So while there may only be ~140 interrupts/sec for the touchpad, whatever related transfer the designware i2c is doing causes a lot more. Presumably, data transfered from touchpad movements can be buffered more (like it is for clicks) so more is read at once.

rx_valid is read from the register at offset 0x78 (macro DW_IC_RXFLR in i2c-designware-core.h). I’d guess that something like this would be controlled on a firmware level rather than kernel/driver, and is something that the HW is capable of considering there click inputs seem to be handled with higher rx_valid values.

Also, according to dmesg times, sequential reads within the same interrupt are around 5 microseconds (including time for printks and aside from the first 3 reads for larger rx_valid values which are closer to 20 microseconds for some reason), while the time to read between interrupts is around 50-100 microseconds (usually around 50). Besides reducing the number of context switches, this should mean much less time spent on interrupts if there are more reads/IRQ.

Some other findings:

I can’t write to smp_affinity_list for PIXA also, but it’s set to 0-15 (or 10 sometimes?) for me and after a resume (from sleep or hibernate), the interrutps are handled by cpu0 instead of cpu2.

The high number of interrupts when using the touchpad is also on windows. You can measure this with Performance Monitor or run this in powershell: typeperf "\Processor Information(_Total)\Interrupts/sec" "\Processor Information(_Total)\% Interrupt Time". Interrupt time seems to stays pretty low though (usually around 0.12-0.25%).

From perf top, a lot of % kernel time (more than i2c_dw_read) is spent on intel_gpio_irq (in pinctrl-intel.c, with most time spent specifically between the spin lock in intel_gpio_irq_handler. . Related device is INTC1055:00). I’m not sure if that’s normal (when the system is near idle). I also noticed a lot the reads determine that the pins aren’t enabled. This may be a kernel driver optimization, but if there was some way to tell if a pins are disabled before spending the time reading them, that should help (I think it can be configured in the driver since Tiger Lake?). This probably doesn’t help Framework much though it’s in the kernel/drivers.

Assuming I’m not just completely wrong and this helps at all, I can fix up some of my notes and post them on github or something.

4 Likes
55

Can’t figure out how to edit my previous comment, but I forgot to mention the pretty important part that the kernel only ever reads data from I2C when the devices sends an interrupt saying that it can’t store anything else (as in DW_IC_INTR_RX_FULL is set), thus the kernel has to read the data and there shouldn’t be any optimizations that can be done (unless I’m misunderstanding and the kernel influence when the hardware triggers RX_FULL).

56

I stumbled over this reddit post that talks about blacklisting kernel modules i2c-designware-pci or intel_lpss_pci.

I wonder if anyone has given this a try, possible in combination with the PS/2 Mouse emulation BIOS option?

In the reddit thread, the reported touchpad is elan, so the difference in hardware could yield different results?!

1 Like
57

Disabling i2c-designware-pci or intel_lpss_pci both seem to do the same thing of forcing the touchpad to fall back to PS/2 emulation (or not work if PS/2 emulation is disabled in bios). This does drop (presumably just handled) interrupts from ~2.8k to 400-600 handled by i8042, but this also makes it so scrolling and right click don’t work.

Side note: blacklisting i2c_designware with a modprobe config doesn’t work for me, but adding the kernel option initcall_blacklist=dw_i2c_init_driver does work, but then idam64 gets spammed with around 5-8k interrupts per second when using the touchpad (this doesn’t happen if i2c_designware_pci is a module and you can rmmod it. Haven’t tested modprobe blacklist with it built as a module). If you go that route of PS/2 emulation and have this issue, you’ll want to disable idma64 somehow (may have to be delayed if it is required for startup. Not sure).