Hi, I’m Matt (the Linux Support Lead) and I wanted to see if I can replicate this. Now for the oddball questions that may not seem relevant.
Is this on a 11th or 12th gen Framework?
2 Likes
I’ve been experiencing this on a 11th gen Framework, and looking from the posts above they also seem to have 11th gen Framework laptops.
I was curious to see if this was an issue with 12th gen as well. On Arch running Linux 6.0.7-arch1-1, I get the following on my system between doing nothing and holding my finger on the touchpad measuring with turbostat:
# turbostat -c package -S --show IRQ,PkgWatt -i 1
IRQ PkgWatt
227 0.53
203 0.52
216 0.53
238 0.52
261 0.77
1979 1.44
2295 1.58
2686 1.67
2463 1.58
3634 1.82
3577 1.78
3239 1.71
2311 1.51
3559 1.79
3528 1.79
3576 1.81
3592 1.84
2348 1.60
3543 1.80
3232 1.70
2186 1.47
3595 1.79
3575 1.79
3632 1.81
2593 1.53
3521 1.80
3563 1.78
3789 1.80
2649 1.56
2616 1.61
3562 1.79
3157 1.69
2630 1.62
3567 1.77
3649 1.77
1727 1.03
237 0.63
223 0.52
225 0.53
245 0.56
And here’s the comparable dstat numbers for reference (turbostat gives accurate package power usage so more useful for looking at real world effect?)
# dstat -t --top-int
----system---- ---most-frequent----
time | interrupt
10-11 14:51:38|i915 7
10-11 14:51:39|i915 17
10-11 14:51:40|i915 14
10-11 14:51:41|i915 13
10-11 14:51:42|idma64.2, i2c_designware.2 1107
10-11 14:51:43|idma64.2, i2c_designware.2 2898
10-11 14:51:44|idma64.2, i2c_designware.2 2858
10-11 14:51:45|idma64.2, i2c_designware.2 2646
10-11 14:51:46|idma64.2, i2c_designware.2 2926
10-11 14:51:47|idma64.2, i2c_designware.2 2870
10-11 14:51:48|idma64.2, i2c_designware.2 2715
10-11 14:51:49|idma64.2, i2c_designware.2 1784
10-11 14:51:50|idma64.2, i2c_designware.2 1881
10-11 14:51:51|idma64.2, i2c_designware.2 1722
10-11 14:51:52|idma64.2, i2c_designware.2 1493
10-11 14:51:53|idma64.2, i2c_designware.2 2913
10-11 14:51:54|idma64.2, i2c_designware.2 2897
10-11 14:51:55|idma64.2, i2c_designware.2 2629
10-11 14:51:56|idma64.2, i2c_designware.2 2777
10-11 14:51:57|idma64.2, i2c_designware.2 2817
10-11 14:51:58|idma64.2, i2c_designware.2 1688
10-11 14:51:59|idma64.2, i2c_designware.2 2458
10-11 14:52:00|idma64.2, i2c_designware.2 86
10-11 14:52:01|i915 13
10-11 14:52:02|nvme0q8 15
10-11 14:52:03|i915 13
So, maybe someone with a Ryzen 6000 laptop can chime, but with my curiousity piqued, I also decided to test with my old 4800H PF5NU1G (running the same 6.0.7-arch1-1 kernel):
# turbostat -c package -S --show IRQ,PkgWatt,C3% -i 1
IRQ C3% PkgWatt
791 99.53 2.68
815 99.56 2.66
698 99.57 2.69
738 99.54 2.68
3794 88.70 3.40
5542 83.45 3.79
5534 76.96 3.76
5542 79.23 3.85
5507 85.05 3.79
5569 85.37 3.77
5518 83.33 3.74
5425 83.98 3.73
5435 65.37 3.74
5534 80.22 3.77
5555 85.38 3.74
5463 78.40 3.74
1178 87.22 3.42
768 99.54 2.66
668 99.59 2.67
664 99.56 2.66
804 99.41 2.68
As you can see, this seems to have about the same impact (more IRQs and worse on C-state residency even) on a Ryzen 4000 laptop.
@lhl I did similar on my 11th gen and got this:
IRQ PkgWatt
649 0.90
613 0.92
2254 1.78
553 1.11
1939 1.59
604 0.91
627 0.98
604 0.94
579 0.92
548 0.98
593 0.99
570 0.94
592 0.93
643 1.06
590 0.98
497 0.97
555 0.86
616 0.98
355 0.84
606 1.02
563 0.94
605 1.01
608 0.89
627 0.92
572 0.90
539 0.92
579 0.94
510 0.88
576 0.95
579 0.93
1773 1.40
2510 1.93
491 0.79
1935 1.55
665 0.98
553 0.92
531 0.88
575 0.90
559 0.92
544 0.81
594 0.93
579 0.96
530 0.92
546 0.98
655 0.97
494 0.88
586 0.91
600 1.04
472 0.87
635 0.97
359 0.87
485 0.94
950 1.13
564 0.94
630 0.92
633 0.96
557 0.86
584 0.91
704 0.99
658 0.97
550 0.83
650 0.96
618 0.92
1018 1.05
640 0.97
490 0.91
386 0.86
646 1.02
578 0.90
587 0.95
630 0.95
566 0.95
563 0.90
658 1.02
658 1.04
565 0.91
572 0.94
665 0.96
574 0.94
544 1.62
345 0.83
645 0.98
733 1.05
This is on Fedora 37 (beta) with 6.0.7-*
I’ll need to spend some time with dstat Monday.
1 Like
I noticed the increase in CPU pakage power is substantially different to the increase in total system draw however the psys power domain (which also adds the PCH and eDRAM consumption AFAIK) is much more reflective of the change. Perhaps this is more about the PCH than CPU package?
My results from i5 11th gen Fedora 36 kernel 6.07
Initially the system was drawing 3.0W resting a finger increased this to 4.5W with only a ~0.5W increase to the pkg power.
IRQ PkgWatt
548 0.49
511 0.62
463 0.61
479 0.49
552 0.67
573 0.58
459 0.48
515 0.62
422 0.53
565 0.57
800 0.73
2272 0.99
2184 0.97
2500 1.07
2362 1.05
2341 0.98
2618 1.14
2426 1.02
2262 1.00
2502 1.08
2392 0.99
2558 1.05
2436 1.05
Now “drawing” circles on the touchpad again idle at 3.0W rising to ~6.75W with 1.5W increase to pkg pwr.
IRQ PkgWatt
442 0.45
481 0.45
485 0.55
495 0.49
441 0.55
423 0.47
453 0.46
434 0.49
419 0.53
382 0.45
423 0.51
2226 1.03
5549 1.90
5816 2.02
5573 2.00
5412 2.01
5510 2.02
5330 2.00
4921 1.96
5487 2.05
4798 1.95
5449 1.98
5859 2.03
5625 2.05
5583 2.04
Not a framework laptop, but Googling led me here as I am having this issue in Arch on a Dell G5 with an i7-10750H cpu. Same symptoms as above, verified with turbostat. ~1.25w idle and ~2w just moving the mouse. ~1.5w just touching the touchpad…
Edit: Can confirm that I get better battery life and lower powertop system usage on both Fedora and NixOS compared to Arch on this specific laptop… So unfortunately I’ll have to use something other than Arch on this laptop for now 
in Arch ~13-15 idle - 16-20w browsing the web (even a live ISO)
in Fedora or NixOS ~7.5w idle and 10-13w browsing the web so the difference is significant.
Edit2: installed NixOS including nvidia drivers… crummy battery life persisted, and even though in Arch I had enabled udev rules to disable the nvidia card as per arch wiki, my nvidia card still wasn’t disabling. Turns out at least on NixOS I was able to get the dGPU suspend working using the nix config posted in How to use Nvidia PRIME offload to run the X server on the integrated board - #15 by Denommus - Help - NixOS Discourse
Did anybody create a diff of the TLP configuration?
I think i have part of the reason why this is happening. I looked at this using fedora 37, and it seems the gpio interrupt is pinned to one core, and the i2c interrupts are pinned to a separate core. This means every time the touchpad fires an interrupt, both a high performance and efficiency core cluster have to wake up to service the touchpad! And these are probably not even in the same cluster, so lots of cache evictions etc might be happening as both core clusters power on and off and caches are cleared and filled.
What core type are TP interrupts handled by?
The second idea was to investigate what core the interrupts are handled by:
First you can look at the Cores on the system to map them to E or P cores. The E cores will have lower frequencies when looked at using:
lscpu --all --extended
The second thing was to find out what cores the interrupts were handled by:
cat /proc/interrupts
watch -n 1 "cat /proc/interrupts | grep designware"
watch -n 1 "cat /proc/interrupts | grep PIXA"
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19
43: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 43694516 0 0 0 0 IR-IO-APIC 43-fasteoi i2c_designware.2, idma64.2
135: 0 0 1711105 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 intel-gpio 3 PIXA3854:00
And move the touchpad, you will see which core the touchpad interrupts are increasing on both.
However the PIXA GPIO interrupt is pinned to a high performance core, but the i2c-designware interrupt is pinned to an efficiency core.
Lets pin them both to the same core CPU2:
# echo 00004 > /proc/irq/43/smp_affinity
# echo 00004 > /proc/irq/135/smp_affinity # DOESNT WORK, IO error?
Power consumption when using the touchpad seems to drop from about 7W to 5W! (Baseline is 4W)
I would like to pin the PIXA intel-gpio interrupt to an efficiency core, but I have not figured out how to do this.
9 Likes
Wow, that’s an awesome find!
I replicate this on EndeavourOS (Arch), Zorin (Ubuntu), and a Fedora live USB environment. My dstat readouts closely resemble those posted by others above. EndeavourOS is somewhat worse, and Fedore is somewhat less-terrible. On Endeavour, a USB mouse throws about half as many interrupts as the trackpad; I didn’t think to test this on the other OS’s.
I seem to be seeing the same thing - the PIXA interrupt is going HAM on my first (P) core.
Which identifier does one pass to SMP Affinity from the following? I understand that the e cores are the second block, however what is the identifier? Its currently set to 00c0 for btoh, and the smp_affinity_list set to 6-7
I tried to investigate this. I really don’t know some things written in this thread. I tried to find a clue…
- Official document: SMP IRQ affinity — The Linux Kernel documentation
- [RFC PATCH 0/5] irq: sysfs interface improvements for SMP affinity control : It seems this patch is to add the sysfs interface (
/sys/fs/?) to control the SMP affinity. I am not sure if the patch is merged or not. - found the 2 scripts.
1 Like
For anyone testing this, here’s a minimally nicer command:
watch -n 1 "cat /proc/interrupts | grep 'CPU\|designware\|PIXA'"
1 Like
Here’s a simple systemd service touchpad-smp-affinity.service that pins the affinity of the designware interrupt to CPU2:
[Unit]
Description=Pin touchpad interrupts to CPU2
Documentation=https://community.frame.work/t/tracking-touchpad-interrupts-battery-usage-issues-idma64-2/13630
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/sh -c 'echo 2-2 > /proc/irq/$(grep designware.2 /proc/interrupts | cut -d ":" -f1 | xargs)/smp_affinity_list'
ExecStop=/usr/bin/sh -c 'echo "0-$(nproc --all --ignore=1)" > /proc/irq/$(grep designware.2 /proc/interrupts | cut -d ":" -f1 | xargs)/smp_affinity_list'
[Install]
WantedBy=basic.target
Test this by running
watch -n 1 "grep 'CPU\|designware\|PIXA' /proc/interrupts"
and checking if the numbers increase in the same column if using the touchpad.
I don’t think this is the final solution. As I understand from the thread, the number of generated interrupts is still too high. But this workaround is apparently better than nothing.
the large number of interrupts generated are probably from the i2c controller on the SOC, as each physical interrupt requires the CPU to transfer multiple bytes. A more optimized Linux kernel driver may be able to fix this.
Is there anyone here who’s willing to report this to the kernel folks?
4 Likes
Thanks everybody for the large amount of research that has gone into this.
From reading I understand that pinning interrupts to a certain core is a workaround that some people test with, but the process raises a few questions for me. I hope someone can shed a light here
-
Why only pinning the
designware.2interrupts? On my 11th Gen I seedesignware.{0,1,2}interrupts listed in/proc/interrupts.
What are the other 2 for? -
Why pinning to CPU 2?
When I executelscpu --all --extendedmultiple times over a range of a few minutes I see different cores with a high frequency - including CPU2. According to @Kieran_Levin the high frequency is how to recognize a P core, but if the E vs P core is not constant, the pinning ends up to be a hit & miss, no?
Is there a way to dedicate a CPU as E/P when on battery? -
Why not pinning the
PIXAinterrupts too? Just to be explicit that both interrupts are actually handled by the same CPU.
For some reasion, it is not possible to pin the PIXA interrupt, so only thing we can try is to pin the disignware interrupt and assign it to the same core as the PIXA.
True you can’t pin this interrupt during runtime, but using the kernel param irqaffinity one can tell the kernel which CPUs to use for hardware interrupts.
Using irqaffinity=0,1 I was able to move the PIXA interrupt from CPU 7 to CPU 1.
I haven’t fully tested the impact to power consumption and/ or performance (when charging). There’s also a question how reasonable it is to set irqaffinity to a single CPU core?
Pondering on how to resolve my questions above, the plan seems to look like
- Pin hardware IRQs to 2 (or 1 - this needs more research) CPU using kernels
irqaffinity - Pin
designwareCPU to the same as PIXA, if more than 1 core was listed inirqaffinity, otherwise skip this step - Ensure the CPU used for the touchpad interrupts is an E core when on battery.
Step 3 is the one I was struggling with on my machine (11th Gen Framework), until I stumbled over
https://www.makeuseof.com/intel-cpus-explained-what-are-e-cores-and-p-cores/ :
E/P cores is a feature since Intels 12th Gen chips, which explains why the CPU frequency behaviour @Kieran_Levin explained did not match my observations.
Also found this during research: according to x86 64 - How to detect E-cores and P-cores in Linux alder lake system? - Stack Overflow E cores can be identified (if SMT is enabeld in BIOS) by looking at the CORE column for core unique core-numbers - they represent the E cores in the system.
1 Like
@dfh there is another i2c bus connected to the EC for i2c-hid events that are for some keyboard hotkeys and the ALS.
The other one should be the DDR interface to get the module information.
Both of these are pretty low frequency updates so have diminishing returns, and DDR will not have any data transactions happening on it after boot.
2 Likes
I don’t know what I’m doing and am in way over my head (hoping someone with more knowledge can chime in, or tell me if I’m totally wrong), but I think I may have found why there are so many designware interrupts.
Looking at what is happening in the interrupt handler (in linux/drivers/i2c/busses/i2c-designware-core.h in the funcion i2c_dw_isr with a focus on i2c_dw_read), the system reads 38 bytes from i2c. However it does so at a rate of 2 bytes/interrupt (rx_valid, which dictates the number of bytes to read, is 2). Tap/click events seem to be sent more reasonably with 38 bytes over 3 interrupts (rx_valid is between 7-17). So while there may only be ~140 interrupts/sec for the touchpad, whatever related transfer the designware i2c is doing causes a lot more. Presumably, data transfered from touchpad movements can be buffered more (like it is for clicks) so more is read at once.
rx_valid is read from the register at offset 0x78 (macro DW_IC_RXFLR in i2c-designware-core.h). I’d guess that something like this would be controlled on a firmware level rather than kernel/driver, and is something that the HW is capable of considering there click inputs seem to be handled with higher rx_valid values.
Also, according to dmesg times, sequential reads within the same interrupt are around 5 microseconds (including time for printks and aside from the first 3 reads for larger rx_valid values which are closer to 20 microseconds for some reason), while the time to read between interrupts is around 50-100 microseconds (usually around 50). Besides reducing the number of context switches, this should mean much less time spent on interrupts if there are more reads/IRQ.
Some other findings:
I can’t write to smp_affinity_list for PIXA also, but it’s set to 0-15 (or 10 sometimes?) for me and after a resume (from sleep or hibernate), the interrutps are handled by cpu0 instead of cpu2.
The high number of interrupts when using the touchpad is also on windows. You can measure this with Performance Monitor or run this in powershell: typeperf "\Processor Information(_Total)\Interrupts/sec" "\Processor Information(_Total)\% Interrupt Time". Interrupt time seems to stays pretty low though (usually around 0.12-0.25%).
From perf top, a lot of % kernel time (more than i2c_dw_read) is spent on intel_gpio_irq (in pinctrl-intel.c, with most time spent specifically between the spin lock in intel_gpio_irq_handler. . Related device is INTC1055:00). I’m not sure if that’s normal (when the system is near idle). I also noticed a lot the reads determine that the pins aren’t enabled. This may be a kernel driver optimization, but if there was some way to tell if a pins are disabled before spending the time reading them, that should help (I think it can be configured in the driver since Tiger Lake?). This probably doesn’t help Framework much though it’s in the kernel/drivers.
Assuming I’m not just completely wrong and this helps at all, I can fix up some of my notes and post them on github or something.
4 Likes
Can’t figure out how to edit my previous comment, but I forgot to mention the pretty important part that the kernel only ever reads data from I2C when the devices sends an interrupt saying that it can’t store anything else (as in DW_IC_INTR_RX_FULL is set), thus the kernel has to read the data and there shouldn’t be any optimizations that can be done (unless I’m misunderstanding and the kernel influence when the hardware triggers RX_FULL).