[TRACKING] Test results for standby battery use of Expansion Cards

anarcat · October 13, 2022, 1:49pm

After first testing each expansion card’s power usage at runtime — part of my in-depth review — I have finally gone around doing a similar set of tests on standby.

TL;DR: everything but USB-C take power, USB-A and DisplayPort take a lot more than they logically should.

Here is a summary of those results.

Device	Wattage	Amperage	Days	Note
baseline	0.25W	16mA	9	`sleep=deep nvme.noacpi=1`
s2idle	0.29W	18.9mA	~7	`sleep=s2idle nvme.noacpi=1`
normal nvme	0.31W	20mA	~7	`sleep=s2idle` without `nvme.noacpi=1`
1 USB-C	0.23W	15mA	~10
2 USB-C	0.23W	14.9mA		same as above
1 USB-A	0.75W	48.7mA	3	+500mW (!!) for the first USB-A card!
2 USB-A	1.11W	72mA	2	+360mW
3 USB-A	1.48W	96mA	<2	+370mW
1TB SSD	0.49W	32mA	<5	+260mW
MicroSD	0.52W	34mA	~4	+290mW
DisplayPort	0.85W	55mA	<3	+620mW (!!)
1 HDMI	0.58W	38mA	~4	+250mW
2 HDMI	0.65W	42mA	<4	+70mW (?)

In the above table, all tests were done with the “baseline” configuration (sleep=deep nvme.noacpi=1) unless otherwise noted (i.e. only the second and third rows use sleep=s2idle).

Conclusions

USB-C cards take no extra power on suspend, possibly less than empty slots, more testing required
USB-A cards take a lot more power on suspend (300-500mW) than on runtime (~10mW, almost negligible)
1TB SSD and MicroSD cards seem to take a reasonable amount of power (260-290mW), compared to their runtime equivalents (1-6W!)
DisplayPort takes a surprising lot of power (620mW), almost double its average runtime usage (390mW)
HDMI cards take, surprisingly, less power (250mW) in standby than the DP card (620mW)
and oddly, a second card adds less power usage (70mW?!) than the first, maybe a circuit is used by both?

Test procedure

I have deployed batterylog and my own hack to collect metrics. The actual procedure is:

1. resume
2. add/remove modules, change configuration
3. suspend for 60 minutes
4. collect results

At first, the “suspend” was triggered by hitting the power key, which was reconfigured in logind.conf to make systemd suspend the machine instead of shutdown. The timing is done with a “wall clock” (an Android phone timer, technically), and the actual delay will vary between tests. the actual delay used will be the one provided by the tlp-stat -b entry in the log.

We originally set a timer of 5 minutes to expedite the tests, but this was seeing too little variation in the tlp-stat -b output: the first test led to 2 mAh usage (23mA), but that means the margin of error (±1mAh) is basically half of the measurement. batterylog reported 0.36W and it’s unclear what its margin of error is.

At one hour, we’d expect 12 times the usage (so 24mAh), and therefore a more reasonable 4% margin of error.

Eventually, we found out about the rtcwake binary (part of util-linux) to automate that part of the procedure. At the end, our test procedure was:

date ; rtcwake -m no -s 3600 && systemctl suspend ; date ;\
sleep 10 ; date ; /opt/batterylog/batterylog.py ; \
journalctl -b | grep charge_now | tail -2

The mAh metrics were computed with the excellent qalculate with a formula like this for the simple, ~1h tests:

> (2205mAh−2150mAh)/3601s

  (((2205 * meter) * ampère * heure) - ((2150 * meter) * ampère * heure)) /
  (3601 * seconde) =
  approx. 54,984726 A*m

For longer tests, some creative time diffs were done because qalculate doesn’t support complicated calendar calculations. This run, for example:

oct 12 20:19:01 angela systemd-sleep[149720]: /sys/class/power_supply/BAT1/charge_now                     =   1836 [mAh]
oct 13 08:52:54 angela systemd-sleep[150028]: /sys/class/power_supply/BAT1/charge_now                     =   1636 [mAh]

… would result in the formula:

> (1836mAh-1636mAh)/((20h+52min+54s)-(8h+19min+1s))

(((1836 * meter) * ampère * heure) - ((1636 * meter) * ampère * heure)) /
(((20 * heure) + (52 * minute) + (54 * seconde)) - ((8 * heure) + (19 *
minute) + (1 * seconde))) =
approx. 15,917582 A*m

Tests were performed on a 12th gen Framework laptop with the latest BIOS (3.05, as of 2022-10-13), running Debian bookworm, with powerstat --autotune, and tlp.service enabled. Other tweaks might be relevant here, but see the full review on things that were done to this laptop.

a note on turbostat

I’ve seen elsewhere people using turbostat to profile the suspend mode, but I couldn’t make this work.

This, for example, doesn’t give the results I’d expect:

root@angela:/home/anarcat# turbostat --show GFX%rc6,Pkg%pc2,Pkg%pc3,Pkg%pc6,Pkg%pc7,Pkg%pc8,Pkg%pc9,Pk%pc10,SYS%LPI rtcwake -m freeze -s 3600 ; /opt/batterylog/batterylog.py ; journalctl -b | grep charge_now | tail -2 ;\
> turbostat --show GFX%rc6,Pkg%pc2,Pkg%pc3,Pkg%pc6,Pkg%pc7,Pkg%pc8,Pkg%pc9,Pk%pc10,SYS%LPI systemctl suspend ; /opt/batterylog/batterylog.py ; journalctl -b | grep charge_now | tail -2
turbostat version 2022.04.16 - Len Brown <lenb@kernel.org>
CPUID(0): GenuineIntel 0x20 CPUID levels
CPUID(1): family:model:stepping 0x6:9a:3 (6:154:3) microcode 0x421
CPUID(0x80000000): max_extended_levels: 0x80000008
CPUID(1): SSE3 MONITOR SMX EIST TM2 TSC MSR ACPI-TM HT TM
CPUID(6): APERF, TURBO, DTS, PTM, HWP, HWPnotify, HWPwindow, HWPepp, HWPpkg, EPB
cpu4: MSR_IA32_MISC_ENABLE: 0x00850089 (TCC EIST MWAIT PREFETCH TURBO)
CPUID(7): No-SGX
CPUID(0x15): eax_crystal: 2 ebx_tsc: 110 ecx_crystal_hz: 38400000
TSC: 2112 MHz (38400000 Hz * 110 / 2 / 1000000)
CPUID(0x16): base_mhz: 2100 max_mhz: 4400 bus_mhz: 100
cpu4: MSR_MISC_PWR_MGMT: 0x000030c2 (ENable-EIST_Coordination ENable-EPB DISable-OOB)
RAPL: 9362 sec. Joule Counter Range, at 28 Watts
cpu4: MSR_PLATFORM_INFO: 0x804043df0811500
4 * 100.0 = 400.0 MHz max efficiency frequency
21 * 100.0 = 2100.0 MHz base frequency
cpu4: MSR_IA32_POWER_CTL: 0x00e4005b (C1E auto-promotion: ENabled)
cpu4: MSR_TURBO_RATIO_LIMIT: 0x2323232323252c2c
35 * 100.0 = 3500.0 MHz max turbo 8 active cores
35 * 100.0 = 3500.0 MHz max turbo 7 active cores
35 * 100.0 = 3500.0 MHz max turbo 6 active cores
35 * 100.0 = 3500.0 MHz max turbo 5 active cores
35 * 100.0 = 3500.0 MHz max turbo 4 active cores
37 * 100.0 = 3700.0 MHz max turbo 3 active cores
44 * 100.0 = 4400.0 MHz max turbo 2 active cores
44 * 100.0 = 4400.0 MHz max turbo 1 active cores
cpu4: MSR_CONFIG_TDP_NOMINAL: 0x00000011 (base_ratio=17)
cpu4: MSR_CONFIG_TDP_LEVEL_1: 0x000e00a0 (PKG_MIN_PWR_LVL1=0 PKG_MAX_PWR_LVL1=0 LVL1_RATIO=14 PKG_TDP_LVL1=160)
cpu4: MSR_CONFIG_TDP_LEVEL_2: 0x00150118 (PKG_MIN_PWR_LVL2=0 PKG_MAX_PWR_LVL2=0 LVL2_RATIO=21 PKG_TDP_LVL2=280)
cpu4: MSR_CONFIG_TDP_CONTROL: 0x00000000 ( lock=0)
cpu4: MSR_TURBO_ACTIVATION_RATIO: 0x00000010 (MAX_NON_TURBO_RATIO=16 lock=0)
cpu4: MSR_PKG_CST_CONFIG_CONTROL: 0x74008008 (UNdemote-C1, demote-C1, locked, pkg-cstate-limit=8 (unlimited))
/dev/cpu_dma_latency: 2000000000 usec (default)
current_driver: intel_idle
current_governor: menu
current_governor_ro: menu
cpu4: POLL: CPUIDLE CORE POLL IDLE
cpu4: C1E: MWAIT 0x01
cpu4: C6: MWAIT 0x20
cpu4: C8: MWAIT 0x40
cpu4: C10: MWAIT 0x60
cpu4: cpufreq driver: intel_pstate
cpu4: cpufreq governor: powersave
cpufreq intel_pstate no_turbo: 0
cpu4: MSR_MISC_FEATURE_CONTROL: 0x00000000 (L2-Prefetch L2-Prefetch-pair L1-Prefetch L1-IP-Prefetch)
cpu0: MSR_PM_ENABLE: 0x00000001 (HWP)
cpu0: MSR_HWP_CAPABILITIES: 0x01111638 (high 56 guar 22 eff 17 low 1)
cpu0: MSR_HWP_REQUEST: 0xc0003806 (min 6 max 56 des 0 epp 0xc0 window 0x0 pkg 0x0)
cpu0: MSR_HWP_REQUEST_PKG: 0x8000ff01 (min 1 max 255 des 0 epp 0x80 window 0x0)
cpu0: MSR_HWP_INTERRUPT: 0x00000001 (EN_Guaranteed_Perf_Change, Dis_Excursion_Min)
cpu0: MSR_HWP_STATUS: 0x00000000 (No-Guaranteed_Perf_Change, No-Excursion_Min)
cpu0: EPB: 7 (custom)
cpu0: MSR_RAPL_POWER_UNIT: 0x000a0e03 (0.125000 Watts, 0.000061 Joules, 0.000977 sec.)
cpu0: MSR_PKG_POWER_INFO: 0x000000e0 (28 W TDP, RAPL 0 - 0 W, 0.000000 sec.)
cpu0: MSR_PKG_POWER_LIMIT: 0x42820000dd80f0 (UNlocked)
cpu0: PKG Limit #1: ENabled (30.000 Watts, 28.000000 sec, clamp ENabled)
cpu0: PKG Limit #2: ENabled (64.000 Watts, 0.002441* sec, clamp DISabled)
cpu0: MSR_VR_CURRENT_CONFIG: 0x000002d0
cpu0: PKG Limit #4: 90.000000 Watts (UNlocked)
cpu0: MSR_DRAM_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: DRAM Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_PP0_POLICY: 9
cpu0: MSR_PP0_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: Cores Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_PP1_POLICY: 13
cpu0: MSR_PP1_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: GFX Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x80640000 (100 C)
cpu0: MSR_IA32_PACKAGE_THERM_STATUS: 0x88420000 (34 C)
cpu0: MSR_IA32_PACKAGE_THERM_INTERRUPT: 0x02000003 (100 C, 100 C)
cpu4: MSR_PKGC3_IRTL: 0x00000000 (NOTvalid, 0 ns)
cpu4: MSR_PKGC6_IRTL: 0x00000000 (NOTvalid, 0 ns)
cpu4: MSR_PKGC7_IRTL: 0x00000000 (NOTvalid, 0 ns)
cpu4: MSR_PKGC8_IRTL: 0x00000000 (NOTvalid, 0 ns)
cpu4: MSR_PKGC9_IRTL: 0x00000000 (NOTvalid, 0 ns)
cpu4: MSR_PKGC10_IRTL: 0x00000000 (NOTvalid, 0 ns)
rtcwake: assuming RTC uses UTC ...
rtcwake: wakeup from "freeze" using /dev/rtc0 at Wed Oct 12 03:03:31 2022
3601.483927 sec
GFX%rc6 Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 Pkg%pc8 Pkg%pc9 Pk%pc10 SYS%LPI
15.63   0.03    0.01    0.00    0.00    0.00    0.00    100.50  99.80
15.63   0.03    0.01    0.00    0.00    0.00    0.00    100.50  99.80
Slept for 1.01 hours
Used 0.75 Wh, an average rate of 0.75 W
For your 54.53 Wh battery this is 1.38%/hr or 33.03%/day
oct 11 20:19:44 angela systemd-sleep[12975]: /sys/class/power_supply/BAT1/charge_now                     =   3284 [mAh]
oct 11 21:20:04 angela systemd-sleep[13222]: /sys/class/power_supply/BAT1/charge_now                     =   3235 [mAh]
turbostat version 2022.04.16 - Len Brown <lenb@kernel.org>
CPUID(0): GenuineIntel 0x20 CPUID levels
CPUID(1): family:model:stepping 0x6:9a:3 (6:154:3) microcode 0x421
CPUID(0x80000000): max_extended_levels: 0x80000008
CPUID(1): SSE3 MONITOR SMX EIST TM2 TSC MSR ACPI-TM HT TM
CPUID(6): APERF, TURBO, DTS, PTM, HWP, HWPnotify, HWPwindow, HWPepp, HWPpkg, EPB
cpu1: MSR_IA32_MISC_ENABLE: 0x00850089 (TCC EIST MWAIT PREFETCH TURBO)
CPUID(7): No-SGX
CPUID(0x15): eax_crystal: 2 ebx_tsc: 110 ecx_crystal_hz: 38400000
TSC: 2112 MHz (38400000 Hz * 110 / 2 / 1000000)
CPUID(0x16): base_mhz: 2100 max_mhz: 4400 bus_mhz: 100
cpu1: MSR_MISC_PWR_MGMT: 0x000030c2 (ENable-EIST_Coordination ENable-EPB DISable-OOB)
RAPL: 9362 sec. Joule Counter Range, at 28 Watts
cpu1: MSR_PLATFORM_INFO: 0x804043df0811500
4 * 100.0 = 400.0 MHz max efficiency frequency
21 * 100.0 = 2100.0 MHz base frequency
cpu1: MSR_IA32_POWER_CTL: 0x00e4005b (C1E auto-promotion: ENabled)
cpu1: MSR_TURBO_RATIO_LIMIT: 0x2323232323252c2c
35 * 100.0 = 3500.0 MHz max turbo 8 active cores
35 * 100.0 = 3500.0 MHz max turbo 7 active cores
35 * 100.0 = 3500.0 MHz max turbo 6 active cores
35 * 100.0 = 3500.0 MHz max turbo 5 active cores
35 * 100.0 = 3500.0 MHz max turbo 4 active cores
37 * 100.0 = 3700.0 MHz max turbo 3 active cores
44 * 100.0 = 4400.0 MHz max turbo 2 active cores
44 * 100.0 = 4400.0 MHz max turbo 1 active cores
cpu1: MSR_CONFIG_TDP_NOMINAL: 0x00000011 (base_ratio=17)
cpu1: MSR_CONFIG_TDP_LEVEL_1: 0x000e00a0 (PKG_MIN_PWR_LVL1=0 PKG_MAX_PWR_LVL1=0 LVL1_RATIO=14 PKG_TDP_LVL1=160)
cpu1: MSR_CONFIG_TDP_LEVEL_2: 0x00150118 (PKG_MIN_PWR_LVL2=0 PKG_MAX_PWR_LVL2=0 LVL2_RATIO=21 PKG_TDP_LVL2=280)
cpu1: MSR_CONFIG_TDP_CONTROL: 0x00000000 ( lock=0)
cpu1: MSR_TURBO_ACTIVATION_RATIO: 0x00000010 (MAX_NON_TURBO_RATIO=16 lock=0)
cpu1: MSR_PKG_CST_CONFIG_CONTROL: 0x74008008 (UNdemote-C1, demote-C1, locked, pkg-cstate-limit=8 (unlimited))
/dev/cpu_dma_latency: 2000000000 usec (default)
current_driver: intel_idle
current_governor: menu
current_governor_ro: menu
cpu1: POLL: CPUIDLE CORE POLL IDLE
cpu1: C1E: MWAIT 0x01
cpu1: C6: MWAIT 0x20
cpu1: C8: MWAIT 0x40
cpu1: C10: MWAIT 0x60
cpu1: cpufreq driver: intel_pstate
cpu1: cpufreq governor: powersave
cpufreq intel_pstate no_turbo: 0
cpu1: MSR_MISC_FEATURE_CONTROL: 0x00000000 (L2-Prefetch L2-Prefetch-pair L1-Prefetch L1-IP-Prefetch)
cpu0: MSR_PM_ENABLE: 0x00000001 (HWP)
cpu0: MSR_HWP_CAPABILITIES: 0x011c1638 (high 56 guar 22 eff 28 low 1)
cpu0: MSR_HWP_REQUEST: 0xc0003806 (min 6 max 56 des 0 epp 0xc0 window 0x0 pkg 0x0)
cpu0: MSR_HWP_REQUEST_PKG: 0x8000ff01 (min 1 max 255 des 0 epp 0x80 window 0x0)
cpu0: MSR_HWP_INTERRUPT: 0x00000001 (EN_Guaranteed_Perf_Change, Dis_Excursion_Min)
cpu0: MSR_HWP_STATUS: 0x00000000 (No-Guaranteed_Perf_Change, No-Excursion_Min)
cpu0: EPB: 7 (custom)
cpu0: MSR_RAPL_POWER_UNIT: 0x000a0e03 (0.125000 Watts, 0.000061 Joules, 0.000977 sec.)
cpu0: MSR_PKG_POWER_INFO: 0x000000e0 (28 W TDP, RAPL 0 - 0 W, 0.000000 sec.)
cpu0: MSR_PKG_POWER_LIMIT: 0x42820000dd80f0 (UNlocked)
cpu0: PKG Limit #1: ENabled (30.000 Watts, 28.000000 sec, clamp ENabled)
cpu0: PKG Limit #2: ENabled (64.000 Watts, 0.002441* sec, clamp DISabled)
cpu0: MSR_VR_CURRENT_CONFIG: 0x000002d0
cpu0: PKG Limit #4: 90.000000 Watts (UNlocked)
cpu0: MSR_DRAM_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: DRAM Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_PP0_POLICY: 9
cpu0: MSR_PP0_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: Cores Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_PP1_POLICY: 13
cpu0: MSR_PP1_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: GFX Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x80640000 (100 C)
cpu0: MSR_IA32_PACKAGE_THERM_STATUS: 0x884f0000 (21 C)
cpu0: MSR_IA32_PACKAGE_THERM_INTERRUPT: 0x02000003 (100 C, 100 C)
cpu1: MSR_PKGC3_IRTL: 0x00000000 (NOTvalid, 0 ns)
cpu1: MSR_PKGC6_IRTL: 0x00000000 (NOTvalid, 0 ns)
cpu1: MSR_PKGC7_IRTL: 0x00000000 (NOTvalid, 0 ns)
cpu1: MSR_PKGC8_IRTL: 0x00000000 (NOTvalid, 0 ns)
cpu1: MSR_PKGC9_IRTL: 0x00000000 (NOTvalid, 0 ns)
cpu1: MSR_PKGC10_IRTL: 0x00000000 (NOTvalid, 0 ns)
0.012176 sec
GFX%rc6 Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 Pkg%pc8 Pkg%pc9 Pk%pc10 SYS%LPI
82.13   0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
90.10   0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
Slept for 1.01 hours
Used 0.75 Wh, an average rate of 0.75 W
For your 54.53 Wh battery this is 1.38%/hr or 33.03%/day
oct 11 20:19:44 angela systemd-sleep[12975]: /sys/class/power_supply/BAT1/charge_now                     =   3284 [mAh]
oct 11 21:20:04 angela systemd-sleep[13222]: /sys/class/power_supply/BAT1/charge_now                     =   3235 [mAh]

See how the second suspend didn’t correctly run? It looks like rtcwake doesn’t correctly trigger the systemd hooks. And indeed, the above it doesn’t even go in the correct sleep more (freeze instead of mem).

Another run, with mem:

root@angela:/home/anarcat# turbostat --show GFX%rc6,Pkg%pc2,Pkg%pc3,Pkg%pc6,Pkg%pc7,Pkg%pc8,Pkg%pc9,Pk%pc10,SYS%LPI rtcwake -m mem -s 3600 ; /opt/batterylog/batterylog.py ; journalctl -b | grep charge_now | tail -2
turbostat version 2022.04.16 - Len Brown <lenb@kernel.org>
CPUID(0): GenuineIntel 0x20 CPUID levels
CPUID(1): family:model:stepping 0x6:9a:3 (6:154:3) microcode 0x421
CPUID(0x80000000): max_extended_levels: 0x80000008
CPUID(1): SSE3 MONITOR SMX EIST TM2 TSC MSR ACPI-TM HT TM
CPUID(6): APERF, TURBO, DTS, PTM, HWP, HWPnotify, HWPwindow, HWPepp, HWPpkg, EPB
cpu6: MSR_IA32_MISC_ENABLE: 0x00850089 (TCC EIST MWAIT PREFETCH TURBO)
CPUID(7): No-SGX
CPUID(0x15): eax_crystal: 2 ebx_tsc: 110 ecx_crystal_hz: 38400000
TSC: 2112 MHz (38400000 Hz * 110 / 2 / 1000000)
CPUID(0x16): base_mhz: 2100 max_mhz: 4400 bus_mhz: 100
cpu6: MSR_MISC_PWR_MGMT: 0x000030c2 (ENable-EIST_Coordination ENable-EPB DISable-OOB)
RAPL: 9362 sec. Joule Counter Range, at 28 Watts
cpu6: MSR_PLATFORM_INFO: 0x804043df0811500
4 * 100.0 = 400.0 MHz max efficiency frequency
21 * 100.0 = 2100.0 MHz base frequency
cpu6: MSR_IA32_POWER_CTL: 0x00e4005b (C1E auto-promotion: ENabled)
cpu6: MSR_TURBO_RATIO_LIMIT: 0x2323232323252c2c
35 * 100.0 = 3500.0 MHz max turbo 8 active cores
35 * 100.0 = 3500.0 MHz max turbo 7 active cores
35 * 100.0 = 3500.0 MHz max turbo 6 active cores
35 * 100.0 = 3500.0 MHz max turbo 5 active cores
35 * 100.0 = 3500.0 MHz max turbo 4 active cores
37 * 100.0 = 3700.0 MHz max turbo 3 active cores
44 * 100.0 = 4400.0 MHz max turbo 2 active cores
44 * 100.0 = 4400.0 MHz max turbo 1 active cores
cpu6: MSR_CONFIG_TDP_NOMINAL: 0x00000011 (base_ratio=17)
cpu6: MSR_CONFIG_TDP_LEVEL_1: 0x000e00a0 (PKG_MIN_PWR_LVL1=0 PKG_MAX_PWR_LVL1=0 LVL1_RATIO=14 PKG_TDP_LVL1=160)
cpu6: MSR_CONFIG_TDP_LEVEL_2: 0x00150118 (PKG_MIN_PWR_LVL2=0 PKG_MAX_PWR_LVL2=0 LVL2_RATIO=21 PKG_TDP_LVL2=280)
cpu6: MSR_CONFIG_TDP_CONTROL: 0x00000000 ( lock=0)
cpu6: MSR_TURBO_ACTIVATION_RATIO: 0x00000010 (MAX_NON_TURBO_RATIO=16 lock=0)
cpu6: MSR_PKG_CST_CONFIG_CONTROL: 0x74008008 (UNdemote-C1, demote-C1, locked, pkg-cstate-limit=8 (unlimited))
/dev/cpu_dma_latency: 2000000000 usec (default)
current_driver: intel_idle
current_governor: menu
current_governor_ro: menu
cpu6: POLL: CPUIDLE CORE POLL IDLE
cpu6: C1E: MWAIT 0x01
cpu6: C6: MWAIT 0x20
cpu6: C8: MWAIT 0x40
cpu6: C10: MWAIT 0x60
cpu6: cpufreq driver: intel_pstate
cpu6: cpufreq governor: powersave
cpufreq intel_pstate no_turbo: 0
cpu6: MSR_MISC_FEATURE_CONTROL: 0x00000000 (L2-Prefetch L2-Prefetch-pair L1-Prefetch L1-IP-Prefetch)
cpu0: MSR_PM_ENABLE: 0x00000001 (HWP)
cpu0: MSR_HWP_CAPABILITIES: 0x01141638 (high 56 guar 22 eff 20 low 1)
cpu0: MSR_HWP_REQUEST: 0xc0003806 (min 6 max 56 des 0 epp 0xc0 window 0x0 pkg 0x0)
cpu0: MSR_HWP_REQUEST_PKG: 0x8000ff01 (min 1 max 255 des 0 epp 0x80 window 0x0)
cpu0: MSR_HWP_INTERRUPT: 0x00000001 (EN_Guaranteed_Perf_Change, Dis_Excursion_Min)
cpu0: MSR_HWP_STATUS: 0x00000000 (No-Guaranteed_Perf_Change, No-Excursion_Min)
cpu0: EPB: 7 (custom)
cpu0: MSR_RAPL_POWER_UNIT: 0x000a0e03 (0.125000 Watts, 0.000061 Joules, 0.000977 sec.)
cpu0: MSR_PKG_POWER_INFO: 0x000000e0 (28 W TDP, RAPL 0 - 0 W, 0.000000 sec.)
cpu0: MSR_PKG_POWER_LIMIT: 0x42820000dd80f0 (UNlocked)
cpu0: PKG Limit #1: ENabled (30.000 Watts, 28.000000 sec, clamp ENabled)
cpu0: PKG Limit #2: ENabled (64.000 Watts, 0.002441* sec, clamp DISabled)
cpu0: MSR_VR_CURRENT_CONFIG: 0x000002d0
cpu0: PKG Limit #4: 90.000000 Watts (UNlocked)
cpu0: MSR_DRAM_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: DRAM Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_PP0_POLICY: 9
cpu0: MSR_PP0_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: Cores Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_PP1_POLICY: 13
cpu0: MSR_PP1_POWER_LIMIT: 0x00000000 (UNlocked)
cpu0: GFX Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled)
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x80640000 (100 C)
cpu0: MSR_IA32_PACKAGE_THERM_STATUS: 0x884b0000 (25 C)
cpu0: MSR_IA32_PACKAGE_THERM_INTERRUPT: 0x02000003 (100 C, 100 C)
cpu6: MSR_PKGC3_IRTL: 0x00000000 (NOTvalid, 0 ns)
cpu6: MSR_PKGC6_IRTL: 0x00000000 (NOTvalid, 0 ns)
cpu6: MSR_PKGC7_IRTL: 0x00000000 (NOTvalid, 0 ns)
cpu6: MSR_PKGC8_IRTL: 0x00000000 (NOTvalid, 0 ns)
cpu6: MSR_PKGC9_IRTL: 0x00000000 (NOTvalid, 0 ns)
cpu6: MSR_PKGC10_IRTL: 0x00000000 (NOTvalid, 0 ns)
rtcwake: assuming RTC uses UTC ...
rtcwake: wakeup from "mem" using /dev/rtc0 at Wed Oct 12 14:11:02 2022

264.737865 sec
GFX%rc6 Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 Pkg%pc8 Pkg%pc9 Pk%pc10 SYS%LPI
0.84    1609.15 1609.15 1609.15 0.00    1609.15 0.00    1609.15 6967928093745.08
0.84    100.57  100.57  100.57  0.00    100.57  0.00    100.57  6968124499895.82
Slept for 9.04 hours
Used 2.09 Wh, an average rate of 0.23 W
For your 54.53 Wh battery this is 0.42%/hr or 10.20%/day
oct 11 23:03:32 angela systemd-sleep[42344]: /sys/class/power_supply/BAT1/charge_now                     =   2969 [mAh]
oct 12 08:05:52 angela systemd-sleep[42611]: /sys/class/power_supply/BAT1/charge_now                     =   2833 [mAh]

That seems to have worked, but then the actual numbers from turbostat are not really that meaningful. It seems we’re using 100% of the CPU there, and millions of percent of SYS. Anyways, I don’t understand how turbostat could be used that way anyways, since, well, it’s suspended…

assumptions

we assume standby battery usage doesn’t fluctuate depending on the level of the battery; as of this writing, the battery is at 95% with an estimated 10h remaining
we assume we will get similar results than runtime power usage, that is the SSD and MicroSD expansion cards use a lot of power, HDMI and DisplayPort a little, and USB-A the less, while USB-C should use none at all
we have heard theories that USB-A uses a lot more power on suspend than at runtime
we assume deep sleep provides better power saving
we expect to get close to 14mA drain on suspend

Previous work

The previous results (from @Nils) were:

System on, locked and screen off, with HDMI & USB expansion cards: around 3W
s2idle sleep with USB-A and HDMI expansion cards: around 1.9 W
s2idle sleep with only USB-C expansion cards: around 0.8 W
deep sleep with USB-A and HDMI expansion cards: around 0.9 W
deep sleep with only USB-C expansion cards: around 0.4 W

Detailed log at this page which also includes parts of the above narrative.

idle tests

Here is a copy of the results I had for idle power consumption tests:

Device	Minimum	Average	Max	Stdev	Note
Screen, 100%	2.4W	2.6W	2.8W	N/A
Screen, 1%	30mW	140mW	250mW	N/A
Radios	100mW	250mW	N/A	N/A
USB-C	N/A	N/A	N/A	N/A	negligible power drain
USB-A	10mW	10mW	?	10mW	almost negligible
DisplayPort	300mW	390mW	600mW	N/A	not passive
HDMI	380mW	440mW	1W?	20mW	not passive
1TB SSD	1.65W	1.79W	2W	12mW	significant, probably higher when busy
MicroSD	1.6W	3W	6W	1.93W	highest power usage, possibly even higher when busy

… for comparison with the above. Notice, in particular, how the USB-A cards only take 10mW here, as opposed to 300mW+ during suspend. The results of the DisplayPort card are also a little disappointing.

Future work

turn USB ports on suspend?

The next step, for me, is to figure out how to plug those leaks. Surely there is a way to turn off those USB sticks before suspend and reactivate them on resume. In that other [TRACKING] High Battery Drain During Suspend thread, there was this suggestion:

I originally thought this was tried in the thread and it didn’t work, but now I can’t find this anymore, so I guess that’s the next step.

Support?

I would love to hear from the @Framework team what’s up with this as well. Back in April, @Daouadi_Philippe contacted Framework support and their answer was:

What’s the status now, 6 months later? I am aware of the beta update to the DP card firmware, and would be happy to try to reproduce the results with this test, provided with there is a way to do it in Linux.

I find it pretty dramatic that USB-A cards, in particular, would use so much power, especially considering they use so little in standby. For me that’s the biggest deal-breaker, as I still have USB-A devices I use on a daily basis… The DisplayPort results are also troubling, also because they use much less during suspend.

I am also available for further tests with the current (or donated!) hardware.

Thanks, and I hope this helps!

Nils · October 13, 2022, 3:56pm

Thank you for the excellent work! A few comments:

if there are software mitigations possible then it’s likely kernel configuration and daemon setup can affect these. Can you list what you’re using? (TLP, kernel version, distro, perhaps how much has been tweaked from standard configuration)
can you clarify for each entry or each table whether the results are for ‘deep’ or for ‘s2idle’?
can you clarify if with ‘idle’ you mean ‘system on but minimal activity’ (screen blanked/locked perhaps) or whether you mean ‘s2idle’ suspend?

anarcat · October 13, 2022, 4:01pm

The kernel is the latest in Debian bookworm, which at the time of writing, is 5.19.11 I believe. See this page for the latest.

I have powertop autotuned, tlp.service running, and tried every trick in the book you can see in this review.

I thought the table was pretty clear, but unless otherwise noted, all tests are done in deep mode. The second and third lines are the only ones in s2idle.

I guess that is a little ambiguous, but you should assume “idle” means “computer powered on but not doing any special processing” and “s2idle” means “computer suspended in s2idle mode”. If you could spot more precisely which place is ambiguous for you, i’d be happy to fix the summary.

Nils · October 13, 2022, 4:26pm

Thanks. The main ambiguity to me was whether the difference in marginal power use for USB-A was between deep and s2idle or between deep and idle. From what you describe it’s the latter, and s2idle would likely show a similar margin as deep.

In a way that makes sense if indeed something in the cards prevents something internal on the motherboard to power down properly.The motherboard component is probably supposed to be on whenever the system is on (also when it’s idle), so the difference only really shows when suspended (with deep or s2idle).

Your figures also suggest that 12th gen motherboard is significantly better for suspend, or that recent kernel development has significantly improved suspend power usage.

anarcat · October 13, 2022, 4:31pm

I see. Yes, the difference in USB-A power usage is really between suspend and runtime. Adding a USB-A card while the machine is running takes up an extra 10mW, while adding it during sleep (deep, haven’t tested with s2idle specifically) takes up hundreds of miliwatts (300-500mW).

Update: I have tried to clarify by adding a comment under the table, and by ditching the ambiguous “idle” in the conclusions.

Ah that’s an interesting theory. It would explain the difference between one USB-A card (+500mW) and 2/3/n USB-A cards (+360mW each).

Still, the extra 140mW there (500mW for one minus 360mW for additional) is less than the cost of a single one, so I can’t help but wonder that there’s something specifically bad with the USB-A firmware.

Ah, that’s really nice to hear. It would be interesting to see you test the latest kernels to see if you can get similar results as mine in the 11th gen.

lhl · October 13, 2022, 5:47pm

Awesome testing, and glad that batterylog was useful. BTW, for shutting off USB here’s a thread w/ a script that another user built that may work for shutting off USB devices on suspension: [Guide] Automatically disable USB devices for battery savings

Sadly, I don’t think this will work for the USB-A ports, as they don’t show up in lsusb at all afaict. I believe that there’s probably something going on with those plugged in that are keeping the TB4 active when it shouldn’t be or something. @nrp mentioned that the Framework Chromebook was able to pass Google’s battery requirements of 14 days of standby and that they learned a lot about Intel’s [Burnside Bridge TB4] re-timers and that this improved behavior is being backported as a firmware update.

14 days == 336 hours. If it was able to pass that test, on a 55Wh battery, that means that the Framework Chromebook was able to average <163 mW of power drain while in suspend. (I’m also really interested in whether an updated firmware could really get the Framework to >10h of active usage - that would be <5.5W avg under whatever Google defines as “common tasks”).

anarcat · October 13, 2022, 6:01pm

Thanks, I hadn’t seen that thread (so many threads!). I guess that won’t work for the SSD card since that would crash the filesystem (assuming you can’t unmount…) But still worth a shot.

@Nils had the same thought in:

It’s really too bad because fixing USB-A is my primary objective here… If i can’t turn off those cards, it means I need to swap them in and out all the time, which makes me wonder what the spec is for those expansion ports, in terms “how many times can i put things in and out until it breaks apart”… I bet it’s not as good as we might like!

This is what is so frustrating here; it looks like the Framework team did figure out a lot of that stuff, but it’s being kept somewhat secret or at least in limited distribution (e.g. to the Chromebook version).

Is the firmware update you’re refering to the DisplayPort beta or is it at the system level?

I’m not that far from this metric in my tests: with everything turned off (well, with USB-C cards, actually, turns out that helps, weirdly), i’m at 0.23W or 230mW… So close! It’s 10 days, which, for my use case, is pretty good… The problem is the second I add anything on there (and especially USB-A, which I use on a daily basis), it crashes down to 3 days (or explodes to 1W+ depending on your perspective)…

I bet it could. I can actually get > 10h by tweaking stuff a bit and holding my breath while powerstat runs. My original power tests gave me 2.5W idle (no cards, again), and I’d typically see 6-9W usage under moderate load (includes playing music, for example).

But start doing anything though, and things go sort of bonkers… I haven’t tested playing videos with, say, a simple player like mpv, but playing videos in Youtube already jumps you past 9W easily. Playing music in a web player like Navidrome threw me up to 12W… So yeah, tricky stuff, power management: you need to manage the entire stack otherwise you leak everything like a sieve everywhere.

Kind of fascinating…

lhl · October 13, 2022, 6:23pm

I don’t think it’s being kept secret just that it’ll take time to backport. Also, I think you’re looking at it from the wrong perspective - the Framework team learned (presumably undocumented, or at the very least non-obvious) tips and tricks from Google’s more experienced team during the course of co-development - stuff that they might never have figured out otherwise (Google has the learnings from having released dozens of devices with basically every single major OEM over the past decade+). That Framework is now able to take these learnings and improve their core products is a huge win.

Yeah, I could get it down to <2.5W on console idling doing absolutely nothing (ready to advertise >20h of battery life, lol), but as soon as you get out of spending 99% time in C10, power usage skyrockets. Even idle at C8 seems to draw 4-5W. For very light usage (text typing, very light web pages) I can average under 10W, but video playback (mpv of yt-dlp’d videos or YouTube w/ VA-API acceleration) are all >10W at 200 nits.

anarcat · October 13, 2022, 6:45pm

Good point. It just feels like the non-Android Linux users are often left in the sidelines here. But you’re right, a lot of this good stuff is trickling down to us, even if Android is really not Linux all that much.

That sums up my experience as well. Is that with the 12th gen?

Update: just browsing around lightly, editing files and reading/answering mails in Emacs looks like this:

  Time    User  Nice   Sys  Idle    IO  Run Ctxt/s  IRQ/s Fork Exec Exit  Watts
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------
 Average   1.7   0.0   0.5  97.6   0.2  1.2 4684.9 1985.2 126.6 39.1 128.0   7.57
 GeoMean   1.4   0.0   0.4  97.6   0.1  1.2 4416.6 1734.5 111.6 27.9 113.3   7.54
  StdDev   1.0   0.2   0.2   1.2   0.0  0.5 1584.7 1058.3 82.1 44.0 80.2   0.71
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------
 Minimum   0.2   0.0   0.2  94.9   0.1  1.0 2242.0  698.2 82.0 17.0 82.0   6.36
 Maximum   4.1   1.1   1.0  99.4   0.2  3.0 8687.4 4445.1 463.0 249.0 449.0   9.10
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------
Summary:
System:   7.57 Watts on average with standard deviation 0.71

That’s with 2xUSB-C and 1xUSB-A…

Interestingly, playing a video in a (720p) window in a window takes up more power (10.5W) than in full screen (9.5W) but I blame that on my desktop setup (i3 + compton)… Not sure if mpv hits VA-API. Similar results with 1080p, interestingly, except the window struggles to keep up. Full screen fine, and at a comfortable 9.5W, which means a solid 5h+ of playback, fine by me!

Update 2: Fooling around the web, small edits, youtube-dl, and i’m still at around 80% battery… i had a 7h remaining estimate before I started goofing around Discourse, so I suspect this very website is a pretty big battery drain, actually. I see about 10 W, while I was probably at half that just playing music with mpv in the background…

Confirmed: it looks like editing this website in discourse takes a solid 4-6W. Amazing.

Heisenbug? Or schrödinbug?

Joshua_Ham · October 13, 2022, 9:38pm

I don’t think that is a problem, because the USB c port doesn’t need to hold the cards. The plastic lock holds them in and if that wears out, then you can just replace the plastic piece.

Azure · October 13, 2022, 11:41pm

Awesome testing and beautiful write-up! Only thing I’d add is that I assume you did your testing on BIOS 3.10, but it would be nice to have that noted in your post. I’d also love to see if the DisplayPort Beta firmware does for your results if anything. Either way, thanks for doing all this testing and sharing your results with the community!

anarcat · October 14, 2022, 12:39am

I would assume that the USB-C socket itself has a limited lifecycle. Sure, you can replace the whole case or tweak it, or replace the card, but if the USB-C socket itself gets screwed, you basically need to change the whole board.

anarcat · October 14, 2022, 12:50am

Thanks! Nice to hear that!

That’s an excellent question, and I don’t think so! The 12th gen, from what I understand, ships with the latest BIOS version, there is no available update, at least. Mysteriously, however, it seems I’m running version 3.05, not 3.10:

anarcat@angela:~$ sudo lshw | sed -n '/BIOS/,/version/{/version/p}'
          version: 03.05

Even more bizarre, the above BIOS information page says the latest version is… 3.04!

That said, it seems you might be talking about the 11th gen BIOS, which does have a 3.10 release… It looks like those are two different BIOSes with different release cycles…

Update: I added those details to the post anyways.

… so do I, but unfortunately I don’t have the means to update the firmware on that card here (no Windows, nor do I want one). I’d be happy to test an updated card if shipped, same with the ethernet card or any other card people would fancy.

lhl · October 14, 2022, 2:27am

While ChromeOS is it’s own slightly closed offshoot, it’s actually much closer to “Linux” than Android - it runs much closer to mainline kernels w/ lots of upstream interaction (currently on 5.15), and is built on Gentoo/portage+chroots (w/ a custom DE, but not a completely custom userspace). Android apps on Chromebooks are actually run through a container (ARCVM).

Yep, a 1260P w/ 2x32GB DDR4-3200 and a 4TB Phison PCIe4 Gen4 drive.

Eric_Crawford-Anton · October 14, 2022, 8:13am

A third party developers found security issues with 3.05 for the 12th gen BIOS. So they haven’t released it yet until they fix it.

Azure · October 14, 2022, 11:50pm

There’s the detail I missed! Didn’t realize this was on the 12th gen board! Thanks for clarifying. They do indeed have separate BIOS versions for the two laptops.

So glad to see it always good for future users to have this context if they view your post in the future!

Cipher · October 15, 2022, 7:44pm

Thank you for doing this detailed testing on expansion cards @anarcat, very much appreciated.
I was aware of issues with HDMI, Displayport, MicroSD and storage cards having high power draw, which is why I only ordered 1 HDMI and 1 MicroSD expansion card for occasional use.
USB A using any more power than USB C is very disappointing to see. I would love to understand why this might be the case.

Animesh_Sahu · October 16, 2022, 1:40am

Are you sure USB-A has no effect in idle? I seem to have 0.5W bump every new port.

anarcat · October 16, 2022, 1:59am

i’m not 100% sure, but it’s what i have seen in my tests, yes. See the USB-A tests. Are you on a 12th gen?

Nils · October 16, 2022, 2:09am

Perhaps someone with more electronic expertise can provide the appropriate references. From what I understand, the power delivery protocols between USB-A and USB-C are different and while there doesn’t need to be any active electronic components in an adapter, a USB-A adapter needs to connect some leads with a specific resistance in order to signal that power may need to be provided. From what I understand, having that connection forces something in the USB-C interface on the motherboard to change its activity level. For a motherboard that’s on this doesn’t make too much of a difference, but apparently if the system is in suspended state (s2idle or deep) then this difference is measurable.