Any chance you can compile a kernel with the patches to cros_ec that let you access the EC debug log?
I think that’s the next clue here on where the EC bug is.
I can build a patched kernel, searched here for cros_ec patch but couldn’t find them.
Edit: wait is this still the correct one?
I’ve booted into a patched 6.7.4 kernel, how/what to debug when this happens again?
Get a copy of the EC tool that @DHowett has mentioned in his blog and various posts.
Get familiar with using it when there isnt a problem (like how to launch it and view the EC debug log).
Then hopefully next time this happens you can run it and capture exactly what it says is going on.
It will be a clue for the Framework firmware guys to look at for understanding it.
I think I’m set up for debugging: cros_ec kernel patches applied (V2 from the one I highlighted here, and ectool built from the hx20-hx30 branch of FrameworkComputer/EmbeddedController.
I think this means it works?
ectool console output
$ sudo ./build/bds/util/ectool console
Place your right index finger on the fingerprint reader
quest(16632mV, 0mA)]
[2996894.945000 charge_request(16624mV, 0mA)]
[2996897.698800 charge_request(16632mV, 0mA)]
[2996897.947700 charge_request(16624mV, 0mA)]
PORT80: 3C01
PORT80: 3C08
PORT80: 3C01
PORT80: 0DB6
[2996909.714000 charge_request(16632mV, 0mA)]
[2996909.964600 charge_request(16624mV, 0mA)]
[2996911.716100 charge_request(16632mV, 0mA)]
[2996911.966300 charge_request(16624mV, 0mA)]
[2996912.718400 charge_request(16632mV, 0mA)]
[2996912.968900 charge_request(16624mV, 0mA)]
PORT80: 3C08
[2996918.726200 charge_request(16632mV, 0mA)]
[2996918.976800 charge_request(16624mV, 0mA)]
[2996922.731400 charge_request(16632mV, 0mA)]
[2996922.980600 charge_request(16624mV, 0mA)]
[2996926.728700 charge_request(16632mV, 0mA)]
[2996926.987000 charge_request(16624mV, 0mA)]
PORT80: 3C01
[2996930.740500 charge_request(16632mV, 0mA)]
[2996930.991600 charge_request(16624mV, 0mA)]
PORT80: 3C08
[2996933.650400 HC 0x0115 err 1]
PORT80: F90E
PORT80: 3C01
[2996941.142900 HC 0x0002]
[2996941.145100 HC 0x000b]
[2996941.146400 HC 0x0002]
[2996941.148500 HC 0x0004]
[2996941.754200 charge_request(16632mV, 0mA)]
[2996942.000700 charge_request(16624mV, 0mA)]
PORT80: 3C08
[2996943.754900 charge_request(16632mV, 0mA)]
[2996944.007200 charge_request(16624mV, 0mA)]
[2996954.769700 charge_request(16632mV, 0mA)]
[2996955.021000 charge_request(16624mV, 0mA)]
[2996956.773200 charge_request(16632mV, 0mA)]
[2996957.023300 charge_request(16624mV, 0mA)]
PORT80: 3C01
PORT80: 3C08
[2996962.780500 charge_request(16632mV, 0mA)]
[2996963.029800 charge_request(16624mV, 0mA)]
[2996963.779600 charge_request(16632mV, 0mA)]
[2996964.032400 charge_request(16624mV, 0mA)]
[2996965.784600 charge_request(16632mV, 0mA)]
[2996966.034800 charge_request(16624mV, 0mA)]
[2996967.786400 charge_request(16632mV, 0mA)]
[2996968.033600 charge_request(16624mV, 0mA)]
[2996968.787600 charge_request(16632mV, 0mA)]
[2996969.038700 charge_request(16624mV, 0mA)]
[2996973.792800 charge_request(16632mV, 0mA)]
[2996974.043300 charge_request(16624mV, 0mA)]
[2996975.797000 charge_request(16632mV, 0mA)]
[2996976.047200 charge_request(16624mV, 0mA)]
[2996977.799200 charge_request(16632mV, 0mA)]
[2996978.049500 charge_request(16624mV, 0mA)]
[2996978.800700 charge_request(16632mV, 0mA)]
[2996979.049600 charge_request(16624mV, 0mA)]
[2996981.802600 charge_request(16632mV, 0mA)]
[2996982.055100 charge_request(16624mV, 0mA)]
PORT80: 3C01
[2996982.804900 charge_request(16632mV, 0mA)]
[2996983.056300 charge_request(16624mV, 0mA)]
[2996985.809500 charge_request(16632mV, 0mA)]
[2996986.060900 charge_request(16624mV, 0mA)]
[2996991.816400 charge_request(16632mV, 0mA)]
[2996992.067300 charge_request(16624mV, 0mA)]
[2996994.645900 HC 0x0115 err 1]
PORT80: F90E
[2996994.819400 charge_request(16632mV, 0mA)]
[2996995.070900 charge_request(16624mV, 0mA)]
PORT80: 3C08
PORT80: 3D01
PORT80: 3D08
[2997021.852300 charge_request(16632mV, 0mA)]
[2997022.102400 charge_request(16624mV, 0mA)]
PORT80: 3C08
PORT80: 3C01
PORT80: 3C08
[2997036.869400 charge_request(16632mV, 0mA)]
[2997037.118700 charge_request(16624mV, 0mA)]
PORT80: 3C01
[2997040.865500 charge_request(16632mV, 0mA)]
[2997041.123200 charge_request(16624mV, 0mA)]
PORT80: 3C08
[2997055.695400 HC Suppressed: 0x97=0 0x98=0 0x113=0 0x103=0 0x115=60 0x2b=0 0x67=0 0x121=0]
[2997055.715200 HC 0x0115 err 1]
PORT80: F90E
PORT80: 3C01
PORT80: 3C08
PORT80: 3C01
PORT80: 3C08
[2997077.156500 charge_request(16632mV, 0mA)]
[2997077.407400 charge_request(16624mV, 0mA)]
PORT80: 3C01
PORT80: 3C08
PORT80: 3C01
PORT80: 3C08
PORT80: 3C01
[2997116.765300 HC 0x0115 err 1]
PORT80: F90E
PORT80: 3C08
PORT80: 3C01
PORT80: 3C08
PORT80: 3C01
PORT80: 3C08
PORT80: 3C01
PORT80: 3C08
[2997175.245500 charge_request(16632mV, 0mA)]
[2997175.496400 charge_request(16624mV, 0mA)]
[2997177.828700 HC 0x0115 err 1]
PORT80: F90E
PORT80: 3C01
[2997194.263000 charge_request(16632mV, 0mA)]
[2997194.513600 charge_request(16624mV, 0mA)]
[2997196.264700 charge_request(16632mV, 0mA)]
[2997196.513600 charge_request(16624mV, 0mA)]
PORT80: 3C08
PORT80: 3C01
[2997219.062200 HC 0x0002]
[2997219.064600 HC 0x000b]
Yup! That’s working. 
It gets the information needed to debug this next time it happens.
Starting to see this, with several of those kworkers showing near/at the top of htop again.
EC console
$ sudo ./build/bds/util/ectool console
Place your right index finger on the fingerprint reader
FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F81
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
[3021680.430600 HC 0x0115 err 1]
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3C08
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3C01
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: F90E
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F80
PORT80: 3FA0
[3021693.112600 HC 0x0002]
[3021693.114900 HC 0x000b]
So what’s PORT80?
Those are BIOS post codes. I think you should file a bug with framework support with all of these details so they can dig in and understand them.
Thanks Mario, will do. For completeness since this thread will probably be referred to, here’s the console output with the last part showing the logspam mostly going away when the machine is unplugged from AC.
... (lots more PORT80 spam preceding)
PORT80: 3F84
PORT80: 3F88
PORT80: 3F80
PORT80: 3FA0
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F88
PORT80: 3F80
PORT80: 3FA0
[3030047.208300 HC 0x0115 err 1]
PORT80: 3FA4
PORT80: 3F84
PORT80: 3F88
PORT80: 3F80
PORT80: 3FA0
PORT80: 3F84
PORT80: 3F88
[3030047.690900 update charger!!]
[3030047.700000 AC off]
[3030047.701900 event set 0x0000000000000010]
P:3 SET TYPEC RP=1[3030047.713500 cypd_write_reg8_wait_ack pre 0x84 ]
[3030047.717200 cypd_write_reg8_wait_ack pre 0x84 ]
[3030047.720200 PORT_DISCONNECT]
PORT80: AA83
[3030047.726100 board_set_active_charge_port port -1, prev:3]
[3030047.737200 cypd_write_reg8_wait_ack pre 0x4 ]
[3030047.749200 event set 0x0400000000000000]
[3030047.767200 cypd_write_reg8_wait_ack pre 0x80 ]
[3030047.770000 event set 0x0400000000000000]
[3030047.784000 Battery 81% (Display 81.9 %) / ??h:?? to empty]
[3030047.788600 CL: p-1 s-1 i500 v0]
[3030047.790100 TODO Implement pd_set_new_power_request port 3]
PORT80: AA8F
[3030047.888900 cypd_update_power_status:0=0x8]
[3030047.892600 cypd_update_power_status:1=0x8]
PORT80: 3F44
PORT80: 3F48
PORT80: AA8F
[3030052.341300 Battery 81% (Display 81.8 %) / 11h:21 to empty]
[3030058.377500 HC 0x0002]
[3030058.380100 HC 0x000b]
Edit: opened the support ticket, issue summarized there and pointing back to this thread. I expect it’ll be a few days, especially with the Lunar New Year holiday.
1 Like
I know it’s generally not useful to get “me too” posts, but since nobody else has corroborated this, I’ll say it. I haven’t dug in with ectool, but I have a bunch of busy kworker threads, and gpe10 is firing.
Thank you @dimitris for investigating and opening the bug report.
I’ve opened a support case with FW, where I owe them some logs/info. In the meantime, I’ve characterized this - at least as it happens with my setup - as related to the dock (Kensington 5780T) I’m using for power and USB devices.
The pattern is that, on the majority of resumes from s2idle I’ll notice these parasitic kworkers being busy. It always involves a pm worker - I assume that’s power management - and often the acpi worker is involved too. This never happens on fresh boot.
I can work around this by cycling power on the dock once or a few times. After the first (or second or third) quick-ish power cycle (just a few seconds off), these workers stop eating CPU.
The amount of state machinery involved here (PD/USB4/dock controllers + firmware as well as Linux drivers) seems so be making this race condition a little daunting to chase down.
However, one good first step would be decoding these EC console codes above. Any help on that would be appreciated.
I have this happen as well. Kernel 6.7.5-gentoo-x86_64, on AC power, USB4 dock (mokin) connected.
See if you can rebuild as described here and see if this helps:
I applied V2 of the patch to 6.7.8;
You say: see if this helps. Do you mean the patch is supposed to help with the issue? I was under the impression it only exposed the EC and allowed console output access.
Have you received any useful reply to your support case? I’ve been staying on 6.6.13 for now since it doesn’t exhibit the problem. I’m not anxious to file a support case personally, but I hope Framework is giving this attention.
Sorry I haven’t yet, things got a little busy. I’ll try to build a 6.7.8 plus the patches today to get up to date console output now that the behavior is more reproducible.
BTW if power cycling the hub/dock involved is an option then I wouldn’t hold onto 6.6, there are lots of important fixes, both security and FW-AMD specific, in the 6.7 series.
@Matt_Hartley I’ll go ahead and send an update on my support thread, unless I hear otherwise that it makes more sense to track here? (edit: I’ve responded to the ticket with logs/console output)
1 Like
Just a thought; does reverting ACPI: EC: Fix acpi_ec_dispatch_gpe() · torvalds/linux@b5539eb (github.com) help?
1 Like
Ticket is easier. Thanks. We are at a workshop this week, so my replies here will be very limited.