Framework 400MHz issue

Please cite where I lost my temper. Being critical ≠ angry.

I’m not so sure AMD would like to help diagnose an issue with an Intel chip.

Regardless, I didn’t buy a computer from Intel, AMD or Insyde. It’s Framework’s responsibility to deliver the item I was sold. They’ve had 4 years to cross collaborate and make us right. I understand all of the points you make, but this should never have made it to a consumer’s hands. There should not be a single unhappy customer 4 years later.

There is no argument you can bring up that I do not have a rebuttal for. You’re giving them way too much slack.

You raise some good points, thought they are somewhat marred with unnecessary pettiness. My only point has always been that I will call out any injurious and unpleasant attitudes directed towards customer-facing staff. I am fairly robust, and I can continue to do that if it is necessary.

I must have missed the election for Hall Monitor.

1 Like

I am still troubleshooting. I think I may have a “solution”. I will post when I can confirm.

Ok. When able, if your solution doesn’t work, just pop what you’ve tried into the thread. Since you said you’ve already completed quite a bit, I wouldnt want you to have to repeat steps if its not necessary. :slightly_smiling_face:

1 Like

I can now reliably reproduce the 400MHz lock with a combination of GPU stress-load and CPU stress-load. Neither alone does it.

To reproduce:

  • Max the GPU: glmark2-wayland --off-screen --size 7680x4320 --frame-end none --run-forever -b 'refract:duration=60'
  • Max (1/2) the CPU: stress-ng --cpu 8 --timeout 10m

Wait. I get ordinary thermal and power throttling to 1700MHz on the CPU cores. And then after a few minutes, 400MHz all cores, locked, no way to reset, doesn’t change after cooldown – the usual symptoms. Power down → disconnect power → wait → reboot is the only fix. (I haven’t checked recently whether disconnecting the power is essential – I have a vague memory that it is. I’ve checked all other power states: nothing short of full power down resets the state).

The GPU maxing command I fiddled with a bit to get it to push the GPU to the max. The CPU I just ran at 8 cores. It took a couple of minutes to trigger the fault, but I didn’t test other values.

I am going to reboot now, set the GPU max clock at 900MHz (from 1300MHz default), and see if that keeps things sufficiently under control not to trigger the frequency lockdown.

Here’s some relevant diagnostics:

jbaxter@framework:~$ sudo rdmsr -p 0 0x1b1
881c080c
0x881c080c decodes as:

  • Bits 2 and 3 set (0xC): external/package PROCHOT# is currently asserted and has been logged.
  • Bits 0 and 1 clear: the CPU package’s own thermal threshold is not currently causing the throttle.
  • Bit 11 set (0x800): a package power-limit event occurred earlier, but bit 10 is clear, so that power limit is not currently active.
  • Bits 22:16 are 0x1c = 28°C below the thermal target. With a 100°C target, the package was about 72°C when it was read

grep -H . /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq
/sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq:399721 /sys/devices/system/cpu/cpu10/cpufreq/scaling_cur_freq:399995 /sys/devices/system/cpu/cpu11/cpufreq/scaling_cur_freq:400005 /sys/devices/system/cpu/cpu12/cpufreq/scaling_cur_freq:400036 /sys/devices/system/cpu/cpu13/cpufreq/scaling_cur_freq:400315 /sys/devices/system/cpu/cpu14/cpufreq/scaling_cur_freq:400000 /sys/devices/system/cpu/cpu15/cpufreq/scaling_cur_freq:400000 /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq:400770 /sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq:400000 /sys/devices/system/cpu/cpu3/cpufreq/scaling_cur_freq:400214 /sys/devices/system/cpu/cpu4/cpufreq/scaling_cur_freq:400004 /sys/devices/system/cpu/cpu5/cpufreq/scaling_cur_freq:400000 /sys/devices/system/cpu/cpu6/cpufreq/scaling_cur_freq:399761 /sys/devices/system/cpu/cpu7/cpufreq/scaling_cur_freq:400000 /sys/devices/system/cpu/cpu8/cpufreq/scaling_cur_freq:399960 /sys/devices/system/cpu/cpu9/cpufreq/scaling_cur_freq:399816

Speaks for itself. The dreaded 400MHz frequency lock.

sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +73.0°C (high = +100.0°C, crit = +100.0°C)
Core 0: +73.0°C (high = +100.0°C, crit = +100.0°C)
Core 4: +69.0°C (high = +100.0°C, crit = +100.0°C)
Core 8: +69.0°C (high = +100.0°C, crit = +100.0°C)
Core 12: +68.0°C (high = +100.0°C, crit = +100.0°C)
Core 16: +66.0°C (high = +100.0°C, crit = +100.0°C)
Core 17: +66.0°C (high = +100.0°C, crit = +100.0°C)
Core 18: +66.0°C (high = +100.0°C, crit = +100.0°C)
Core 19: +66.0°C (high = +100.0°C, crit = +100.0°C)
Core 20: +70.0°C (high = +100.0°C, crit = +100.0°C)
Core 21: +71.0°C (high = +100.0°C, crit = +100.0°C)
Core 22: +70.0°C (high = +100.0°C, crit = +100.0°C)
Core 23: +70.0°C (high = +100.0°C, crit = +100.0°C)

BAT1-acpi-0
Adapter: ACPI interface
in0: 17.25 V
curr1: 0.00 A

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +49.0°C

nvme-pci-0100
Adapter: PCI adapter
Composite: +56.9°C (low = -273.1°C, high = +81.8°C)
(crit = +84.8°C)
Sensor 1: +56.9°C (low = -273.1°C, high = +65261.8°C)
Sensor 2: +61.9°C (low = -273.1°C, high = +65261.8°C)

acpitz-acpi-0
Adapter: ACPI interface
temp1: +72.8°C
temp2: +67.8°C
temp3: +67.8°C
temp4: +65.8°C
temp5: +45.8°C`

3 Likes

Thanks @Jonathan_Baxter . I’ve shared that info with the Linux leads just now.

Hey! Apologies for the double reply so close together. The Linux Leads want to look into this with you. I’m going to pull your ticket and get it over to them now.

1 Like

Can confirm power disconnect for a non-trivial amount of time is necessary for the reset.
And 950MHz max on the GPU did not fix the problem. Trying 750MHz now.

Yeah, that I understand. But I would think the troubleshooting flow sequence for the 400MHz symptom would be ironed out / standardised by now to be in a KB, to allow for a higher degree of customer / end-user independence, un-tethered from support.

In my head, owning your own hardware includes empowering users with how to troubleshoot issues independently.

2 Likes

Please cite where I had an unpleasant attitude towards staff.

@Jonathan_Baxter , sorry if it’s an insult to ask, but do you have a windows install to test? I found that earlier in it’s life, there would need to be some video de/encoding needed to trigger it, but I can do it by purely poking the CPU now.

I don’t have windows. It turns out throttling the GPU doesn’t fix it. I still get the same problem with max frequency set to 750MHz. Now I am trying throttling overall power consumption, which it turns out you can do. At 30W max I am not seeing the problem. I will increment the peak wattage until I trigger the issue again.

I think it would be helpful if FW published the schematic details, showing all devices/chips that are in some way linked to the prochot pin on the cpu.
Then users could methodically work through each device until the problem one is found.

1 Like

Unfortunately, this issue isn’t that simple. As we’ve been able to resolve multiple cases, that we have found to have different causes, using different methods … its not a 1 size fits all fix. If it was, trust me, we’d have that fix uploaded to a KB and highlight it in neon faster than a streak of blue lightening.

1 Like

sigh no need precision fixing just completely nuke the external prochot signal via an EC update as we all know the CPU 100% won’t overheat and burn since it has its own thermal throttling and power limit there’s zero need for destructive 400MHz prochot crying wolf.

If EC programming doesn’t work, I’ll go as far as physically cutting the wire connected to the prochot pin

1 Like

I fixed it. Here is my diagnosis in case it helps anyone else who finds themselves in a similar predicament. Disclosure: I generated this report with the AI I was using to help with the diagnosis, with only light post-editing, which is why it reads weird. But I have spent too long on this already, so this is what you get :slight_smile:

TL;DR

I traced my recurring 400 MHz lock to a failing heatsink/fan assembly that caused the system to peg at 100°C while drawing only ~17 W. This eventually triggers a platform-level (i.e. external to the CPU) PROCHOT condition that latches and does not clear after the system cools. Replacing the heatsink/fan assembly with a spare I had lying around fixed the issue because the system can no longer be forced into the pathological thermal state that triggers the latch.

Diagnosis of recurring 400 MHz lock on 12th-gen Framework: failed cooling assembly triggering latched external PROCHOT

I think I have finally diagnosed the recurring 400 MHz throttling problem on my 12th-gen Framework.

Symptoms

The failure mode was:

  • All CPU cores became locked at approximately 400 MHz regardless of load.
  • CPU frequency policy still allowed the normal maximum frequency.
  • Temperatures would fall after the workload stopped, but the 400 MHz limit remained.
  • Suspend, including deep suspend, did not clear it.
  • Normal reboot is insufficient.
  • Shutting down and removing external power for several minutes cleared the condition.
  • It would eventually recur, particularly after combined CPU + GPU workloads that kept the fans running hard for an extended period.

For reference, while locked:

grep . /sys/devices/system/cpu/cpu0/cpufreq/{scaling_driver,scaling_governor,scaling_cur_freq,scaling_max_freq,cpuinfo_max_freq}

gave:

scaling_driver:intel_pstate
scaling_governor:powersave
scaling_cur_freq:399841
scaling_max_freq:4400000
cpuinfo_max_freq:4400000

So Linux was not imposing a 400 MHz frequency limit. The requested maximum was still 4.4 GHz.

Reproducing and monitoring the problem

I used turbostat to monitor CPU, GPU, temperatures and package power:

sudo turbostat --quiet --Summary --interval 1 \
    --show Busy%,Bzy_MHz,CoreTmp,CoreThr,PkgTmp,GFX%C0,PkgWatt,CorWatt,GFXWatt,PKG_%

For GPU load I used:

glmark2-wayland \
    --off-screen \
    --size 7680x4320 \
    --frame-end finish \
    --run-forever \
    -b 'refract:duration=60'

and added CPU load with stress-ng, for example:

stress-ng --cpu 8

After replacing the cooling assembly I went as far as:

stress-ng --cpu 16

while simultaneously saturating the iGPU.

The important clue: 100°C at only ~17 W

With the original heatsink/fan assembly, a combined CPU+iGPU load produced this:

Busy%  Bzy_MHz  CoreTmp  PkgTmp  GFX%C0  PkgWatt  CorWatt  GFXWatt
50.66     1452       99     100  100.51     17.30     7.89     5.38
51.66     1467       99     100  100.44     17.48     7.97     5.46
52.11     1444       99     100  100.89     17.69     7.99     5.62
52.03     1438      100     100  100.68     17.41     7.87     5.38
51.69     1398      100     100  100.34     17.17     7.61     5.44

CoreThr was also increasing rapidly during this state.

This was the clue I initially missed: the processor was sitting at its 100°C thermal limit while dissipating only about 17 W. That is extremely poor cooling performance for this system.

Identifying what was causing the 400 MHz lock

I installed msr-tools and loaded the MSR driver:

sudo apt install msr-tools
sudo modprobe msr

The most useful register was Intel’s package thermal-status MSR:

sudo rdmsr -p 0 0x1b1

This is IA32_PACKAGE_THERM_STATUS.

To watch the low four status bits continuously:

sudo watch -n 0.5 '
v=$(rdmsr -p 0 0x1b1)
printf "THERM_STATUS=0x%s low=0x%x\n" "$v" "$((0x$v & 15))"
'

Intel documents/interprets the low bits as:

bit 0   internal thermal status
bit 1   internal thermal status log
bit 2   external PROCHOT status
bit 3   external PROCHOT log

So useful low-nibble values are:

0x0    no internal or external thermal condition indicated

0x1    internal thermal control currently active
0x2    internal thermal event logged
0x3    internal thermal control active + logged

0x4    external PROCHOT currently asserted
0x8    external PROCHOT event logged
0xc    external PROCHOT currently asserted + logged

Intel specifically recommends using IA32_PACKAGE_THERM_STATUS to distinguish internal thermal throttling from an externally asserted PROCHOT condition; bits 0-1 describe the internal thermal sensors and bits 2-3 the external PROCHOT signal.

While the processor was genuinely at approximately 100°C, I saw the low nibble oscillating through values such as 0x0, 0x2 and 0x3, consistent with ordinary internal thermal control engaging and disengaging around the thermal limit.

But once the pathological 400 MHz state occurred, I read:

IA32_PACKAGE_THERM_STATUS = 0x881c080c

The key part is the final 0xc.

That means the internal thermal-status bits 0 and 1 were clear, while the external PROCHOT status and log bits 2 and 3 were set.

This was particularly significant because by then the processor had already cooled:

Busy%  Bzy_MHz  CoreTmp  PkgTmp  GFX%C0  PkgWatt
50.99      400       79      79  100.64      4.93
56.47      400       79      78  100.60      5.11
55.55      400       79      79  100.54      5.07
53.85      400       79      79  100.11      5.00
52.96      400       78      78  101.10      5.11

So the CPU was no longer thermally hot and package power was only about 5 W, yet the platform was still asserting external PROCHOT and forcing the processor to its minimum frequency.

That explains why changing Linux governors, frequency limits, etc. did nothing.

Checking package power limits

I also checked Intel RAPL:

rapl=/sys/class/powercap/intel-rapl:0

for i in 0 1; do
    printf 'constraint %s: ' "$i"
    cat "$rapl/constraint_${i}_name"
    printf 'power: '
    cat "$rapl/constraint_${i}_power_limit_uw"
    printf 'window: '
    cat "$rapl/constraint_${i}_time_window_us"
done

Before I started experimenting with the limits, this reported:

constraint 0: long_term
power: 200000000
window: 27983872

constraint 1: short_term
power: 64000000
window: 2440

I also checked the processor’s advertised long-term thermal-spec power:

cat /sys/class/powercap/intel-rapl:0/constraint_0_max_power_uw

which returned:

28000000

or 28 W.

For lower-level inspection, the relevant RAPL MSRs can be read with:

sudo rdmsr -p 0 0x606   # MSR_RAPL_POWER_UNIT
sudo rdmsr -p 0 0x610   # MSR_PKG_POWER_LIMIT
sudo rdmsr -p 0 0x614   # MSR_PKG_POWER_INFO

At one point, after I had manually configured PL1=20 W and PL2=64 W, I obtained:

0x606: a0e03
0x610: 43820000dd80a0
0x614: e0

0x606 gives a power unit of 1/8 W, so the 0xe0 thermal-spec power field in 0x614 corresponds to:

224 × 0.125 W = 28 W

MSR_PKG_POWER_LIMIT contains the PL1 and PL2 power limits, enable/clamp fields and their respective time-window encodings. Intel documents PL1 in bits 14:0 and PL2 in bits 46:32, with their enable, clamp and time-window fields adjacent.

I experimented with lower RAPL limits and found that reducing package power could prevent the problem during ordinary workloads. However, that turned out to be treating the symptom rather than the cause: it was while messing around with the RAPL limits that I noticed the package temperature at 100°C while consuming only ~17 W, and realized the cooling was borked.

Replacing the heatsink/fan assembly

I had a vague recollection of buying a spare heatsink/fan during one of framework’s clearance sales a while back. After finding it in the spare PC parts bin, I replaced the complete heatsink/fan assembly.

That completely changed the behavior.

Under combined CPU+iGPU stress, the system can now dissipate roughly 30-36 W while remaining substantially below 100°C for most of the test:

Busy%  Bzy_MHz  CoreTmp  PkgTmp  GFX%C0  PkgWatt  CorWatt  GFXWatt
99.77     2519       85      85  100.58     36.18    25.75     6.69
99.77     2523       89      89  100.57     36.17    26.00     6.41
99.77     2528       92      92  100.57     36.18    25.91     6.49
99.77     2530       94      94  100.58     36.50    26.09     6.53
99.77     2097       92      92  100.57     30.05    20.53     5.71

Even with essentially 100% CPU load and 100% iGPU load, the machine remains functional.

There were occasional normal thermal-throttle events during the most extreme part of the test. For example, CoreThr increased when package temperature briefly reached the high 90s. That is qualitatively different from the old cooling system, where the machine continuously sat at 100°C at only ~17 W and accumulated thermal-throttle events rapidly. Intel’s normal hardware thermal monitor reduces processor power when the trip temperature is reached.

Most importantly, I have been unable to reproduce the permanent 400 MHz lock since replacing the cooling assembly, even using:

stress-ng --cpu 16

while simultaneously saturating the iGPU.

Working diagnosis

My current interpretation is:

failed/degraded heatsink, fan, TIM or heatpipe
                |
                v
very high thermal resistance
                |
                v
package reaches 100°C at only ~17 W
                |
                v
normal Intel internal thermal throttling engages repeatedly
                |
                v
continued severe thermal condition eventually causes some
platform-level protection mechanism to assert external PROCHOT
                |
                v
CPU and GPU are forced to their minimum performance state
                |
                v
external PROCHOT remains asserted even after the CPU cools
                |
                v
CPU remains locked at ~400 MHz until full power removal

The first part of this chain is now strongly supported experimentally: replacing the cooling assembly increased sustainable package dissipation from roughly 17 W at 100°C to roughly 30-36 W at lower temperatures under a substantially heavier workload.

The register evidence also establishes that the eventual 400 MHz state was not ordinary CPU thermal throttling. While locked and already cooled, IA32_PACKAGE_THERM_STATUS showed an active external PROCHOT assertion (...0c).

What I cannot determine from software is exactly which Framework component asserts that external signal, or why it remains asserted after the initiating thermal condition has disappeared. It could be EC/firmware behavior, VR protection, another board-level thermal sensor, or an intentionally latched hardware protection circuit.

So I would separate the two issues:

  1. Primary fault on my machine: the cooling assembly had degraded badly enough that the CPU package reached 100°C at only ~17 W.
  2. Secondary Framework/platform behavior: after sufficiently severe/prolonged thermal stress, external PROCHOT can apparently become asserted and remain latched, leaving the machine permanently at ~400 MHz until power is completely removed.

With a healthy cooling assembly, I can no longer reproduce the second condition because even an extreme CPU+iGPU workload does not put the machine into the same pathological thermal state.

9 Likes

This is excellent work @Jonathan_Baxter, same for your earlier long post. While I have spoken my mind on matters of tone within this thread, I hope the following remark is heard with the sincerity and kindness I intend: when one turns one’s hand optimistically to a difficult and intermittent issue, it is remarkable what can be achieved.

I should love to hear that your fix works for others too.

1 Like

You’re talking about fix. I’m talking about troubleshooting, cause analysis / identification.

e.g. If you’re seeing this…try this…then this…if this still have issue, look here, check this…board hardware fault → reach out to support. If you didn’t see this, try this, then that, and this…do you see this…nope, ok, HFM (Heatsink & Fan Module) issue → Under warranty → call support, out of warranty → purchase HFM…etc.

Free up support from emails.

e.g. Whatever support flow you have internally for the 400Mhz, publish it. It’s not like every support individual is an engineer to know how to respond to the 400Mhz issue over emails on the fly.

Your customer / user base is a spectrum of users with varying degree of technical competency, time availability, cost sensitivity. It would be good to have options for them to decide if they want to engage support or self-initiate troubleshooting (again, (root) cause analysis).

1 Like

So to sum it up in one sentence …

It’s basically a heat or cooling issue of the whole system/laptop?

1 Like

That’s 50%. It’s a cooling issue that triggers it. But then there’s a fault that keeps the laptop in the degraded state.

That second part is perhaps why this fault shows up with apparently different causes. There could be different scenarios under which whatever asserts PROCHOT gets triggered (failed cooling in my case), but then once it triggers the symptoms are identical.

1 Like