I’m running Debian testing (kernel 5.18.16) on my new 12th-gen i7-1280P, and I’m seeing some painfully overzealous CPU throttling that I assume is due to heat, even though it’s not getting that hot.
I was playing a game (Stellaris), and after a while (a couple hours), it gets incredibly slow and laggy. I quit back to the desktop, and the desktop is laggy. I don’t see anything in dmesg. I check temperatures with sensors, and they’re elevated, but not too bad – around 65C. I check powertop, and see that all CPU cores are pinned at 399MHz. I run stress -c 20. CPU usage goes to 100%, but all cores are still running at 399MHz.
I fully shut down, and start it up again. Even bootup seems slow. The GRUB boot menu paints slowly – I can nearly see it painting the screen line by line. I boot, which takes quite a while, and everything is still slow and laggy, and CPUs are still pinned to 399MHz. Temperatures have dropped a little under 50C, and have been there steady for a while.
So I shut down, and let it sit for a while (15-20 minutes maybe?) and then turn it back on, and things are ok. I go back to Stellaris, and play for a couple hours… and then it gets slow and laggy again. Rinse and repeat. It seems like I don’t have to shut down and let it sit; leaving it running and letting it sit for 15-20 minutes also fixes it up.
Any idea what’s going on? Is this an OS issue? Kernel issue? BIOS/firmware issue? Board issue? I do believe there was a kernel update a couple days ago that I just rebooted into yesterday. I’m not exactly sure when I rebooted into the new kernel, if it was before or after I first saw this issue. Unfortunately looks like the old kernel package is gone from the Debian mirrors, and even from my local cache, so I can’t go back to test.
It may be that it reaches its critical temperature, thermal throttles, and there’s a bug where it doesn’t know how to throttle back up from 399 mhz.
Maybe changing the thermal throttling limits down 3-5C on throttlestop or wherever would help… I dunno what it is right now, but if its set to throttle at 100C, switch it down to 97-95, or if it is at 95C, switch it down to 92-93C. This causes it to throttle sooner and maybe more frequently, so if the bug could happen at anytime you thermal throttle, you may experience this bug sooner, though.
Another fix may be to limit the max turbo frequency on every core 100mhz, which helps decrease max power spike, and or set PL1 and PL2 to infinite time and change the value to a TDP where it CPU is happy to stay at, say somewhere between 92-93C, where it shouldn’t thermal throttle. You might be able to do this by setting TDP to 28W and running either Throttlestop benchmark or Prime95 for 5 minutes and see where to max Temp stabilizes at a certain temperature and adjust the TDP to a point where it will never throttle.
Not really sure how to evaluate that… nothing looked odd on the board when I was installing the RAM and storage, but I imagine all that is hidden under other parts?
A few things to try out, since we know the issue (basically your CPU is hitting 105C and trigger PROCHOT), a few solutions:
Since you’re only hitting this after a few hours of gaming/being fully thermally saturated, have you considered using a laptop stand or an active cooling pad? Honestly, anything to make sure the fans and vents aren’t being blocked might be enough to make sure it doesn’t happen.
Make sure you are running thermald and adjust the thermal-conf.xml and lower the TripPoint Temperature. You should be able to force your CPU to throttle before it gets to the danger zone.
You can also run an app like MangoHud which will show you your CPU and GPU temps (or more detailed tools like s-tui, turbostat, or sensors on a remote terminal) if you need to be more aggressive w/ your thermal mitigation measures.
I suppose I could, but I like to game in various places, often with the laptop on my lap (in bed, sometimes, even), and ensuring I have the thermal pad nearby and ready seems like a pain. (Yes, I know, I’m a little weird.)
Ah, will give it a try! I just installed throttled and was going to see if that helped. But seems like thermald might be easier to configure.
I’ve been using s-tui for the past few days and it’s been really useful to help me see what’s going on.
Another useful tool for gaming would be an fps limiter.
Get GOverlay and limit your CPU to e.g. 30 fps.
Your system will now only render the 30 fps you have set, even if it would be capable of rendering more. This will keep your system cool(er) and even get you more consistent frame times
Hmm, I think I spoke too soon. My attempt to use thermald resulted in triggering the 400MHz issue while the laptop was sitting in a cool room on a cool glass table, with all temperatures in the 40-45C range. I waited over an hour afterward, and eventually had to shut off the laptop and leave it powered down for 10 minutes (after 5 minutes, it was still throttled when I powered up) to get it to go back to normal. Back to giving throttled a try. I think I just don’t know how to configure thermald correctly; seems more complicated than it needs to be (thanks, Intel…).
Edit: nevermind after another half hour or so, one of the core temperatures spiked to 100C, and now all cores are stuck throttled down to 400MHz again.
Ok, finally I think this is solved. After going through a torturous amount of troubleshooting steps with Framework Support, they finally decided to send me a mainboard replacement. I’ve been testing 100% CPU loads for the past hour and a half or so, and I haven’t encountered any throttling.
I’m feeling pretty confident about this because of how drastically different the thermals look with the new board vs. the old. With two simultaneous kernel compiles (using make -j20; make clean over and over in a loop), with the laptop on a glass table (bottom vents unobstructed), temperatures for all cores are stable, and max out at around 71C. If I block the bottom vents (laptop on my blanket-covered lap), temps slowly creep up to about 85C and stay there. Touching the bottom of the laptop at 85C is a little uncomfortable, but doesn’t actually cause pain like with the old board.
With the old mainboard, this kind of load (even with the vents unobstructed) would cause unpredictable temperatures, constantly jumping around to various values between 80C and 100C, without settling to any particular stable temperature.
I am a bit curious as to what’s wrong with the original mainboard. Support had me ensure the fan was clean and working; re-apply thermal paste; remove expansion cards and even RAM, storage, and the WiFi card; and reset the mainboard. The BIOS version on the new and old boards is the same (3.05), and BIOS and OS settings around power management etc. haven’t changed. A visual inspection of the old board doesn’t show any physical damage that I can see.
So, if you’re running into this problem, and you’ve gone through all the troubleshooting steps, I’d suggest you ask your case to be escalated to someone who can authorize a mainboard replacement. Hopefully that helps others as well.
This thread is getting pretty long and goes back a bit, and there are a few things that deviate outside of a vanilla install. If you do not have an active ticket, please create one and link to your posts.
My 12th gen does not exhibit this behavior, but I also keep to a pretty vanilla (guides specific) installation.
Support had me test on the “blessed” Ubuntu LiveCD linked from Framework’s website, and I was able to easily reproduce the issue, both on my original board, and the replacement board that was shipped to me.
(Regardless, if the Framework laptop cannot run a vanilla Debian stable install – I’m now running stable since testing was promoted to stable in June – that’s… pretty atrociously messed up.)
Not sure if my support ticket is still “active”; they had me perform a bunch of troubleshooting steps, and I got shuffled back and forth between several support people who kept asking me to do the same things I’d already done and had provided results for. At this point I’m not sure what else to do on that support ticket, as they’ve given me no further instructions. It’s been incredibly frustrating, to be honest.
Okay, if this is happening on a replacement board and we cannot replicate it. That does put us in a rock and a hard place as something else we cannot duplicate is happening.
We do not test again distros outside of what we have listed. We do however, have multiple pinned community Debian guides.
I would reply to to your last email from the ticket and see what the suggested next steps would be. I imagine if you were sent another board and are still seeing the issue, it would need to be elevated to engineering as we cannot replicate the environment affecting two separate boards (yours and the replacement).
I’m out of town for the next couple weeks, but will do so when I get back home.
Would it not make sense for me to return one of the two mainboards I have that exhibits the problem, and then your support or engineering team can attempt to reproduce it on that board?
Hi everyone,
I encounter the same problem regularly as well. Since I use an eGPU I tend to play a bit heavier games that do not take this throttling lightly: They stutter, freeze and/or crash and sometimes even take down the whole system. I reported this initially in another thread because I didn’t monitor the CPU frequencies back then (htop doesn’t show them): However using s-tui I can monitor that before every crash/freeze the frequency of all cores bumps down to 400 MHz and stays there even after the crash. It can take some time (even through reboots) until the CPU recovers, sometimes it takes half a minute, sometimes more than 10 minutes.
Reproducing this issue is a bit cumbersome, since to me it only occurs after a certain amount of time in the game (0,5-2 hours depending on the game and whether I use the iGPU or not). What I find particularly strange is that in the moment of throttling the CPU isn’t even extremely hot: With my eGPU it sits around 90° up to 95° tops even right before the crash. If I do other CPU-heavy work (which don’t last as long) like compiling code, the CPU can get up to 100° hot without any core throttling down to 400MHz. The only difference I can think of is, that after a 2 hours gaming session the heat has spread over the whole laptop (to other components and the outside of the housing which is quite hot at this point). Maybe this somehow triggers this throttling instead of the temperature inside the CPU? I will definitely do some more testing on this, maybe even on Windows (if this is indeed a Firmware issue, then Windows should be affected as well)
Since we do not test against eGPUs, we cannot speak to this.However, if you see throttling, please do test on Windows well for the eGPU. Linux eGPU handling is a mixed bag, so it can be helpful to have a comparison.
Hi @Matt_Hartley, I again put this off out of frustration, but finally emailed again on the support ticket 3 weeks ago, and again 2 weeks ago, after receiving no response. Any idea why Framework Support seems to be ghosting me?
after another half hour or so, one of the core temperatures spiked to 100C, and now all cores are stuck throttled down to 400MHz again.