I think I am reproducing this problem on my Framework Desktop. I was able to stop my frequent hangs / freezes while using Ollama by disabling PSR. I am connected to the onboard HDMI port with my TV.
Another thing to add is that the issue only produced itself on the DP-connected monitor, but after testing and seeing that my issues went away after disconnecting the TV, I realized that when I connected the TV is when the issue started for me!
Thank you for this @Jan_Theofel !
Well, the HDMI output is definitely part of my issue. I frequently see things like this happen still, but with only a “blackout” before the display comes back. When the HDMI is connected, I see the freeze. I guess disabling PSR had no effect on my issue, however similar it seems to me…
[ +2.693634] amdgpu 0000:c3:00.0: amdgpu: MES failed to respond to msg=REMOVE_QUEUE
[ +0.000008] amdgpu 0000:c3:00.0: amdgpu: failed to remove hardware queue from MES, doorbell=0x1002
[ +0.000002] amdgpu 0000:c3:00.0: amdgpu: MES might be in unrecoverable state, issue a GPU reset
[ +0.000005] amdgpu 0000:c3:00.0: amdgpu: Failed to evict queue 1
[ +0.000004] amdgpu 0000:c3:00.0: amdgpu: GPU reset begin!
[ +0.000106] amdgpu 0000:c3:00.0: amdgpu: Failed to evict process queues
[ +0.000031] amdgpu 0000:c3:00.0: amdgpu: Dumping IP State
[ +0.000922] amdgpu 0000:c3:00.0: amdgpu: Dumping IP State Completed
[ +2.083439] amdgpu 0000:c3:00.0: amdgpu: MES failed to respond to msg=REMOVE_QUEUE
[ +0.000009] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] ERROR failed to unmap legacy queue
[ +0.001985] amdgpu 0000:c3:00.0: amdgpu: MODE2 reset
[ +0.026373] amdgpu: Freeing queue vital buffer 0x7f5c97200000, queue evicted
[ +0.000023] amdgpu: Freeing queue vital buffer 0x7f5ca4a00000, queue evicted
[ +0.000006] amdgpu: Freeing queue vital buffer 0x7f5d1ca00000, queue evicted
[ +0.000004] amdgpu: Freeing queue vital buffer 0x7f630d400000, queue evicted
[ +0.000004] amdgpu: Freeing queue vital buffer 0x7f6324400000, queue evicted
[ +0.000782] amdgpu 0000:c3:00.0: amdgpu: GPU reset succeeded, trying to resume
[ +0.000527] [drm] PCIE GART of 512M enabled (table at 0x0000008001300000).
[ +0.000036] amdgpu 0000:c3:00.0: amdgpu: [drm] AMDGPU device coredump file has been created
[ +0.000004] amdgpu 0000:c3:00.0: amdgpu: [drm] Check your /sys/class/drm/card1/device/devcoredump/data
[ +0.000004] amdgpu 0000:c3:00.0: amdgpu: SMU is resuming…
[ +0.010470] amdgpu 0000:c3:00.0: amdgpu: SMU is resumed successfully!
[ +0.013701] amdgpu 0000:c3:00.0: amdgpu: [drm] DMUB hardware initialized: version=0x09002E00
[ +0.067299] amdgpu 0000:c3:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
[ +0.000007] amdgpu 0000:c3:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0
[ +0.000001] amdgpu 0000:c3:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0
[ +0.000001] amdgpu 0000:c3:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 6 on hub 0
[ +0.000001] amdgpu 0000:c3:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 7 on hub 0
[ +0.000001] amdgpu 0000:c3:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 8 on hub 0
[ +0.000000] amdgpu 0000:c3:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 9 on hub 0
[ +0.000001] amdgpu 0000:c3:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 10 on hub 0
[ +0.000001] amdgpu 0000:c3:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 11 on hub 0
[ +0.000001] amdgpu 0000:c3:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0
[ +0.000000] amdgpu 0000:c3:00.0: amdgpu: ring vcn_unified_0 uses VM inv eng 0 on hub 8
[ +0.000001] amdgpu 0000:c3:00.0: amdgpu: ring vcn_unified_1 uses VM inv eng 1 on hub 8
[ +0.000001] amdgpu 0000:c3:00.0: amdgpu: ring jpeg_dec_0 uses VM inv eng 4 on hub 8
[ +0.000000] amdgpu 0000:c3:00.0: amdgpu: ring jpeg_dec_1 uses VM inv eng 6 on hub 8
[ +0.000001] amdgpu 0000:c3:00.0: amdgpu: ring mes_kiq_3.1.0 uses VM inv eng 13 on hub 0
[ +0.000001] amdgpu 0000:c3:00.0: amdgpu: ring vpe uses VM inv eng 7 on hub 8
[ +0.004846] amdgpu 0000:c3:00.0: amdgpu: GPU reset(53) succeeded!
[ +0.000015] amdgpu 0000:c3:00.0: [drm] device wedged, but recovered through reset
In my case it was exactly the opposite:
When I had and screen attached the system did never crash.
It only crashed when NOT connected to a screen…
Yeah this actually fixed it for me!!!
Instead of the module parameter there is a kernel fix in 6.17.2 and 6.18-rc1 if you have new enough microcode for GPU.
What kernel are you on and what version of microcode (debugfs amdgpu_firmware_info file will tell you)
Do I have a recent enough firmware? What version do I need to find?
Caveat: This is output from a FW16 7840HS APU
My firmware info:
cat /sys/kernel/debug/dri/0000:c1:00.0/amdgpu_firmware_info
VCE feature version: 0, firmware version: 0x00000000
UVD feature version: 0, firmware version: 0x00000000
MC feature version: 0, firmware version: 0x00000000
ME feature version: 35, firmware version: 0x00000063
PFP feature version: 35, firmware version: 0x00000067
CE feature version: 0, firmware version: 0x00000000
RLC feature version: 1, firmware version: 0x0000008a
RLC SRLC feature version: 0, firmware version: 0x00000000
RLC SRLG feature version: 0, firmware version: 0x00000000
RLC SRLS feature version: 0, firmware version: 0x00000000
RLCP feature version: 1, firmware version: 0x0000000f
RLCV feature version: 0, firmware version: 0x00000000
MEC feature version: 35, firmware version: 0x00000043
IMU feature version: 0, firmware version: 0x0b012d00
SOS feature version: 0, firmware version: 0x00000000
ASD feature version: 553648378, firmware version: 0x210000fa
TA XGMI feature version: 0x00000000, firmware version: 0x00000000
TA RAS feature version: 0x00000000, firmware version: 0x00000000
TA HDCP feature version: 0x00000000, firmware version: 0x17000049
TA DTM feature version: 0x00000000, firmware version: 0x1200001a
TA RAP feature version: 0x00000000, firmware version: 0x00000000
TA SECUREDISPLAY feature version: 0x00000000, firmware version: 0x00000000
SMC feature version: 0, program: 0, firmware version: 0x004c6000 (76.96.0)
SDMA0 feature version: 60, firmware version: 0x00000017
VCN feature version: 0, firmware version: 0x09118016
DMCU feature version: 0, firmware version: 0x00000000
DMCUB feature version: 0, firmware version: 0x08005300
TOC feature version: 0, firmware version: 0x0000000b
MES_KIQ feature version: 6, firmware version: 0x00000106
MES feature version: 1, firmware version: 0x00000080
VPE feature version: 0, firmware version: 0x00000000
VBIOS version: 113-PHXGENERIC-001
0x7f is the min version. You’ve got new enough. I don’t think you have the same issue.
I think I might be experiencing this issue, but it is hard to be certain.
I’m running Xubuntu on my amd Framework 13, with the latest BIOS and kernel 6.14.0-36 and I frequently get system crashes. These typically happen when on video calls, or flicking between photos or videos in a browser.
The crash manifests with a screen freeze, but the mouse pointer continues to move and audio continues to play. After a few more seconds the screens go blank, and then finally the audio/mouse stops and everything is dead.
Here is some journalctl output.
Dec 09 10:11:04 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: Dumping IP State
Dec 09 10:11:09 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x0000001A SMN_C2PMSG_82:0x00000000
Dec 09 10:11:09 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: Failed to disable gfxoff!
Dec 09 10:11:14 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x0000001A SMN_C2PMSG_82:0x00000000
Dec 09 10:11:14 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: Failed to disable gfxoff!
Dec 09 10:11:19 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x0000001A SMN_C2PMSG_82:0x00000000
Dec 09 10:11:19 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: Failed to disable gfxoff!
Dec 09 10:11:23 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=MISC (WAIT_REG_MEM)
Dec 09 10:11:23 tom-sb kernel: [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR* failed to reg_write_reg_wait
Dec 09 10:11:24 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x0000001A SMN_C2PMSG_82:0x00000000
Dec 09 10:11:24 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: Failed to disable gfxoff!
Dec 09 10:11:24 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: Dumping IP State Completed
Dec 09 10:11:24 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: ring sdma0 timeout, signaled seq=52554, emitted seq=52557
Dec 09 10:11:24 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: Starting sdma0 ring reset
Dec 09 10:11:25 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=MISC (WAIT_REG_MEM)
Dec 09 10:11:25 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: reset sdma queue (0:0:0)
Dec 09 10:11:25 tom-sb kernel: [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR* failed to reg_write_reg_wait
Dec 09 10:11:26 tom-sb kernel: amdgpu 0000:c1:00.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring sdma0 test failed (-110)
Dec 09 10:11:26 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: Ring sdma0 reset failure
Dec 09 10:11:26 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: GPU reset begin!
Dec 09 10:11:30 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x0000001A SMN_C2PMSG_82:0x00000000
Dec 09 10:11:30 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: Failed to disable gfxoff!
Dec 09 10:11:33 tom-sb kernel: WARNING: CPU: 14 PID: 263 at drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn314/dcn314_smu.c:159 dcn314_smu_send_msg_with_param+0x11d/0x1c0 [amdgpu]
Dec 09 10:11:33 tom-sb kernel: Modules linked in: hid_logitech_hidpp hid_logitech_dj uas usb_storage ax88179_178a usbnet mii rfcomm snd_seq_dummy snd_hrtimer ccm cmac algif_hash algif_skcipher af_alg qrtr overlay bnep binfmt_misc amdgpu snd_sof_amd_acp70 snd_sof_amd_acp6>
Dec 09 10:11:33 tom-sb kernel: Workqueue: amdgpu-reset-dev drm_sched_job_timedout [gpu_sched]
Dec 09 10:11:33 tom-sb kernel: RIP: 0010:dcn314_smu_send_msg_with_param+0x11d/0x1c0 [amdgpu]
Dec 09 10:11:33 tom-sb kernel: dcn314_smu_enable_pme_wa+0x36/0x50 [amdgpu]
Dec 09 10:11:33 tom-sb kernel: dcn314_enable_pme_wa+0xe/0x20 [amdgpu]
Dec 09 10:11:33 tom-sb kernel: dce110_disable_audio_stream+0x98/0xc0 [amdgpu]
Dec 09 10:11:33 tom-sb kernel: link_set_dpms_off+0x10f/0x410 [amdgpu]
Dec 09 10:11:33 tom-sb kernel: dcn31_reset_back_end_for_pipe.isra.0+0x146/0x2b0 [amdgpu]
Dec 09 10:11:33 tom-sb kernel: dcn31_reset_hw_ctx_wrap+0xef/0x220 [amdgpu]
Dec 09 10:11:33 tom-sb kernel: dce110_apply_ctx_to_hw+0x7c/0x3d0 [amdgpu]
Dec 09 10:11:33 tom-sb kernel: dc_commit_state_no_check+0x385/0xf80 [amdgpu]
Dec 09 10:11:33 tom-sb kernel: dc_commit_streams+0x34c/0x780 [amdgpu]
Dec 09 10:11:33 tom-sb kernel: dm_suspend+0x2af/0x310 [amdgpu]
Dec 09 10:11:33 tom-sb kernel: amdgpu_ip_block_suspend+0x24/0x80 [amdgpu]
Dec 09 10:11:33 tom-sb kernel: amdgpu_device_ip_suspend_phase1+0xa3/0x120 [amdgpu]
Dec 09 10:11:33 tom-sb kernel: amdgpu_device_ip_suspend+0x2a/0x80 [amdgpu]
Dec 09 10:11:33 tom-sb kernel: amdgpu_device_pre_asic_reset+0xf3/0x6b0 [amdgpu]
Dec 09 10:11:33 tom-sb kernel: amdgpu_device_gpu_recover+0x33c/0xb50 [amdgpu]
Dec 09 10:11:33 tom-sb kernel: amdgpu_job_timedout+0x1d0/0x310 [amdgpu]
Dec 09 10:11:40 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=REMOVE_QUEUE
Dec 09 10:11:40 tom-sb kernel: [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
Dec 09 10:11:43 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=REMOVE_QUEUE
Dec 09 10:11:43 tom-sb kernel: [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
Dec 09 10:11:46 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=REMOVE_QUEUE
Dec 09 10:11:46 tom-sb kernel: [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
Dec 09 10:11:48 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=REMOVE_QUEUE
Dec 09 10:11:48 tom-sb kernel: [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
Dec 09 10:11:51 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=REMOVE_QUEUE
Dec 09 10:11:51 tom-sb kernel: [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
Dec 09 10:11:54 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=REMOVE_QUEUE
Dec 09 10:11:54 tom-sb kernel: [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
Dec 09 10:11:56 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=REMOVE_QUEUE
Dec 09 10:11:56 tom-sb kernel: [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
Dec 09 10:11:59 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=REMOVE_QUEUE
Dec 09 10:11:59 tom-sb kernel: [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
Dec 09 10:12:02 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=REMOVE_QUEUE
Dec 09 10:12:02 tom-sb kernel: [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
Dec 09 10:12:02 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: MODE2 reset
Dec 09 10:12:06 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x0000001A SMN_C2PMSG_82:0x00000000
Dec 09 10:12:06 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: Mode2 reset failed!
Dec 09 10:12:06 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: ASIC reset failed with error, -62 for drm dev, 0000:c1:00.0
Dec 09 10:12:06 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: GPU reset(1) failed
Dec 09 10:12:06 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: GPU reset end with ret = -62
Dec 09 10:12:06 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: GPU Recovery Failed: -62
Dec 09 10:12:17 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: Dumping IP State
Dec 09 10:12:17 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: Dumping IP State Completed
Dec 09 10:12:17 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: ring sdma0 timeout, signaled seq=52554, emitted seq=52557
Dec 09 10:12:17 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: Starting sdma0 ring reset
Dec 09 10:12:17 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: reset sdma queue (0:0:0)
Dec 09 10:12:17 tom-sb kernel: amdgpu 0000:c1:00.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring sdma0 test failed (-110)
Dec 09 10:12:17 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: Ring sdma0 reset failure
Dec 09 10:12:17 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: GPU reset begin!
Dec 09 10:12:17 tom-sb kernel: amdgpu 0000:c1:00.0: amdgpu: Failed to disallow df cstate
Dec 09 10:15:18 tom-sb kernel: ? __pfx_amdgpu_job_fence_free+0x10/0x10 [amdgpu]
Dec 09 10:15:18 tom-sb kernel: amdgpu_vm_fini+0xf1/0x610 [amdgpu]
Dec 09 10:15:18 tom-sb kernel: amdgpu_driver_postclose_kms+0x198/0x290 [amdgpu]
Any advice greatly appreciated.
I’m experiencing similar amdgpu hangs with this configuration:
These crashes happen once or twice per day, mostly while using Chrome (with any website). Could be related to how Chrome uses the GPU:
Dec 28 10:39:14 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=MISC (WAIT_REG_MEM)
Dec 28 10:39:14 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: failed to reg_write_reg_wait
Dec 28 10:39:16 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=MISC (WAIT_REG_MEM)
Dec 28 10:39:16 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: failed to reg_write_reg_wait
Dec 28 10:39:19 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=MISC (WAIT_REG_MEM)
Dec 28 10:39:19 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: failed to reg_write_reg_wait
Dec 28 10:39:22 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=MISC (WAIT_REG_MEM)
Dec 28 10:39:22 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: failed to reg_write_reg_wait
Dec 28 10:39:25 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=MISC (WAIT_REG_MEM)
Dec 28 10:39:25 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: failed to reg_write_reg_wait
Dec 28 10:39:27 giskard google-chrome-stable[5317]: [1228/103927.691396:ERROR:third_party/crashpad/crashpad/util/file/directory_reader_posix.cc:43] opendir /home/bernie/.config/google-chrome/Crash Reports/attachments/43fbf416-a0a4-4e80>
Dec 28 10:39:28 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=MISC (WAIT_REG_MEM)
Dec 28 10:39:28 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: failed to reg_write_reg_wait
Dec 28 10:39:30 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=MISC (WAIT_REG_MEM)
Dec 28 10:39:30 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: failed to reg_write_reg_wait
Dec 28 10:39:31 giskard google-chrome-stable[5317]: [35860:1:1228/103931.804024:ERROR:gpu/ipc/client/command_buffer_proxy_impl.cc:321] GPU state invalid after WaitForGetOffsetInRange.
Dec 28 10:39:31 giskard google-chrome-stable[5317]: [36077:7:1228/103931.804352:ERROR:gpu/ipc/client/command_buffer_proxy_impl.cc:122] ContextResult::kTransientFailure: Failed to send GpuControl.CreateCommandBuffer.
Dec 28 10:39:33 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=MISC (WAIT_REG_MEM)
Dec 28 10:39:33 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: failed to reg_write_reg_wait
Dec 28 10:39:36 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=MISC (WAIT_REG_MEM)
Dec 28 10:39:36 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: failed to reg_write_reg_wait
Dec 28 10:39:39 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=MISC (WAIT_REG_MEM)
Dec 28 10:39:39 giskard kernel: amdgpu 0000:c1:00.0: amdgpu: failed to reg_write_reg_wait
I tried booting with amdgpu.cwsr_enable=0 as suggested by @James3 , but unlike @Tristin_Stagg I still experienced a crash identical to the one reported above.
Now I rebooted with amdgpu.mes=0 to see if it helps.
I am not on framework laptop, but hitting similar issues and got solutions here with the help of Claude. So I think it is worth documenting here. The following is the bug report by Claude:
Confirmation: amdgpu.cwsr_enable=0 fixes MES hang on gfx1150 (non-Framework hardware)
Experienced identical MES timeout issues while running CEMU emulator (Zelda: Breath of the Wild) with Vulkan renderer. The crash was 100% reproducible when using the Magnesis/Stasis rune and moving the cursor from the environment towards highlighted objects.
[drm] ring gfx_0.0.0 timeout, signaled seq=X, emitted seq=X+2
[drm] MES failed to respond to msg=RESET
[drm] failed to reset legacy queue
[drm] reset via MES failed and try pipe reset -110
[drm] Ring gfx_0.0.0 reset failed
[drm] GPU reset begin!
GPU Coredump confirmed MES firmware version: 0x00000080
Important: The same CEMU setup worked flawlessly on ROG Ally (Z1 Extreme):
This confirms the bug is specific to gfx1150/gfx1151 (Strix Point) hardware.
Following the discussion here and ROCm issue #5590, I applied:
sudo rpm-ostree kargs --append=amdgpu.cwsr_enable=0
Result: 
Problem completely solved!
amdgpu.mes=0 - did NOT work (MES still loaded, as expected on gfx11+)amdgpu.cwsr_enable=0 is the correct workaroundMES_KIQ feature version: 6, fw version: 0x0000006f
MES feature version: 1, fw version: 0x00000080
VBIOS: AMD STRIX_B0_GENERIC 023.010.001.022.000001 (2024/12/19)
Thanks to everyone in this thread for the debugging guidance! This workaround should help other Strix Point users experiencing similar GPU hangs.
If you read the other thread where I found the fix for some people.
I have a gfx1103 (FW16 7840HS CPU) and it helps me.
The amdgpu.cwsr_enable=0 still crashes the first time after a reboot for me, but then running it again results in completion of the task without crashes.
I have a test program that reproduces the problem.
Did you try running your application twice?
I did not try booting with amdgpu.cwsr_enable=0 more than once. Now I’m testing with amdgpu.mes=0 and haven’t seen a crash in 24h.
Could you share your test program? I can’t reliably reproduce the crash, it just happens while using Chrome in random web pages with no visible WebGL, video or other GPU-heavy content.
The problem I was having is described here:
The test program is this one.
git clone GitHub - jcdutton/rocm-rust: Examples of calling rocm from rust
How to reproduce the problem is in that issues ticket.
After running kernel 6.18.2 for 30 hours with amdgpu.mes=0, I experienced a recoverable GPU hang while typing in Chrome.
The kwin_wayland process recovered with “Desktop effects were restarted due to a graphics reset”, and a signal-desktop thread crashed, but the app is still running.
Smells like an unrelated bug to me, but reporting it here just in case. If it’s something else, please let me know where I should report it.
[123402.863612] amdgpu 0000:c1:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:4 pasid:32793)
[123402.863622] amdgpu 0000:c1:00.0: amdgpu: Process signal-desktop pid 3755 thread signal-des:cs0 pid 3801
[123402.863624] amdgpu 0000:c1:00.0: amdgpu: in page starting at address 0x000083913d558000 from client 10
[123402.863626] amdgpu 0000:c1:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00401430
[123402.863627] amdgpu 0000:c1:00.0: amdgpu: Faulty UTCL2 client ID: SQC (data) (0xa)
[123402.863628] amdgpu 0000:c1:00.0: amdgpu: MORE_FAULTS: 0x0
[123402.863629] amdgpu 0000:c1:00.0: amdgpu: WALKER_ERROR: 0x0
[123402.863630] amdgpu 0000:c1:00.0: amdgpu: PERMISSION_FAULTS: 0x3
[123402.863630] amdgpu 0000:c1:00.0: amdgpu: MAPPING_ERROR: 0x0
[123402.863631] amdgpu 0000:c1:00.0: amdgpu: RW: 0x0
[123412.937414] amdgpu 0000:c1:00.0: amdgpu: Dumping IP State
[123412.939651] amdgpu 0000:c1:00.0: amdgpu: Dumping IP State Completed
[123412.939752] amdgpu 0000:c1:00.0: amdgpu: [drm] AMDGPU device coredump file has been created
[123412.939754] amdgpu 0000:c1:00.0: amdgpu: [drm] Check your /sys/class/drm/card1/device/devcoredump/data
[123412.939755] amdgpu 0000:c1:00.0: amdgpu: ring gfx_0.0.0 timeout, signaled seq=27084423, emitted seq=27084425
[123412.939758] amdgpu 0000:c1:00.0: amdgpu: Process signal-desktop pid 3755 thread signal-des:cs0 pid 3801
[123412.939760] amdgpu 0000:c1:00.0: amdgpu: Starting gfx_0.0.0 ring reset
[123414.943586] amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=RESET
[123414.943597] amdgpu 0000:c1:00.0: amdgpu: failed to reset legacy queue
[123414.943599] amdgpu 0000:c1:00.0: amdgpu: reset via MES failed and try pipe reset -110
[123414.943602] amdgpu 0000:c1:00.0: amdgpu: The CPFW hasn't support pipe reset yet.
[123414.943603] amdgpu 0000:c1:00.0: amdgpu: Ring gfx_0.0.0 reset failed
[123414.943606] amdgpu 0000:c1:00.0: amdgpu: GPU reset begin!. Source: 1
[123415.283756] amdgpu 0000:c1:00.0: amdgpu: Register(0) [regVPEC_QUEUE_RESET_REQ] failed to reach value 0x00000000 != 0x00000001n
[123415.283761] amdgpu 0000:c1:00.0: amdgpu: VPE queue reset failed
[123417.288891] amdgpu 0000:c1:00.0: amdgpu: MES failed to respond to msg=REMOVE_QUEUE
[123417.288898] amdgpu 0000:c1:00.0: amdgpu: failed to unmap legacy queue
[123417.558068] [drm:gfx_v11_0_hw_fini [amdgpu]] *ERROR* failed to halt cp gfx
[123417.559390] amdgpu 0000:c1:00.0: amdgpu: MODE2 reset
[123417.585298] amdgpu 0000:c1:00.0: amdgpu: GPU reset succeeded, trying to resume
[123417.585617] [drm] PCIE GART of 512M enabled (table at 0x0000008001700000).
[123417.585660] amdgpu 0000:c1:00.0: amdgpu: SMU is resuming...
[123417.590599] amdgpu 0000:c1:00.0: amdgpu: SMU is resumed successfully!
[123417.601526] amdgpu 0000:c1:00.0: amdgpu: [drm] DMUB hardware initialized: version=0x09003500
[123417.790282] amdgpu 0000:c1:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
[123417.790289] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0
[123417.790290] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0
[123417.790291] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 6 on hub 0
[123417.790292] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 7 on hub 0
[123417.790293] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 8 on hub 0
[123417.790293] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 9 on hub 0
[123417.790294] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 10 on hub 0
[123417.790295] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 11 on hub 0
[123417.790296] amdgpu 0000:c1:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0
[123417.790296] amdgpu 0000:c1:00.0: amdgpu: ring vcn_unified_0 uses VM inv eng 0 on hub 8
[123417.790297] amdgpu 0000:c1:00.0: amdgpu: ring jpeg_dec_0 uses VM inv eng 1 on hub 8
[123417.790298] amdgpu 0000:c1:00.0: amdgpu: ring mes_kiq_3.1.0 uses VM inv eng 13 on hub 0
[123417.790299] amdgpu 0000:c1:00.0: amdgpu: ring vpe uses VM inv eng 4 on hub 8
[123418.825354] amdgpu 0000:c1:00.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on vpe (-110).
[123418.825694] amdgpu 0000:c1:00.0: amdgpu: ib ring test failed (-110).
[123418.942965] amdgpu 0000:c1:00.0: amdgpu: MODE2 reset
[123418.970364] amdgpu 0000:c1:00.0: amdgpu: GPU reset succeeded, trying to resume
[123418.970876] [drm] PCIE GART of 512M enabled (table at 0x0000008001700000).
[123418.970912] amdgpu 0000:c1:00.0: amdgpu: SMU is resuming...
[123418.978614] amdgpu 0000:c1:00.0: amdgpu: SMU is resumed successfully!
[123418.989400] amdgpu 0000:c1:00.0: amdgpu: [drm] DMUB hardware initialized: version=0x09003500
[123419.180204] amdgpu 0000:c1:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
[123419.180211] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0
[123419.180212] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0
[123419.180213] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 6 on hub 0
[123419.180214] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 7 on hub 0
[123419.180214] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 8 on hub 0
[123419.180215] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 9 on hub 0
[123419.180215] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 10 on hub 0
[123419.180216] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 11 on hub 0
[123419.180217] amdgpu 0000:c1:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0
[123419.180218] amdgpu 0000:c1:00.0: amdgpu: ring vcn_unified_0 uses VM inv eng 0 on hub 8
[123419.180219] amdgpu 0000:c1:00.0: amdgpu: ring jpeg_dec_0 uses VM inv eng 1 on hub 8
[123419.180219] amdgpu 0000:c1:00.0: amdgpu: ring mes_kiq_3.1.0 uses VM inv eng 13 on hub 0
[123419.180220] amdgpu 0000:c1:00.0: amdgpu: ring vpe uses VM inv eng 4 on hub 8
[123419.185098] amdgpu 0000:c1:00.0: amdgpu: GPU reset(1) succeeded!
[123419.185109] amdgpu 0000:c1:00.0: [drm] device wedged, but recovered through reset
[123419.188057] amdgpu 0000:c1:00.0: amdgpu: [drm] *ERROR* Failed to initialize parser -125!
[123419.188353] traps: signal-desktop[3755] trap int3 ip:55555e0f1040 sp:7fffffffc4b0 error:0 in signal-desktop[8b9c040,555557b2a000+923d000]
git clone https://github.com/jcdutton/rocm-rust.git
Thank you for the example. I couldn’t build the binary due to a rocm dependency hell. I’ll try building in some Ubuntu LTS vm and report back.
@James3, I was able to build rocm-rust in Ubuntu 25.04, then I moved the binary to Fedora Rawhide, and thankfully it’s ABI-compatible with the rocm 7.1.1 runtime provided by the system RPMs.
I must run rocm-rust as root (likely due to /dev/kfd permissions), and it spews this error with any matrix size I tried:
[129466.339956] amdgpu 0000:c1:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:153 vmid:8 pasid:32795)
[129466.339966] amdgpu 0000:c1:00.0: amdgpu: Process rocm-rust pid 175105 thread rocm-rust pid 175105
[129466.339968] amdgpu 0000:c1:00.0: amdgpu: in page starting at address 0x00007ffff7fa6000 from client 10
[129466.339970] amdgpu 0000:c1:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00800932
[129466.339971] amdgpu 0000:c1:00.0: amdgpu: Faulty UTCL2 client ID: CPF (0x4)
[129466.339972] amdgpu 0000:c1:00.0: amdgpu: MORE_FAULTS: 0x0
[129466.339973] amdgpu 0000:c1:00.0: amdgpu: WALKER_ERROR: 0x1
[129466.339973] amdgpu 0000:c1:00.0: amdgpu: PERMISSION_FAULTS: 0x3
[129466.339974] amdgpu 0000:c1:00.0: amdgpu: MAPPING_ERROR: 0x1
[129466.339974] amdgpu 0000:c1:00.0: amdgpu: RW: 0x0
After this, it keeps running forever. GPU usage is low (~10%) and CPU usage is 100% on a single core, so I’m guessing the working set is bleeding out of GPU memory or something like that.
Note that I’m still running with amdgpu.mes=0. I didn’t try tweaking the module parameters because it would require a reboot, but I will do it eventually.
Two more hangs with amdgpu.cwsr_enable=0.
While the system was extremely slow, I was able to switch to a kmscon and take these screenshots of dmesg:
https://codewiz.org/pub/amdgpu-crash-dmesg-1.jpg
https://codewiz.org/pub/amdgpu-crash-dmesg-2.jpg
https://codewiz.org/pub/amdgpu-crash-dmesg-3.jpg
The third one has a stack trace from Chrome which might be useful. The process is stuck in kernel mode in amdgpu_mes_reg_write_reg_wait(), which is unsurprising, given that the MES ring buffer is full.
Can you please adjust your GC firmware to match upstream? There was a revert that Fedora is missing at the moment.
