# \[HOW-TO\] Compiling VLLM from source on Strix Halo **URL:** \ **Category:** Framework Desktop\ **Tags:** ai\ **Created:** [October 28, 2025, 8:48pm UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241 "2025-10-28T20:48:44Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![Ziliang\_Qian](https://avatars.discourse-cdn.com/v4/letter/z/c4cdca/32.png) [@Ziliang\_Qian](https://community.frame.work/u/Ziliang_Qian)\ **Post date:** [November 5, 2025, 6:02am UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/22 "2025-11-05T06:02:54Z")
Any way to get MTP(multiple token prediciton) work, with vllm/rocm? ---
**Author:** ![Eugr](https://sea1.discourse-cdn.com/flex001/user_avatar/community.frame.work/eugr/32/42968_2.png) [@Eugr](https://community.frame.work/u/Eugr)\ **Post date:** [November 5, 2025, 6:34am UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/23 "2025-11-05T06:34:46Z")
Haven’t tried yet, so no idea. ---
**Author:** ![Ziliang\_Qian](https://avatars.discourse-cdn.com/v4/letter/z/c4cdca/32.png) [@Ziliang\_Qian](https://community.frame.work/u/Ziliang_Qian)\ **Post date:** [November 7, 2025, 8:56pm UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/24 "2025-11-07T20:56:14Z")
Got MTP work. TPS dropped sharply from 15 tps to 8 tps. ---
**Author:** ![Eugr](https://sea1.discourse-cdn.com/flex001/user_avatar/community.frame.work/eugr/32/42968_2.png) [@Eugr](https://community.frame.work/u/Eugr)\ **Post date:** [November 7, 2025, 9:46pm UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/25 "2025-11-07T21:46:19Z")
Yeah, I tried it on my Spark too and it dropped from 43 t/s to 30 t/s or so. ---
**Author:** ![Ian\_MacDonald](https://avatars.discourse-cdn.com/v4/letter/i/f19dbf/32.png) [@Ian\_MacDonald](https://community.frame.work/u/Ian_MacDonald)\ **Post date:** [November 11, 2025, 3:03am UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/26 "2025-11-11T03:03:05Z")
Not much lift with v0.11.1 .. so many paths with `amdgpu: MES failed to respond to msg=REMOVE_QUEUE` vllm serve cpatonn/Qwen3-Next-80B-A3B-Thinking-AWQ-4bit --dtype float16 --max-num-seqs 1 --max-model-len 32768 --enforce-eager --gpu-memory-utilization 0.8 (APIServer pid=2254) INFO 11-11 02:29:37 [api\_server.py:1965] vLLM API server version 0.11.1rc6 ```auto ============ Serving Benchmark Result ============ Successful requests: 1 Failed requests: 0 Request rate configured (RPS): 10000.00 Benchmark duration (s): 59.91 Total input tokens: 1000 Total generated tokens: 1000 Request throughput (req/s): 0.02 Output token throughput (tok/s): 16.69 Peak output token throughput (tok/s): 17.00 Peak concurrent requests: 1.00 Total Token throughput (tok/s): 33.38 ---------------Time to First Token---------------- Mean TTFT (ms): 893.92 Median TTFT (ms): 893.92 P99 TTFT (ms): 893.92 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 59.08 Median TPOT (ms): 59.08 P99 TPOT (ms): 59.08 ---------------Inter-token Latency---------------- Mean ITL (ms): 59.08 Median ITL (ms): 59.02 P99 ITL (ms): 59.62 ================================================== ``` ---
**Author:** ![Ian\_MacDonald](https://avatars.discourse-cdn.com/v4/letter/i/f19dbf/32.png) [@Ian\_MacDonald](https://community.frame.work/u/Ian_MacDonald)\ **Post date:** [November 12, 2025, 10:01pm UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/27 "2025-11-12T22:01:56Z")
I dropped reproducable uv recipe with specific git commits, for deb based distros that should be adaptable to any distro with a few path tweaks, if you are looking to skip all the container cruft [[Question] Is GFX1201 support planned? · Issue #900 · ROCm/aiter · GitHub](https://github.com/ROCm/aiter/issues/900) ---
**Author:** ![Daniel\_H212](https://avatars.discourse-cdn.com/v4/letter/d/b19c9b/32.png) [@Daniel\_H212](https://community.frame.work/u/Daniel_H212)\ **Post date:** [November 24, 2025, 5:00am UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/28 "2025-11-24T05:00:10Z")
Is there support for using vLLM to run gpt-oss-120b on strix halo yet? I’m under the impression that MXFP4 isn’t supported on AMD, but gguf is supported and there are gguf quants, so would those be a good idea, or do they all use MXFP4 in some way under the hood (considering the similar sizes) and wouldn’t work? ---
**Author:** ![Djip](https://avatars.discourse-cdn.com/v4/letter/d/82dd89/32.png) [@Djip](https://community.frame.work/u/Djip)\ **Post date:** [November 24, 2025, 10:12am UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/29 "2025-11-24T10:12:45Z")
look like some (most) linux python package are not upload to release. (because of test failed?) but there is more on staging path: [https://rocm.nightlies.amd.com/v2-staging/](https://rocm.nightlies.amd.com/v2-staging/) with [gfx1151](https://rocm.nightlies.amd.com/v2/gfx1151/) / [gfx120X-all](https://rocm.nightlies.amd.com/v2/gfx120X-all/) / [gfx110X-all](https://rocm.nightlies.amd.com/v2/gfx110X-all/) (new with gfx1103 (re)added ) more element [[Issue]: torch linux not build for new gfx110X-all · Issue #1939 · ROCm/TheRock · GitHub](https://github.com/ROCm/TheRock/issues/1939#issuecomment-3453345328) ---
**Author:** ![Christian\_Kniep](https://avatars.discourse-cdn.com/v4/letter/c/919ad9/32.png) [@Christian\_Kniep](https://community.frame.work/u/Christian_Kniep)\ **Post date:** [November 24, 2025, 12:35pm UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/30 "2025-11-24T12:35:43Z")
Ok, what can we do about it then? No vllm for the box yet? ☹ ---
**Author:** ![Eugr](https://sea1.discourse-cdn.com/flex001/user_avatar/community.frame.work/eugr/32/42968_2.png) [@Eugr](https://community.frame.work/u/Eugr)\ **Post date:** [November 25, 2025, 7:45pm UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/31 "2025-11-25T19:45:33Z")
I haven’t touched vllm on Strix Halo for a couple of weeks now, so no idea if MXFP4 works there. It’s even broken for DGX Spark now - you have to roll back to Marlin kernel for it to work, so there is that 🙂 I may try when I have time. ---
**Author:** ![Daniel\_H212](https://avatars.discourse-cdn.com/v4/letter/d/b19c9b/32.png) [@Daniel\_H212](https://community.frame.work/u/Daniel_H212)\ **Post date:** [December 3, 2025, 3:49am UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/32 "2025-12-03T03:49:31Z")
I encountered an issue where vllm kept trying to use aiter, even when environment variables were set to not use aiter. I then found out aiter doesn’t support gfx1151, but there is an unmerged PR adding support here: [Add gfx11XX targets by mgehre-amd · Pull Request #1498 · ROCm/aiter · GitHub](https://github.com/ROCm/aiter/pull/1498) I installed that with uv pip install --no-deps “git+https://github.com/ROCm/aiter.git@mgehre-amd/gfx11” I managed to run gpt-oss-20b, the original model, so mxfp4 support seems working though it also seems to be a bit slow. Had to set dtype to bfloat16 and add –trust-remote-code. I tried to run Qwen3-Next the same way you did, and I got: RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::Half [rank0]:[W1202 22:27:07.836698084 ProcessGroupNCCL.cpp:1552] I had to replace –dtype float16 with –dtype auto, which got past that issue, but now I’m getting: TypeError: Qwen3NextMTP.forward() missing 1 required positional argument: ‘intermediate\_tensors’ [rank0]:[W1202 22:40:05.091632576 ProcessGroupNCCL.cpp:1552] and I don’t know how to get around this. ---
**Author:** ![Eugr](https://sea1.discourse-cdn.com/flex001/user_avatar/community.frame.work/eugr/32/42968_2.png) [@Eugr](https://community.frame.work/u/Eugr)\ **Post date:** [December 4, 2025, 8:43pm UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/33 "2025-12-04T20:43:16Z")
Looks like the fix has been merged into AITER - trying to run Qwen3-Next with a fresh VLLM build now. Looks like Qwen did something to the model, as it is re-downloading the weights. ---
**Author:** ![\_sk](https://sea1.discourse-cdn.com/flex001/user_avatar/community.frame.work/_sk/32/34130_2.png) [@\_sk](https://community.frame.work/u/_sk)\ **Post date:** [December 4, 2025, 9:25pm UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/34 "2025-12-04T21:25:01Z")
@Eugr did you manage to make it run? How does it compare to llama.cpp in pp an tg? ---
**Author:** ![Eugr](https://sea1.discourse-cdn.com/flex001/user_avatar/community.frame.work/eugr/32/42968_2.png) [@Eugr](https://community.frame.work/u/Eugr)\ **Post date:** [December 4, 2025, 9:35pm UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/35 "2025-12-04T21:35:25Z")
Yes, it runs, but super slow - getting 11 t/s which is worse than I was getting before (16 t/s). But the previous one was using ROCm/PyTorch nightly and outside of Docker, for this one I tried Dockerfile from VLLM. I had to uninstall and reinstall AITER though. I guess I need to try to compile vllm on host using my previous method and compare. Here is what I’ve done now (I’m using Podman on Fedora instead of Docker, so the parameters are a little bit different; for Docker just follow [the guidance from VLLM](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/#build-an-image-with-vllm): Build: ```bash mkdir vllm-docker cd vllm-docker git clone --recursive https://github.com/vllm-project/vllm.git cd vllm DOCKER_BUILDKIT=1 podman build -f docker/Dockerfile.rocm -t vllm-rocm --build-arg ARG_PYTORCH_ROCM_ARCH=gfx1151 --format docker . ``` Run: ```bash podman run -it --rm \ --network=host \ --group-add=keep-groups \ --ipc=host \ --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ --security-opt label=disable \ --device /dev/kfd \ --device /dev/dri \ -v ~/.cache/huggingface:/root/.cache/huggingface:Z \ vllm-rocm ``` Inside, run a model: ```bash pip uninstall aiter pip install --no-deps "git+https://github.com/ROCm/aiter.git" vllm serve cpatonn/Qwen3-Next-80B-A3B-Thinking-AWQ-4bit --port 8888 --host 0.0.0.0 --max-model-len 32768 --max-num-seqs 10 ``` ---
**Author:** ![\_sk](https://sea1.discourse-cdn.com/flex001/user_avatar/community.frame.work/_sk/32/34130_2.png) [@\_sk](https://community.frame.work/u/_sk)\ **Post date:** [December 4, 2025, 10:00pm UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/36 "2025-12-04T22:00:06Z")
yes there is something funky going on. few things which can be: pass the render and video group. idk if needed: `podman run -it --rm ` `–network=host ` `–group-add=keep-groups ` `–device /dev/kfd ` `–device /dev/dri ` `–group-add video ` `–group-add render \` is the gpu detected, maybe the load falls back to cpu in hte container: `rocminfo | grep -i gfx1151` but indeed the build inside has to be gfx1151 compatible. at least podman/ docker offer this portability and cleanliness. ---
**Author:** ![Eugr](https://sea1.discourse-cdn.com/flex001/user_avatar/community.frame.work/eugr/32/42968_2.png) [@Eugr](https://community.frame.work/u/Eugr)\ **Post date:** [December 4, 2025, 10:04pm UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/37 "2025-12-04T22:04:23Z")
No, the docker build sees GPU just fine, builds CUDA graphs, etc. I’ve just also tried to build on the host and got the same 11 t/s for this model, so there has to be some regression in VLLM itself, given that I’m using the same pytorch/triton/flash\_attn as before. I don’t have time to debug this now, but at least it’s good that the “official” docker build works now with just a minor change (fresh aiter build). I’ll try again in a few weeks - don’t have time to spend on this as vllm support on Strix Halo is not essential for me anymore, now that I have a cluster with dual DGX Sparks. Not that it was trouble-free there, but at least I got it working with more or less acceptable performance. ---
**Author:** ![Daniel\_H212](https://avatars.discourse-cdn.com/v4/letter/d/b19c9b/32.png) [@Daniel\_H212](https://community.frame.work/u/Daniel_H212)\ **Post date:** [December 5, 2025, 2:40am UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/38 "2025-12-05T02:40:08Z")
> [@Eugr](#): > > No, the docker build sees GPU just fine, builds CUDA graphs, etc. I’ve just also tried to build on the host and got the same 11 t/s for this model, so there has to be some regression in VLLM itself, given that I’m using the same pytorch/triton/flash\_attn as before. I think one of the things that happened was they broke multi-token prediction for Qwen3-Next. I seem to recall it working at one point, but now attempting to enable it gets the following error on model load: TypeError: Qwen3NextMTP.forward() missing 1 required positional argument: ‘intermediate\_tensors’ [rank0]:[W1202 22:40:05.091632576 ProcessGroupNCCL.cpp:1552] Not sure if that is the cause of your performance loss though. I’m trying to find a fix for this right now. I can confirm that it is running at only ~9.5 t/s for me at 16-20k context. ---
**Author:** ![Daniel\_H212](https://avatars.discourse-cdn.com/v4/letter/d/b19c9b/32.png) [@Daniel\_H212](https://community.frame.work/u/Daniel_H212)\ **Post date:** [December 5, 2025, 3:33am UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/39 "2025-12-05T03:33:45Z")
Also, since this thread has been probably the most helpful resource for running vLLM on strix halo that I’ve seen, I should note for anyone who comes across this guide: If you are getting errors that look like one of these two: `Memory access fault by GPU node-1 (Agent handle: 0x55981f276340) on address 0x7f4812b5a000. Reason: Page not present or supervisor privilege.` `HW Exception by GPU node-1 (Agent handle: 0x55a709dc3390) reason :GPU Hang` You should first check your MES version by running: ```auto sudo cat /sys/kernel/debug/dri/1/amdgpu_firmware_info | grep MES ``` which should get you something like `MES feature version: 1, firmware version: 0x00000080` You want it to say 80 at the end, not 83. 83 causes memory access fault almost immediately upon trying to load a model. If you update amd-gpu-firmware and linux-firmware to 20251125, the MES firmware will update to 83 and it’s annoying rolling that back. Stay on a version before 20251125, like 20251111 or 20251021, until amd fixes the issue. If you do update to 20251125, MES firmware may update to 83 and not downgrade automatically even if you downgrade amd-gpu-firmware and linux-firmware (at which point consult ChatGPT or something for a guide on how to roll back MES firmware specifically, I don’t quite understand the steps so I will not repeat them here). Secondly, if you are on MES firmware version 80, GPU hangs and memory access faults still happen, just only when you start hitting memory pretty heavily or when you try to do other desktop tasks while vLLM is running in the background. To fix that, add: ```auto amdgpu.cwsr_enable=0 ``` to your kernel parameter by doing: ```auto sudo nano /etc/default/grub ``` and paste amdgpu.cwsr\_enable=0 into the line that starts with GRUB\_CMDLINE\_LINUX=, like this: ```auto GRUB_CMDLINE_LINUX="rhgb quiet amdgpu.cwsr_enable=0" ``` This is the same line in which you set GTT memory allocation. Change this, save, and reboot. This fixed the GPU hang and memory access faults, at the cost of possibly some frame drops/choppiness on the desktop, but so far I’ve not noticed anything. ---
**Author:** ![Daniel\_H212](https://avatars.discourse-cdn.com/v4/letter/d/b19c9b/32.png) [@Daniel\_H212](https://community.frame.work/u/Daniel_H212)\ **Post date:** [December 5, 2025, 4:21am UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/40 "2025-12-05T04:21:12Z")
I was able to get MTP on Qwen3-Next working. However, I’m not sure if it is worth it. Seems like the small prediction model performs too poorly at longer context lengths and actually worsens performance. Within the first few thousand tokens of context it adds maybe 15% performance but at the 16k token context that I was testing, it actually leads to about -10% performance. If anyone here wants to try, there are two ways to get MTP working for Qwen3-Next, one is by doing –enforce-eager (which has it’s own negative performance impact) and one is by patching the qwen3\_next\_mtp.py file in vllm (found in /vllm/model\_executor/models/), which I managed to do with the help of ChatGPT. Basically the issue is that torch compile thinks there’s something wrong, when there actually isn’t anything wrong, so we either use –enforce-eager to skip torch compile entirely or modify the file to skip torch compile for the specific problematic part, which is done by: Replace ```auto from vllm.compilation.decorators import support_torch_compile ``` with: ```auto from vllm.compilation.decorators import support_torch_compile, ignore_torch_compile ``` then replace ```auto @support_torch_compile class Qwen3NextMTP(nn.Module, SupportsPP, QwenNextMixtureOfExperts): ... ``` with: ```auto @ignore_torch_compile class Qwen3NextMTP(nn.Module, SupportsPP, QwenNextMixtureOfExperts): ... ``` I was also told to replace: ```auto hidden_states = self.model( input_ids, positions, hidden_states, intermediate_tensors, inputs_embeds ) ``` under `def forward(` in the `Qwen3NextMTP` class with: ```auto hidden_states = self.model( input_ids=input_ids, positions=positions, hidden_states=hidden_states, intermediate_tensors=intermediate_tensors, inputs_embeds=inputs_embeds, spec_step_idx=spec_step_idx, ) ``` Here is how I ran the model: ```auto vllm serve /path/to/folder/containing/downloaded/model \ --served-model-name "qwen3-next-80b-a3b-thinking (vllm 4 bit)" \ --dtype auto \ --quantization compressed-tensors \ --max-model-len 65536 \ --max-num-seqs 4 \ --gpu-memory-utilization 0.9 \ --reasoning-parser deepseek_r1 \ --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' \ --host 0.0.0.0 \ --port ``` ---
**Author:** ![\_sk](https://sea1.discourse-cdn.com/flex001/user_avatar/community.frame.work/_sk/32/34130_2.png) [@\_sk](https://community.frame.work/u/_sk)\ **Post date:** [December 5, 2025, 5:56am UTC](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241/41 "2025-12-05T05:56:34Z")
@Daniel_H212 what tps are you getting with MTP? [Previous page](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241.md?page=1) [Next page](https://community.frame.work/t/how-to-compiling-vllm-from-source-on-strix-halo/77241.md?page=3)