What’s strange, I didn’t use MTP, or rather I tried, but was not impressed. But I still had 16 t/s without MTP, and now it’s 11.
I mean, even 16 t/s is not right for this hardware - I was getting ~42 t/s on my DGX Spark on pretty much the same VLLM version. And even then, it should be faster.
FYI, on my system it’s /dri/0:
eugr@ai:~$ sudo cat /sys/kernel/debug/dri/0/amdgpu_firmware_info | grep MES
MES_KIQ feature version: 6, firmware version: 0x0000006f
MES feature version: 1, firmware version: 0x00000080
decreased performance may be due to missing flash attention. currently not supported.
i found this branch: https://github.com/ROCm/flash-attention/tree/matthias.gfx1151
Haven’t been able to get flash attention to run. the vllm falls back to Using Triton Attention backend on V1 engine. Poor perf. 10-11 tps
edit: on top of it all Triton maybe leads to:
UserWarning: Input tensor shape suggests potential format mismatch: seq_len (17) < num_heads (32). This may indicate the inputs were passed in head-first format [B, H, T, …] when head_first=False was specified. Please verify your input tensor format matches the expected shape [B, T, H, …].
(EngineCore_DP0 pid=195) return fn(*contiguous_args, **contiguous_kwargs)
This further trips performance. we are in the early days.
Tried to run GLM 4.6V AWQ 4-bit, but no luck:
(EngineCore_DP0 pid=474) ValueError: Failed to find a kernel that can implement the WNA16 linear layer. Reasons:
(EngineCore_DP0 pid=474) CutlassW4A8LinearKernel cannot implement due to: CUTLASS only supported on CUDA
(EngineCore_DP0 pid=474) MacheteLinearKernel cannot implement due to: Machete only supported on CUDA
(EngineCore_DP0 pid=474) AllSparkLinearKernel cannot implement due to: AllSpark currently does not support device_capability = 115.
(EngineCore_DP0 pid=474) MarlinLinearKernel cannot implement due to: Marlin only supported on CUDA
(EngineCore_DP0 pid=474) Dynamic4bitLinearKernel cannot implement due to: Only CPU is supported
(EngineCore_DP0 pid=474) BitBLASLinearKernel cannot implement due to: bitblas is not installed. Please install bitblas by running `pip install bitblas>=0.1.0`
(EngineCore_DP0 pid=474) ConchLinearKernel cannot implement due to: Group size (32) not supported by ConchLinearKernel, supported group sizes are: [-1, 128]
(EngineCore_DP0 pid=474) ExllamaLinearKernel cannot implement due to: Exllama only supports float16 activations
(EngineCore_DP0 pid=474) XPUwNa16LinearKernel cannot implement due to: IPEX wNa16 only supported on XPU/CPU devices
Tried bitblas, but it seems to depend on NVIDIA stuff.
The latest llama.cpp gives much better performance, on Qwen3-Next-80B, could reach > 30 TPS on gfx1151.
For folks who are running into vLLM issues, please file them at AMD · GitHub or AMD CI · GitHub
The team at AMD is monitoring these lists daily to make sure vLLM is stable and competitive on AMD hardware.
Yes, llama.cpp is by far better performing, and still is subpar to what it should be. This has to do with the design of the deltanet and its implementation in llama.cpp. There is an optimization thread on the llama.cpp github if you are interested. The bottom line is that there are multiple chunked computations due to the design of deltanet.
Surprisingly, the vulkan backend is more performant in (tg/s) than the rocm one. This I have not been able to investigate yet.
I think vLLM is further away from being a serious platform at the moment to run inference with the strix halo. I say this from my experience copiling both from source. It has a greater potential but is not ready yet, especially when it comes to qwen3-next-80B.
Seems like llama.cpp with rocm-7.1.1, produced better performance over vulkan backend, with prefill around 400tps, tps around 30 tps.
i agree, pp is better on rocm, but tg is lower. in the case of qwen3-next, none of this matters as the architecture will need a fused-kernel core to run at practical maximum. on bandwidth limited devices like the strix-halo framework desktop MoE models are king, but not qwen3-next.
Looks like llama.cpp is the only viable inferencing engine on Strix Halo at the moment…
Well, DGX Spark has similar bandwidth, but I’m getting 44 t/s on this model (in FP8 quant!) in vLLM…
Right, I have a private llama.cpp branch pushing 40 tg/s but is this the performance expected on the basis of the theoretical or lets say 80% of theoretical bandwidth from a MoE model?
Similarly from a different perspective if you will, what is the performance of Qwen3-Next-80B-A3B on RTX Pro 6000 Blackwell? And again is this what we would expect as performance based on 80% theoretical bandwidth?
The answer is the real issue.
From this perspective, definitely no.
Theoretical max for the model with 3B active parameters (FP8) on Spark would be 273/3 ~= 90 t/s at zero context. Given that memory never performs at max and adding all the routing/compute overhead, we probably lose at least 20% of this, so ~72 t/s.
Qwen3-VL-30B-A3B-FP8 that has the same number of active parameters, on the same vLLM/hardware gives me 52 t/s.
So, yeah, I see your point here - Qwen3-Next is underperforming for its size.
I think the Nemotron or some variation of even more hybridized models will make inference on these UMA hardware architectures most appealing.
For example, nvidia is releasing now the nemotron 3 nano (30B A3B), but it will follow it up with a 100B A10B Nemotron 3 Super and even an Ultra variant in the family. All hybrid MoE models. What Is Mixture of Experts (MoE) and How It Works?
All inference engines however, especially llama.cpp,will have to really double down on efficient fused kernel and careful implementation of these architectures.
Right now, this is not the case.
Thank you Eugr, your recipe really helped me!
We can replace the tarball with a more simple solution:
rocm-sdk init
export ROCM_PATH=$(python -c "import site; print(site.getsitepackages()[0])")/_rocm_sdk_devel
Or get path with on of the rocm-sdk path <> cmd.
(but it is more to use the python package to build c++/hip code like llama.cpp)
I tried that at first, but ROCm SDK that was bundled with python install was missing a few things related to HIP compiler, so I ended up downloading the tarballs instead. Don’t know if they fixed it.
Are you able to compile with the SDK included in their python distribution?
I’ve just tried, and I’m getting this error:
In file included from /usr/include/hip/amd_detail/amd_device_functions.h:347:
/usr/include/hip/amd_detail/amd_warp_functions.h:96:37: error: use of undeclared identifier '__AMDGCN_WAVEFRONT_SIZE'
96 | static constexpr int warpSize = __AMDGCN_WAVEFRONT_SIZE;
| ^~~~~~~~~~~~~~~~~~~~~~~
In file included from /tmp/tmpxzuqqa6n.build-temp/csrc/rocm/attention.hip:24:
/tmp/tmpxzuqqa6n.build-temp/csrc/rocm/../hip_compat.h:17:7: warning: ignoring return value of type 'hipError_t' declared with 'nodiscard' attribute [-Wunused-value]
17 | hipGetDevice(&device_id);
| ^~~~~~~~~~~~~~~~~~~~~~~~
/tmp/tmpxzuqqa6n.build-temp/csrc/rocm/../hip_compat.h:18:7: warning: ignoring return value of type 'hipError_t' declared with 'nodiscard' attribute [-Wunused-value]
18 | hipGetDeviceProperties(&deviceProp, device_id);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/hip/hip_runtime_api.h:92:32: note: expanded from macro 'hipGetDeviceProperties'
92 | #define hipGetDeviceProperties hipGetDevicePropertiesR0600
| ^
/tmp/tmpxzuqqa6n.build-temp/csrc/rocm/attention.hip:3640:5: warning: ignoring return value of type 'hipError_t' declared with 'nodiscard' attribute [-Wunused-value]
3640 | hipGetDevice(&device_id);
| ^~~~~~~~~~~~~~~~~~~~~~~~
/tmp/tmpxzuqqa6n.build-temp/csrc/rocm/attention.hip:3641:5: warning: ignoring return value of type 'hipError_t' declared with 'nodiscard' attribute [-Wunused-value]
3641 | hipGetDeviceProperties(&deviceProp, device_id);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/hip/hip_runtime_api.h:92:32: note: expanded from macro 'hipGetDeviceProperties'
92 | #define hipGetDeviceProperties hipGetDevicePropertiesR0600
| ^
4 warnings and 1 error generated when compiling for gfx1151.
[33/37] Building CXX object CMakeFiles/_rocm_C.dir/csrc/rocm/torch_bindings.cpp.o
[34/37] Building CXX object CMakeFiles/_C.dir/csrc/torch_bindings.cpp.o
ninja: build stopped: subcommand failed.
Can remember, I success to build llama.cpp with it…
But I think I need to remove all rocm install (ie the one from fedora rpm…)
But I test that 1 month ago so… may be my memory is wrong 