Hey everyone, I wanted to share GitHub - shisa-ai/hipEngine · GitHub - a new open source (from-scratch) inference engine that is a pure AMD ROCm implementation (all hot-path code is HIP/C++, all AMD libraries, PyTorch is expressly not a dependency) for RDNA3 GPUs.
It’s been mostly tuned on a spare gfx1100 (W7900, 7900 XTX) GPU, but I did an initial pass for gfx1151 (Strix Halo) support, and it ends up being faster than llama.cpp (HIP or Vulkan) basically across the board for Qwen 3.6 35B-A3B (basically the only model supported atm - you can think of hipEngine currently as closer to something like antirez’s DS4 than llama.cpp).
If you’re running a llama.cpp Vulkan variant, you should expect about 10% faster decode/token generation, and >2X prefill/prompt processing (much faster agentic/coding performance). Versus llama.cpp HIP, it’s up to 10% faster prefill/prompt processing (faster as context gets longer), and up to 30% faster decode/token generation.
Prefill tok/s
Workload
hipEngine PARO
llama.cpp HIP
llama.cpp Vulkan
512/128
983.206
1058.738
638.008
4K/128
1029.402
1004.220
595.400
32K/128
792.296
735.534
407.984
128K/128
413.489
376.070
181.453
Decode tok/s
Workload
hipEngine PARO
llama.cpp HIP
llama.cpp Vulkan
512/128
62.060
50.537
57.615
4K/128
63.605
49.379
55.027
32K/128
50.629
43.435
44.576
128K/128
30.245
31.286
26.935
I announced this on reddit last week to get some testers/eyeballs/feedback on this so it should be in good shape. If anyone tries it out, feel free to drop feedback or file an issue in the Github if you run into any problems.
There has been almost no Strix Halo specific tuning. I have some dedicated test hardware coming soon from Framework though and expect that I can squeeze more out of gfx1151 with some dedicated compute time.
There is decently fast Qwen 3.6 dense support, althought MTP/DFlash is still forthcoming so you may be better off with llama.cpp for that
I’m cooking up some new model support (StepFun 3.7, Gemma 4) and am open to requests.
This started off as a sort of thought experiment, but turned out to be worth sharing. I am working on c>1 perf now, and I will be porting some of my kvcache work, etc in my spare time. There’s not really a roadmap or anything, this is just for fun, but maybe it’ll be useful for some people!
The code is AGPLv3 (share-alike for reals), but I’ve also published a fair amount of docs/ that should be useful for anyone interested in RDNA3 GPU development, and includes extensive details on the AI-assisted kernel optimization approach used.
I did try it, but MTP beat it in my local setup (with nim coding etc) by about 30%. It seemed faster for the default, but couldnt’ handle mtp. Sadness.
MTP/DFlash are WIP atm, although it’s more of a grind than expected (verification is a bottleneck). Also, c>1 and a few other things (StepFun 3.7) are grinding away.
I’ll do a proper update soon now that I’m back from some out of town travel but a few good things:
I have a dedicated gfx1151 board courtesy of Framework now to do ongoing kernel grinding specifically for Strix Halo. Expect to see some big performance gains soon
Concurrency, while not amazing, is also now running. I expect gfx1151 to benefit more than gfx1100. llama.cpp really falls apart at c>1 so this should be a durable improvement for anyone looking for fast multiuser/agent support: GitHub - shisa-ai/hipEngine · GitHub
Standard bare motherboard, I’ve just plugged it in, so I’ll have this running in the corner and just grinding.
BTW @Thomas_Munn I just finished my MTP optimization pass. >AR is very hard for the Qwen 3.5 MoE (combinations of experts, attention, linear layers) but it is outperforming llama.cpp now so I assume that once I dig into the gfx1151, there will be a benefit.
llama.cpp’s c=2 is great actually, but it dies above that - at c=8, llama.cpp’s aggregate throughput is basically equal to hipEngine’s per-sequence performance (hipEngine is 7.2X faster total throughput).
For those following along, over the past month I’ve been grinding iterations on kernels, and it’s been interesting to see what’s shaken out. Thanks again to the folks at Framework for sending a dedicated motherboard to make this work possible.
(basically, Vulkan RADV/ACO appears to be in some important cases, much more performant than ROCm HIP/LLVM on RDNA3, and even more so for RDNA 3.5. I’ve built a new set of HIP vs Vulkan microbenchmarks that anyone can run if they’re interested).
So, what has a month or so of grinding on gfx1151 led to? Firstly, some big improvements on prefill and memory usage:
Prefill tok/s
Workload
hipEngine PARO
hipEngine GGUF
llama.cpp HIP
llama.cpp Vulkan
512/128
1140.101
430.767
1061.260
1067.770
1K/128
1208.343
437.467
1043.230
1069.870
4K/128
1089.031
403.946
1009.240
1016.580
32K/128
906.145
369.942
743.547
814.923
64K/128
716.775
334.395
573.611
660.974
128K/128
474.641
270.601
390.441
476.788
Decode tok/s
Workload
hipEngine PARO
hipEngine GGUF
llama.cpp HIP
llama.cpp Vulkan
512/128
66.767
49.536
50.939
62.396
1K/128
61.746
52.192
50.818
62.136
4K/128
62.715
52.999
50.126
60.097
32K/128
50.342
43.947
44.240
51.319
64K/128
42.094
37.477
39.326
44.422
128K/128
30.386
27.862
32.114
34.948
Peak memory GiB
Workload
hipEngine PARO
hipEngine GGUF
llama.cpp HIP
llama.cpp Vulkan
512/128
18.039
21.478
21.375
21.551
1K/128
18.051
21.710
21.387
21.501
4K/128
19.026
22.995
21.444
21.507
32K/128
19.729
23.559
21.987
22.191
64K/128
20.403
24.203
22.666
22.627
128K/128
22.124
25.493
23.862
24.254
Secondly, massively improved GGUF and GGUF+MTP support that basically matches llama.cpp. One thing worth noting here. It took much longer than expected to match llama.cpp - it turns out llama.cpp’s MTP implementation is very fast because it is not exactly numerically correct. I’ve retained an “exact” mode, where you do get some MTP benefit, but also have a “llama-compat” mode to match llama.cpp performance.
Metric
hipEngine GGUF exact/default
hipEngine GGUF llama-compat
llama.cpp HIP
Route
B5, fixed 10 cycles
B2, natural24/cyclecap24
B2, natural25 request / 24 timed transitions
Canonical/native MTP decode
51.81 tok/s (0.9571x own AR)
69.50 tok/s (1.2776x own AR)
69.44 tok/s native (1.3752x own AR; not cross-engine comparable)
Cross-engine MTP decode-transition rate
n/a: fixed-cycle horizon
69.38 tok/s
66.66 tok/s
Cross-engine own AR transition rate
n/a: fixed-cycle horizon
54.40 tok/s
48.47 tok/s
Cross-engine MTP / own AR
n/a
1.2755x
1.3752x
Draft acceptance
72.33%
77.72%
79.56%
Accepted draft/output
53.49%
59.58%
57.60%
Full-cycle/predicted wall per counted output or timed transition
19.360 ms/output
14.413 ms/output
15.001 ms/transition
State/commit contract
exact/default, serial-prefix preserving
direct partial commit/dp4a; accuracy-traded
native llama.cpp compatibility target
Scope
Prompts
True AR tok/s
llama-compat tok/s
MTP / AR
Draft acceptance
Accepted/output
Cycle wall/output
Full
10
54.40
69.50
1.2776x
77.72%
59.58%
14.413 ms
Train
6
54.44
70.96
1.3034x
82.08%
60.42%
14.116 ms
Heldout
4
54.33
67.42
1.2408x
71.79%
58.33%
14.858 ms
code
4
54.42
74.81
1.3747x
91.04%
63.54%
13.387 ms
general_en
2
54.50
67.62
1.2407x
71.79%
58.33%
14.811 ms
general_ja
2
54.40
66.60
1.2242x
69.23%
56.25%
15.042 ms
mixed_ja_en
2
54.25
64.90
1.1964x
69.23%
56.25%
15.438 ms
Now of course, all this work just to match llama.cpp isn’t all that impressive, but the next update will be c=N (updated concurrency numbers), which I think will be significantly more interesting.
This is great work. I find the MTP findings quite interesting. Looking forward to further progress. Compared to last sept things are MUCH better. I do miss the exponential growth in “new model land” Its still cool that we are ‘good enough’ to code in weird languages like NIM with current home tech though!
@lhl I’m sorry for a noob question but is there some reason why llama.cpp HIP can’t be improved? I’ve been looking into adding a discrete GPU to this platform as the sparse experts are what the big memory pool are good for, but it seems like llama.cpp is the best backend for splitting across devices.
At first I hadn’t noticed how big the prefill speedup was from your work, there seems to be a lot of room for improvement.
llama.cpp HIP can be improved, but over the past couple years, I’ve seen that the maintainers simply do not take/prioritize submissions from outsiders for improving ROCm/HIP performance and are happy to leave things broken for months.
While I think it’s understandable and I have empathy for the maintainers since there’s a lot of lower quality work (especially in the age vibe-coding), fighting for trivial PRs isn’t high on my “things I like to do w/ my free time list.” llama.cpp also doesn’t really play so well with the things I wanted to experiment with (custom kvcache eviction schemes, higher concurrency kernels, etc) and since I had really good lucky building my own engine w/ FastDMS, I decided to just go my own way.
I think the trend for the Strix Halo community atm is to fork llama.cpp and just do their own thing for particular models/optimizations if you’re looking for something more llama.cpp-like (warning, a fair number of these have major quality/correctness issues):
Thanks for the answer. With LLMs being so competent I’m thinking of trying to custom patch what I can. One hacky llama.cpp compatible solution (I think…) to the multi-device problems I came up with is disabling MTP then running a separate draft model, since your results show llama.cpp HIP seems to be OK, but there’s major overhead in MTP.
I haven’t tested your hipEngine paroquant W4 (35b) for coding yet but will try it when I get back from vacation, also will see if I can quantize Ornith 1.5 the same way, but I think something closer to frontier is still a huge step up in productivity.
A couple warnings, while LLMs are good, they are also prone to mistakes and it’s easy to get in trouble vibe-coding with them. I have to take back my recommendation for Nathanw1014’s fork for example. While working on my Qwen 3.8 Flash Next implementation and testing/comparing numbers, I discovered that it fails repeatibility for all my test cases - on a 1024 token array, it produced 16 distinct continuations for 16 tries. (llama.cpp itself has some accuracy compromises, but they’re usually deterministic (or bounded bit-flips)).
BTW, while ParoQuant is very neat research [2511.10645] ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference and I’m happy with my implementation, it is non-trivial to quantize compared to many other quant schemes (it can take days), so I’ve swapped to regular GGUF K-quants and have been working to make it as fast as my PARO path. I haven’t been at a good stopping point to cut a new release, but HEAD is generally working. Here’s what’s new:
Qwen 3.x MoE was my primary target but now Qwen 3.x dense (eg, Qwen 3.8 27B is performant as well)
Initial Qwen 3.8 Flash Next support, tuning now
Brand new concurrency engine - the prior one was efficient only with very specific submission windows, the new one does better continuous batching and should have much better real world performance (on W7900, same-protocol c1–c8 results improved by as much as 84% vs before)
New concurrency also supports better KV pooling and I’ve implemented DMS now as well which gives a lossless free halving for KV storage (not such a big deal for Strix Halo, but important for other GPUs)
MTP work is ongoing. It’s competent and beats llama.cpp HIP (but not Vulkan atm); system now does continuous specdec w/ better scaling to boot - n-gram proposals can compose as well, although I haven’t found the perf to be that strong in my testing. Basic DFlash2 support added too.
Added more reasoning effort controls and other serving support (max_completion_tokens, etc)
Lots and lots of new and improved kernels
For best coding models on a single Strix Halo, I think Qwen 3.8 27B and Qwen 3.8 Flash Next (122B-A6B) are by far the best. DS4 Flash is good, but I don’t think the quants that fit it into <128GB are doing it any favors. Same with GLM-5.3 Flash, which is much too large for a single Strix Halo.
I ordered a V100 with the idea of layer splitting the more often read weights. I haven’t written any of this CUDA/HIP stuff before, so my plan is have multiple coding agents take turns reviewing/debugging until it works
From what I learned, the plan is to tap an early transformer layer to a custom trained draft head (which is small enough to quickly train on 1 GPU) to avoid PCI-e crossings for the draft tokens, and then all the LLMs seem confident they can rewrite some code to push the latency down to <20 microseconds x60-70 crossings per pass, might be able to hit 50t/s with MTP.
In relation to hipEngine but also in general, what do you think about ktransformers and the architecture they are proposing? maybe future en-gram model’s qwen4 and deepseek will align closer to splitting huge portions of the inference into CPU capable and GPU capable parts?
Hey been heads down on work, but in my spare time I’ve been continuing work on hipEngine - hipEngine v0.5.0 is now released and it’s a pretty big one.
The headline is now a very fast Qwen 3.8 27B implementation for Q4_K_M GGUF of anything out there (comparison). At c=1 pp512/tg128/mtp it’s 405 / 12.2 / 21.0 tok/s (competitively fast) but it also has the best concurrency scaling (c=1..8) that I’ve tested against:
Engine
C1
C2
C3
C4
C5
C6
C7
C8
llama.cpp mainline Vulkan
10.226
17.471
14.905
13.246
18.086
24.083
29.342
35.376
llama.cpp stock HIP
10.635
17.681
15.266
13.537
18.271
23.296
26.544
30.325
Laurent Vulkan
11.047
19.835
16.359
14.255
20.294
28.322
35.896
45.614
Nathan Vulkan
11.162
19.662
16.341
14.321
20.099
27.575
34.317
41.511
q38rocm Vulkan
10.582
18.645
15.663
13.866
19.054
24.970
29.576
35.021
hipEngine
11.518
19.249
24.526
31.478
37.995
43.093
46.153
50.605
BTW, there is also very good MTP support now for all the Qwen models. By default the engine intelligently switches on/off MTP based on concurrency (profiled speed) but you can also override and turn it on/off on a per request if you want:
Model
Quant
Prompt processing
Text generation
With MTP
Max context
Maple-Preview
2-bit
754.5
153.2
—
—
Qwen3.6-35B-A3B
GGUF UD-Q4_K_M
1369.5
54.3
80.1
—
Qwen3.8-27B Dense
GGUF Q4_K_S
396.1
13.1
23.9
—
Qwen3.8-27B Dense
GGUF Q4_K_M
404.5
12.2
21.0
—
I’ve started optimizing Qwen 3.8 Flash Next, but there are a lot of neat forks out now (with varying levels of correctness/accuracy), I think GitHub - halo-box/strix-llama.cpp: LLM inference in C/C++ · GitHub is the most promising (community fork, not worrying about being slavish to upstream llama.cpp), so for popular models, that may be best (they’re moving very quickly), but probably the more interesting things with hipEngine now from my most recent work:
I’ve overhauled the concurrency engine - it has true continuous batching, shared kvpool, and radix prefix caching - hipEngine has always been more of a a mini vLLM/SGLang than llama.cpp, and that continues
hipEngine was built with DMS support in mind, and that promise is now fulfilled. It now has a demonstrated 5.7x KV cache compression with 100% Top-1 and 0.001 KLD (much more accurate than even direct INT8 KV)
I’ve been adding a bunch of models I’m interested in. Moonshine realtime-ASR, TimesFM time series, EVIE late interaction visual document retrieval - these don’t have llama.cpp or (any non-torch inference support) at all. I’ll be adding more modalities and things that I think are cool now that the engine is working well. I’ve begun using bits and pieces for production workloads even.
There’s a lot of QoL stuff focused around running/agentic work - auto-calculation of memory pools when loading, logit-biasing and other reasoning adjustments, per-request MTP (and default MTP so it only applies when it gives a speedup), etc. Neat stuff I think an inference engine should do.
@_sk Engrams are very cool, Qwen 3.8 Flash Next is pretty great. The more o(1) math the better. I’m also enjoying DS4.1 Flash, but it’s too big for 2xStrix Halos atm. For fun right now I’m having ds41f tune itself: GitHub - shisa-ai/ds41f · GitHub
@Anders_Rudkjaer_Norg haven’t looked at HRX. I think for people looking to have fun/poke, go for it, but it’s too early to tell how far they’ll get. I don’t think it’s super relevant to my work. For low level perf I’ve been submitting what I’ve noticed into LLVM/HIP/ROCm repos and they are making their way through.
I’ve never tested it on Windows. You could give it a spin (but I’d guess it probably doesn’t work).
if you install now btw, try to do so from source HEAD. I’ve pushed a bunch of MTP/prefix cache fixes (was off by default in the v0.5.0 review, oops). v0.6.0 w/ a lot of new stuff will be out soon, but there’s also real-world improvements on agentic flows w/ improved prefix caching especially:
I pushed v0.6.1 out, which fixes some niggles with the engine itself (there a bunch of cases where MTP or prefix caching was being improperly gated, also sampling was not optimized, had to revamp that, should all be running at full speed now).
I’ve actually been working on a bunch of fun things like additional unique model support:
There is now full Surya OCR 2 support for SOTA document OCR (multilingual support, layout analysis, table recognition)
I also added Microsoft VibeVoice ASRand1.5 TTS and actually have no non-torch way of running, so should be useful. (they run in hipEngine faster than Torch of course).
Also, just added an internal benchmark to the chat so you can see how MTP and prefix caching affect performance:
