Hello, we are happy to share a project that a friend of mine and I have been working on for a while. It’s an extremely optimized inference engine for the Strix Halo platform. We both have a Framework Desktop with 128 GB of RAM and were tired of juggling multiple forks of llama.cpp and other projects trying to optimize model X or model Y. We wanted something vertically integrated for the hardware we own, extremely optimized, and an all-in-one solution including LLMs, audio, video, image, etc.
The tool is called Gufo, it’s fully open source under the MIT license, and we welcome any contributions. I’ll summarize our philosophy in these few points:
Contributions are welcome! We need the help of the Strix Halo community to keep improving Gufo!
We would like this to be the one-stop shop for Strix Halo Local AI enthusiasts: batteries included for text, audio, image, and video models.
Built and optimized specifically for Strix Halo 128 GiB hardware. Smaller memory configurations should still work and preserve the speed benefits for models that can fit in memory.
Support only the best available models for their size that can run on this hardware: less code to maintain, and more focused optimization and testing.
Preserve quality when optimizing. Each model’s quality report records independent numerical checks, execution consistency, and unresolved gaps. We don’t reuse kernels across different models to limit the blast radius of code changes.
Treat concurrent requests, cancellation, and conversation caching as first-class workloads.
Keep production dependencies minimal and development tools separate.
We also provide ready to use toolboxes for docker and podman.
And now the most interesting part: some benchmarks (limited to keep the post short). We currently support Qwen 3.8 27B, DeepSeek V4 Flash, Qwen 3.8 Flash-Next, Qwen3 ASR and TTS, Qwen Image 2.1, and MiniMax H3.
Model
Benchmark summary
Qwen 3.8 27B
Q4: 656.33 tok/s pp; up to 70.56 tok/s tg, 123 tok/s concurrency 8
Qwen 3.8 Flash-Next
1,628.52 tok/s pp; up to 59.41 tok/s tg, 157 tok/s concurrency 8
DeepSeek V4 Flash
484.62 tok/s pp; up to 26.62 tok/s tg, 54 tok/s concurrency 8
Only the gpu for now, we have a dedicated branch in which we experimented with the NPU in the following ways:
drafter (mtp, dflash, dspark, etc.) on the NPU
second smaller model on the NPU
audio model for a audio → text → audio pipeline
With none of them we managed to obtain better results than using only the GPU. There is a new driver the AMD team is working on that should improve the situation, we have another branch for that but no luck so far, not enough mature for now. We hope to get in touch with them for trying having some internal help from them
Oh wow 1600 pp2048 on qw3.8 flash udq4kxl is very high, I couldn’t get anywhere near that with llama.cpp, including with a V100 assisting (I guess this mainly helps decode since it has fast memory but old architecture slow core). Was using a few coding agents to optimize llama.cpp and they got me pretty large improvements over the starting point but still a long way from your result (I don’t know much about GPUs, wasn’t able to give them much useful guidance).
For your 59t/s generation claim though, did you make a mistake? The benchmarks page shows 26 at 0 depth dropping down to 22. That’s what the graph you provided says too. That’s about what I got with Vulkan llama.cpp out of the box, which is a pretty high % of the ceiling imposed by memory bandwidth.
Hey, no the decode is right but as an extreme case with a repetitive corpus. You can find the prompt and answer in the repo. We asked to repeat one word X times to make the drafter acceptance rate goes to 100%. This gives us the theoretical ceiling for the model throughput
Wow I came from Reddit and Redditors are loving Gufo! I’m running a project that looks to find the absolute best configuration for agentic coding sessions (several hours) for different hardware setups, and I’m using a fairly basic configuration on my Strix Halo (Qwen3.8 Flash-Next, llama.cpp MTP).
I’m exploring whether gufo might be better for my workflow and needs. First contrast that is interesting:
The gufo benchmarks use short prompts, greedy decoding and thinking off,
the coding agent setup runs at 70–130k context, with thinking on and lots of tool calls.
My proposal is an A/B test for my situation specifically, making sure to respect your playbook and config so we have apples to apples. I’m mindful also though that there seem to be a bunch of issues raised that could well affect this test.
I’ll compare the two on the same machine, with the same weights and gufo set up per its docs: speed at depth, tool-call correctness over a long session, and real app-building tasks scored by held-out tests. Anything that stands out as unfair for gufo with this plan? I’m also thinking it might also be fairer to wait until certain bugs (e.g tool calling, and issues when thinking is on) are fixed. No rush on my side.
Thanks in advance for your guidance – and of course for your amazing work!
I have been using agentionai AP Q5_K_M (XL has the same actual neural net part), ran it with gufo engine, <500t/s prefill out of the box. With these changes (done with codex), measured 1063t/s on the same 2048 token prompt (my GPU is running at 2960MHz):
Added Q6_K to the packed HIP MMQ prefill path used by AP-Q5’s dense trunk. The model now keeps packed Q6_K weights available and dispatches them to the quantized kernel for large prefill batches
IQ4_NL expert-down decoded inside the routed F16 matrix-core kernel, so the compacted Q5_K gate/up path runs on AP-Q5_K_M
Did a similar build in llama.cpp that’s 807t/s, and using my V100 to support a few operations I’m at right over 1000t/s. I’m having codex rewrite a bit more of llama.cpp, hopefully it can get closer. I think I will buy a MI100 and see if I can get a fork of gufo engine working with layer split, theoretically it should be 3x faster than Strix Halo alone.
Hey @Federico_Izzo, this looks awesome. I haven’t tested it yet on my 128 GB FD, but the performance from your benchmarks seem way better than what I’m currently getting with a self-quantized Qwen 3.8 27B @ Q4_K_M. I’ll need to play with your project a bit in the future and see what happens. This will be ran on CachyOS on top of OpenZFS.
Thanks, we are working hard to improve the stability right now. the tool calling fails sometimes and the caching mechanism can be better. If you notice something off, please open an issue, we would like to build gufo with the community
This is really cool. I had claude spend most of the weekend taking over my mainboard and building out support for Ling-3.0-Flash, and this is such an absurd level of improvement over llama.cpp, I almost couldn’t believe the results I was getting.
I’d be interested in contributing this upstream once y’all are open to more models if you are interested in adding this specific model. I know everyone loves qwen models right now, but I like a diversity of sources for my code reviewing agents (although I actually like Ling’s agentic coding capability too), and this special quantization of Ling gives a good space trade-off compared to larger models at lower quantization.
