I am purchasing a Desktop with Max+ 395 - 64GB to run local AI. I will use voice to text (e.g. whisper), documents (RAG) and general AI prompts.
I have experience with Ubuntu and Arch (CatchyOS), also Ollama in the context of a laptop Intel CPU / NVIDIA GPU set up.
AMD / Ryzen is new to me, with the combined memory architecture.
I presume many have walked this journey already. May I ask from this experience:
The best Linux distro to use ? AI will be the primary purpose for this machine.
What changes to bios are recommended if any?
Are there any setup potential hurdles such as with device drivers?
I will be accessing the AI both directly (e.g. LMStudio) but also via browser on my local network and VPN. What is the best setup for network access? This is both for general AI prompt usage, but also RAG. For RAG I use Open Notebook currently.
After that starting point, it’s mostly up to exploration and tinkering. Local AI seems to be in the state that Linux was in 20-30 years ago where everyone is sharing around their own homebrew solutions and scripts. Which one works best for you depends on your own lived experience, i.e. “the best benchmark is your benchmark”. Have fun!
I like this advice. I am using Claude Code to develop a software project, and I don’t think I ever read the manual, or watched a video on how to construct prompts. I judge that my speed-up factor is in the order of 4x to 5x, and that’s enough for me.
I am thus sure that a writer could spend a lot of time polishing their writing desk, but that’s time they could spend writing .
FWIW I’ve had success having Codex (using 5.6 sol medium if that matters) tune llama.cpp to improve speed. I tuned spec-drafting myself on Qwen 3.6 35b a3b (I use max 3 draft tokens, 0.3 acceptance threshold) but I pretty much just let codex rewrite HIP kernels and test them and I came back to faster running LLMs.
Switching from Vulkan to HIP seemed to be around a significant prefill speedup but -30% on generation. Codex rewrote some of the kernels and now I’m benchmarking it at approx +40% prefill -10% generation. I’m using the AP-Q5_K_M GGUF file with ngram-map-k spec drafting, and it’s something like 25-40t/s generation. This speed is actually acceptable on smaller projects.
I have a Framework Desktop 128GB, 14TB SSD version that I purchased in Nov 2025. I am using it with a AMD 7900xtx 24gb VRAM via eGPU (Oculink). You didn’t ask this specific question- but I can share one thing I’ve experienced. When I run models (typically 27B class), you’d be surprised how much system RAM you need to run them well. I have the 128GB version and I have found that that I have to drop it down to 64gb VRAM/64GB RAM to avoid throttling. I’ve tried to do 96GB VRAM/32GB RAM, but it seems to eventually lock up when it is being used a lot. I sometimes connect a second eGPU AMD v620 32gb VRAM via USB4 (currently down because I fried the eGPU board and need to replace it) and I can run three or four models side by side, with one dedicated LLM per agent, which makes them run a lot faster. That does pretty well, but system RAM is usually the bottleneck.
Last week I tried applying the suggestions from Strix Halo AI Toolboxes (just the ones from the Kernel parameters and unified memory section) and that alone gave me a boost of 10-15t/s generation with Qwen 3.6 35b a3b.
I also had the system memory bottleneck @Ernie_Hobbs mentioned (with some models) but the kernel configuration change solved it for me.
Switching from LM Studio to Unsloth Studio and switching to the MTP version of Qwen 3.6 35b a3b also gave me quite a big boost in token generation, now it runs around 75-90t/s.
One of my problems is my own fault- I use Comet Browser, which I like, but uses a lot of system memory. So it is hard to use both AI local models, and also do normal things in the browser (it doesn’t help that I typically have five monitors running and usually at least six browsers (with many tabs open on each) at all times- Personal, Day Job, Startup Job accounts. When I am running heavy compute tasks, I have to start closing a bunch of stuff to make it all work. I’m also using Windows 11 Pro, which I know contributes to the problem. I tried Linux but it just didn’t work for me (I have a CNC design software that I use a lot that Linux drivers don’t exist). Sometimes I just have to take a break from the Framework and use my laptop for a while until the jobs finish. I think I may be pushing the limit of what my Framework can do and my next computer isn’t going to be a home system- I’m probably looking at a professional server class system (way out of budget for now).
Update with the V100, used a lot of coding agent sessions to rewrite llama.cpp to coordinate across the two different vendors, basically this is the split routing:
CUDA: attention / GDN / norms / shared experts
│ activation
├──────────────► HIP: routed MoE experts
│
└──────────────► CUDA: shared experts
│
HIP MoE output ───────────────────┴──► CUDA join → next layer
Some findings:
Need to prevent the GDN state from being passed, that’s 3MB on the PCIe bus, even PCIe 4.0 x16 is going to be slow.
V100 is a very old architecture with big efficiency disadvantage for this workload despite the high memory bandwidth, needed to write a lot of CUDA to trim a few ms per token.
In particular, IQ4NL ops are extremely bad on the V100, I switched from agentionai AP Q5_K_M to the much bigger unsloth UD Q5_K_XL where the IQ4NL layer is Q8_0.
Strix halo also needs some kernel work, you can see what lhl/shisa-ai did in the other thread.
It might have been better to buy a W7800 or 7900XT or MI100 to get PCIe 4.0 + better hardware acceleration + same vendor. I think it would probably cut a few more ms per token and use a little less electricity.
Multiple V100s on SXM server board is still a great choice if you need to train at 8/16/32bit as far as I can tell, but for local inference it’s a bit of headache. I might swap it for one of the above mentioned at some point, since reselling this card should be pretty easy.
.