I am purchasing a Desktop with Max+ 395 - 64GB to run local AI. I will use voice to text (e.g. whisper), documents (RAG) and general AI prompts.
I have experience with Ubuntu and Arch (CatchyOS), also Ollama in the context of a laptop Intel CPU / NVIDIA GPU set up.
AMD / Ryzen is new to me, with the combined memory architecture.
I presume many have walked this journey already. May I ask from this experience:
The best Linux distro to use ? AI will be the primary purpose for this machine.
What changes to bios are recommended if any?
Are there any setup potential hurdles such as with device drivers?
I will be accessing the AI both directly (e.g. LMStudio) but also via browser on my local network and VPN. What is the best setup for network access? This is both for general AI prompt usage, but also RAG. For RAG I use Open Notebook currently.
After that starting point, it’s mostly up to exploration and tinkering. Local AI seems to be in the state that Linux was in 20-30 years ago where everyone is sharing around their own homebrew solutions and scripts. Which one works best for you depends on your own lived experience, i.e. “the best benchmark is your benchmark”. Have fun!
I like this advice. I am using Claude Code to develop a software project, and I don’t think I ever read the manual, or watched a video on how to construct prompts. I judge that my speed-up factor is in the order of 4x to 5x, and that’s enough for me.
I am thus sure that a writer could spend a lot of time polishing their writing desk, but that’s time they could spend writing .
FWIW I’ve had success having Codex (using 5.6 sol medium if that matters) tune llama.cpp to improve speed. I tuned spec-drafting myself on Qwen 3.6 35b a3b (I use max 3 draft tokens, 0.3 acceptance threshold) but I pretty much just let codex rewrite HIP kernels and test them and I came back to faster running LLMs.
Switching from Vulkan to HIP seemed to be around a significant prefill speedup but -30% on generation. Codex rewrote some of the kernels and now I’m benchmarking it at approx +40% prefill -10% generation. I’m using the AP-Q5_K_M GGUF file with ngram-map-k spec drafting, and it’s something like 25-40t/s generation. This speed is actually acceptable on smaller projects.
I have a Framework Desktop 128GB, 14TB SSD version that I purchased in Nov 2025. I am using it with a AMD 7900xtx 24gb VRAM via eGPU (Oculink). You didn’t ask this specific question- but I can share one thing I’ve experienced. When I run models (typically 27B class), you’d be surprised how much system RAM you need to run them well. I have the 128GB version and I have found that that I have to drop it down to 64gb VRAM/64GB RAM to avoid throttling. I’ve tried to do 96GB VRAM/32GB RAM, but it seems to eventually lock up when it is being used a lot. I sometimes connect a second eGPU AMD v620 32gb VRAM via USB4 (currently down because I fried the eGPU board and need to replace it) and I can run three or four models side by side, with one dedicated LLM per agent, which makes them run a lot faster. That does pretty well, but system RAM is usually the bottleneck.
Last week I tried applying the suggestions from Strix Halo AI Toolboxes (just the ones from the Kernel parameters and unified memory section) and that alone gave me a boost of 10-15t/s generation with Qwen 3.6 35b a3b.
I also had the system memory bottleneck @Ernie_Hobbs mentioned (with some models) but the kernel configuration change solved it for me.
Switching from LM Studio to Unsloth Studio and switching to the MTP version of Qwen 3.6 35b a3b also gave me quite a big boost in token generation, now it runs around 75-90t/s.
One of my problems is my own fault- I use Comet Browser, which I like, but uses a lot of system memory. So it is hard to use both AI local models, and also do normal things in the browser (it doesn’t help that I typically have five monitors running and usually at least six browsers (with many tabs open on each) at all times- Personal, Day Job, Startup Job accounts. When I am running heavy compute tasks, I have to start closing a bunch of stuff to make it all work. I’m also using Windows 11 Pro, which I know contributes to the problem. I tried Linux but it just didn’t work for me (I have a CNC design software that I use a lot that Linux drivers don’t exist). Sometimes I just have to take a break from the Framework and use my laptop for a while until the jobs finish. I think I may be pushing the limit of what my Framework can do and my next computer isn’t going to be a home system- I’m probably looking at a professional server class system (way out of budget for now).
.