I bought a Framework Desktop 128 back when they were affordable. As I think about my future computer purchases, I’m wondering if the Framework Desktop is a one-off, a line that will continue to expand like the laptop series, or something in between.
More specifically, I’m wondering if Framework has any interest in producing a 256g version based on Gorgon Halo. I’m only in the budgeting phase for my next computer, but with bigger and better AI models that won’t fit in 128g I’m wondering if I should be anticipating a possible Framework PC or not.
I’m unsure that the Framework Desktop really fits in the whole Framework ethos, with the soldered RAM, but I’m loving my machine, and will probably start saving up for an upgrade if Framework is planning on producing one.
My gut feeling is that soldered RAM isn’t going away unless there’s a crash in the LLM market.
High-bandwidth unified memory is super attractive for local LLM inference - especially now that the most recent open models have 10^11 or 10^12 parameters.
Many local LLM enthusiasts have a lot of disposable income (either directly or through their employers), which makes them attractive to manufacturers.
It’s just a lot cheaper and easier to solder high-bandwidth memory than to use some LPCAMM variant.
For the purposes of running LLMs, the non-upgradeability is fine, because this machine is primarily bottlenecked by bandwidth, and a hypothetical upgrade would not change that.
I bought a Strix Halo because Gorgon Halo is only a small incremental improvement, and the power efficiency of the new GAAFET nodes doesn’t really matter in a desktop. In hindsight if I found a 96GB for an abnormally low price that would’ve been fine. I have about 60t/s generation on Qwen 35b a3b q6kl, which is pretty borderline as a coding assistant. Could hypothetically try Nemotron 3 super but I suspect it would be almost intolerable.
I think if you’re looking at more than 192gb of unified memory, it should probably be paired with a memory bus more comparable to Mac Studio Ultra. IMHO there’s going to be a capex slowdown of some sort but memory will continue to be a large share of it so prices probably won’t come down too much.
I completely agree. Socketing costs a significant percentage of the memory speed (30-50%), and burns more power. Given the choice between upgradeable and slower, or non-upgradeable and faster, personally I’d like the latter. For inference machines, I think soldering is here to stay. Thanks physics.
I don’t even use my Desktop’s full 128GB of memory day-to-day. Sure, modelmaxxing is fun, but pushing giant models over this memory interface gets very sloooow. Until AMD significantly increases the memory bandwith, 128GB feels pretty far past the maximum useful limit to me (unless you don’t care about speed at all, or someone figures out a more effective way to serve the model).
I can max out the memory if I use qwen 27b dense, since the KV cache is a little more hungry. I think in the future models will converge on more linear attention usage though, that’s what the research suggests IMO. The next top 3-7b activated param model could have even less kv cache footprint than qwen 35b a3b.
I’ll hit maybe 90gb usage or something if I prompt codex to use 16 workers for python multiprocessing some data + have qwen 35b a3b in the background + have parakeet stt in the background, but really the latter 2 are just wasteful usage.
In the lower memory configs, I guess this a decent budget mobile gaming platform, but tbh for productivity it probably makes more sense to get an older single socket EPYC with DDR4 and run cloud LLMs. It’s cool to have a backup option with local LLMs but the speed difference is vast.