I just received my Framework 13 Pro with the X7 358H with 32gb Crucial LP-CAMM2 7500Mhz in Omarchy 4. I just tried running a simple prompt with a lm-studio and gemma-4-12b-qat (a 7gb model). I was a bit surprised that it ran at about 9.7tps, whereas my 4 year old m2 macbook air ran the same prompt at 41.35tps.
I was wondering if I’m doing something wrong (or perhaps my memory is not up to par) or if others are noticing similar performance?
lm-studio shows that it detects the GPU, and when running the prompt it shows about 96-97% utilization with gputop.
I can’t speak for the power delivery of the framework pro, but I have a clevo barebones same processor, soldered lpcamm and I noticed the same thing. After much investigation, it came down to lower power delivery for laptops. I am still very impressed with the panther lake 358h and bought a Asrock nuc box that has the same 358h, but DDR5 (I had an extra 96G) and it has much better power delivery, and performs where the chip is capable. I get 11 t/s with GPT-oss-120b. and other better speed with other moe type arch LLM. For dense models, even small ones, it’s still slower, on gemma4-26b-a4b-Q4_K_M I get 17.4 t/s. I don’t have gemma4-12b on it yet, but I’m downloading it if you want to know what it gets. on my clevo barebones I got 7.2 t/s for gemma4 12b. the differences in memory bandwith are somewhat responsible, but the asrock nuc does show me what the system is capable of. This should help some:
What I heard is that you want to use SYCL and oneAPI and not Vulkan with the Intel Arc graphics for better performance. But I don’t know how you can set this up in lm-studio, I use lemonade and use ROCm with my AMD system.
I recommend trying to use Intel Performance Counter Monitor (PCM) to check what’s the actual memory bandwidth being used during inference. If the current RAM read speed is nowhere near the theoretical limit then there is probably a problem there, somehow. I dunno what you’d do after finding the value regardless of if it’s within the expected range or not, but it’s just an idea I wanted to throw out. Here’s a screenshot of what it looks like on my (currently idle) i5-1334U Framework 12. At the bottom left you can see RAM r/w speeds.
Thanks for your benchmarks @Joe_Name I’d love to hear what you’re getting with the gemma4 12b model.
I eventually realized the MacBook Air was getting such drastically better results because lm-studio link had automatically routed it to a preferred device, my Mac Studio M1 Ultra. After finally realizing this, the MacBook Air itself actually gets about 10.8tps.
I tried setting things up with SYCL as mentioned by @gentux and see an improvement to about 11.57tps, so actually about 13% improvement from Vulkan.
While that definitely brings things a bit more in-line/within reason, it’s a tiny bit surprising/disappointing that this brand new machine with far higher sustained wattage is essentially on par with a 4 year old mac, especially since I bought the Macbook for around 70% of the cost, but perhaps this is just a limitation of memory bandwidth as a whole and is otherwise a wonderful machine.
I think that’s just kind of the nature of the beast with llm workloads specifically; they are so demanding of memory bandwidth that more wattage / compute often doesn’t help unless you can feed it more bandwidth. Although that should be specific to the streaming response / tokens per second part of the workload. Where you might look for bigger performance uplift is prompt processing / time to first token as in general that tends to shift away from bandwidth constraint and more towards compute.
