Local llm on FW16

Community Support
,
1

Hi

I’m trying local llm capabilities of my FW16 laptop.

           .-------------------------:                    martin@cachyos-fw16
          .+=========================.                    -------------------
         :++===++==================-       :++-           OS: CachyOS x86_64
        :*++====+++++=============-        .==:           Host: Laptop 16 (AMD Ryzen 7040 Series) (AG)
       -*+++=====+***++==========:                        Kernel: Linux 7.0.11-1-cachyos
      =*++++========------------:                         Uptime: 6 mins
     =*+++++=====-                     ...                Packages: 1502 (pacman)
   .+*+++++=-===:                    .=+++=:              Shell: fish 4.7.1
  :++++=====-==:                     -*****+              Display (BOE0BC9): 2560x1600 @ 1.15x in 16", 60 Hz [Built-in]
 :++========-=.                      .=+**+.              DE: KDE Plasma 6.6.5
.+==========-.                          .                 WM: KWin (Wayland)
 :+++++++====-                                .--==-.     WM Theme: Breeze
  :++==========.                             :+++++++:    Theme: Breeze (Dark) [Qt], Breeze-Dark [GTK2], Breeze [GTK3]
   .-===========.                            =*****+*+    Icons: breeze-dark [Qt], breeze-dark [GTK2/3/4]
    .-===========:                           .+*****+:    Font: Noto Sans (10pt) [Qt], Noto Sans (10pt) [GTK2/3/4]
      -=======++++:::::::::::::::::::::::::-:  .---:      Cursor: breeze (24px)
       :======++++====+++******************=.             Terminal: konsole 26.4.2
        :=====+++==========++++++++++++++*-               CPU: AMD Ryzen 7 7840HS (16) @ 5.14 GHz
         .====++==============++++++++++*-                GPU: AMD Radeon 780M Graphics [Integrated]
          .===+==================+++++++:                 Memory: 3.67 GiB / 27.21 GiB (13%)
           .-=======================+++:                  Swap: 521.86 MiB / 59.21 GiB (1%)
             ..........................                   Disk (/): 679.72 GiB / 898.51 GiB (76%) - btrfs
                                                          Local IP (eth0): 192.168.2.27/24
                                                          Battery (FRANDBA): 80% [AC Connected]
                                                          Locale: cs_CZ.UTF-8

i installed lm studio. I’m trying to load Qwen3 Coder 30B A3B Instruct Q4_K_M. Because memory is shared it should be able to run the whole thing on GPU. But it fails on

2026-06-07 18:11:27 [DEBUG]
 LlamaV4::load called with model path: /home/martin/.lmstudio/models/lmstudio-community/Qwen3-Coder-30B-A3B-Instruct-GGUF/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf
LlamaV4::load config: n_parallel=4 n_ctx=4096 kv_unified=true
2026-06-07 18:11:27 [DEBUG]
 0.00.045.076 I srv    load_model: loading model '/home/martin/.lmstudio/models/lmstudio-community/Qwen3-Coder-30B-A3B-Instruct-GGUF/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf'
2026-06-07 18:11:27 [DEBUG]
 0.00.152.483 W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
2026-06-07 18:11:33 [DEBUG]
 0.06.535.421 W warning: failed to mlock 436264960-byte buffer (after previously locking 0 bytes): Cannot allocate memory
Try increasing RLIMIT_MEMLOCK ('ulimit -l' as root).
2026-06-07 18:12:28 [DEBUG]
 radv/amdgpu: Not enough memory for command submission.
2026-06-07 18:12:28 [DEBUG]
 1.01.496.792 E llama_model_load: error loading model: vk::Queue::submit: ErrorDeviceLost
1.01.496.804 E llama_model_load_from_file_impl: failed to load model
1.01.496.841 E common_init_from_params: failed to load model '/home/martin/.lmstudio/models/lmstudio-community/Qwen3-Coder-30B-A3B-Instruct-GGUF/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf'
1.01.496.858 E srv    load_model: failed to load model, '/home/martin/.lmstudio/models/lmstudio-community/Qwen3-Coder-30B-A3B-Instruct-GGUF/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf': error loading model: vk::Queue::submit: ErrorDeviceLost
2026-06-07 18:12:28 [DEBUG]
 [LLMProcess] Failed to load model _0x41fe3f [Error]: Failed to load model.
    at _0x446b3e.loadModel (/opt/LM-Studio/resources/app/.webpack/lib/llmworker.js:1:613074)
    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
    at async _0x446b3e.handleMessage (/opt/LM-Studio/resources/app/.webpack/lib/llmworker.js:1:605001) {
  cause: 'Failed to load model',
  suggestion: undefined,
  errorData: undefined,
  data: undefined,
  displayData: undefined,
  title: 'Failed to load model.'
}

failed to mlock 436264960-byte buffer that is 436MB. i first thought that it needs that much dedicated memory so i switched in bios from auto (512MB) to gaming (3GB?) but i have the same error.
ulimit -l reports 8192, but I’m not sure it is the problem, because I’m able to load this model on my PC with 7900 XT (20GB).

Is there anybody who also experimented with local llms on framework laptops an can explain to me this issue and limitations of my hw?

2

Hi

afte r some research and testing i found out that the issue was that by defaultm amdgpu has GTT of 16GB that is limit how much RAM can GPU allocate. this particular model was too large for that. Increasing it by adding amdgpu.gttsize=24576 (24GB) solved the issue.

But i have another issue. these bigger models takes extremely long to load and run extremely slowly. i don’t see any reason why. i tried swappong gpu allocation from auto to gaming and having 4GB dedicated to GPU. I’m continuing my tests.

3

The thing that jumps out first: that “slow even after it’s loaded” part probably isn’t a bug you can chase down — it’s the memory bandwidth ceiling. Decode on the 780M is bound by how fast it can stream the active weights out of memory, and the 7840HS is dual-channel DDR5-5600, so you’re looking at roughly ~90 GB/s at the very top. That number is basically your speed limit for token generation, and no config flag really moves it.

What makes it feel mysterious is the MoE side of it. A3B means only ~3B params are active per token, so intuitively it should feel light, and compute-wise it *is*, the 780M is barely sweating. But that’s the trap: the bottleneck was never compute, it’s bandwidth plus memory pressure. So you get this “the GPU looks idle but it’s crawling” thing, which is exactly what you’d expect on this hardware, not a sign something’s set up wrong.

The “loads slow” part I’d actually treat as a separate issue from the “runs slow” part. Q4_K_M on a 30B is ~18GB of weights, and you’ve got ~27GB usable. So a load is ~18GB coming off disk, getting staged into GTT, and that pushes you fairly close to the edge of your memory, so if any of it spills to swap during load, that alone would explain a painful load time. Worth remembering GTT isn’t pinned VRAM, it’s pageable system memory, so it’s all sharing that same ~27GB pool with everything else.

So given all that, the stuff actually worth doing is the unglamorous stuff. A smaller quant helps on both fronts at once: fewer bytes to stream means faster decode, and less sitting in memory means an easier load. Dropping the context length is the other free win, since it shrinks the KV cache and hands you back some of that ~27GB headroom. The one thing I’d double-check rather than treat as a speed knob is that all the layers are genuinely getting offloaded to the GPU; if a couple are quietly sitting on the CPU, you’ll get slowness that looks like the same ceiling but isn’t, and that’s worth ruling out.

On the backend itself, I’d stay on Vulkan. For gfx1103 it’s generally the faster and more stable option, and it’s what the llama.cpp folks lean toward for this hardware anyway, so you’re already on the right track. That ErrorDeviceLost was a load-time thing you’ve already handled with the gttsize tweak, so I wouldn’t read it as a reason to jump to ROCm. I’d actually steer clear of the HSA_OVERRIDE flag that gets passed around for these APUs, since on this chip it can hard-lock the whole machine. Usual caveat: I’m on gfx1151, not a 780M, so take the specifics as direction more than exact numbers to copy.