Step-by-step guide: Ubuntu 24.04 + ROCm 7.2.1 + llama.cpp on Framework Desktop

Framework Desktop
,
1

Hello folks,

If you’ve been wanting to run LLMs locally on your Framework Desktop but weren’t sure where to start — I put together a walkthrough that covers the full setup from scratch.

The guide walks through:

  • UEFI settings to get the iGPU memory working correctly
  • Installing the right kernel and firmware versions (there are a few version-specific pitfalls to avoid)
  • ROCm 7.2.1 driver setup
  • Building and running a llama.cpp server

It also works on any machine with an AMD Ryzen AI Max series APU, not just the Framework Desktop.

A Traditional Chinese version is also available in the same repo.

Happy to answer questions or take feedback here.

1 Like
2

You can use lemonade for a prebuilt solution for LLMs (it wraps llama.cpp for example).

3

I know AMD GAIA and lemonade, but the whole point of this guide is to install ROCm driver/runtime and choose stable kernel and firmware. git pull and build llama.cpp from source is easy, also it gives you latest ROCm runtime, bugfix/improvements and best performance.