LLM Benchmark (AMD 7840u)

Framework Laptop 13 Linux
1

For anyone interested, this is my LLM benchmark on my Framework 13 equipped with the AMD 7840u mainboard (32gb RAM), running KDE Neon 6 (based on Ubuntu 22.04 LTS), kernel is Linux fw 6.5.0-1020-oem, cpu-governor on balanced. (laptop plugged on AC)


-------Linux----------

No GPU detected.
{
    "mistral:7b": "12.12",
    "gemma:2b": "26.39",
    "gemma:7b": "9.75",
    "llama2:7b": "12.01",
    "llama2:13b": "6.82",
    "llava:7b": "12.67",
    "llava:13b": "7.02",
    "uuid": "c830e285-c9af-5bcf-b477-85e89d03a7bf",
    "ollama_version": "0.1.32"
}
----------
====================
-------Linux----------

No GPU detected.
-------Linux----------

No GPU detected.
{
    "system": "Linux",
    "memory": 30.65096664428711,
    "cpu": "AMD Ryzen 7 7840U w/ Radeon  780M Graphics",
    "gpu": "Advanced Micro Devices, Inc. [AMD/ATI]",
    "os_version": "KDE neon 6.0",
    "system_name": "Linux",
    "uuid": "c830e285-c9af-5bcf-b477-85e89d03a7bf"
}

benchmark tool used : https://github.com/aidatatools/ollama-benchmark

1 Like
2

Interesting benchmark. Here are my results with same laptop!

-------Linux----------

No GPU detected.
{
    "mistral:7b": "12.41",
    "gemma:2b": "28.17",
    "gemma:7b": "9.97",
    "llama2:7b": "12.54",
    "llama2:13b": "6.87",
    "llava:7b": "12.85",
    "llava:13b": "7.03",
    "uuid": "dff15406-d2b5-5c70-bed1-xxxxxxxxxxxx",
    "ollama_version": "0.1.32"
}
----------
====================
-------Linux----------

No GPU detected.
-------Linux----------

No GPU detected.
{
    "system": "Linux",
    "memory": 58.662559509277344,
    "cpu": "AMD Ryzen 7 7840U w/ Radeon  780M Graphics",
    "gpu": "no_gpu",
    "os_version": "Fedora Linux 39 (Sway)",
    "system_name": "Linux",
    "uuid": "dff15406-d2b5-5c70-bed1-xxxxxxxxxxxx"
}
1 Like
3

Anyone with 16 + dGPU?

4

Here’s the results with my 13" 1240p. Running nixos with 32GB of RAM.

{
    "mistral:7b": "5.53",
    "llama3:8b": "4.98",
    "phi3:3.8b": "8.17",
    "gemma:2b": "10.44",
    "gemma:7b": "4.16",
    "llava:7b": "5.29",
    "llava:13b": "3.24",
    "uuid": "d340a0a4-05cf-59b8-8bef-b37c560083d5",
    "ollama_version": "0.1.38"
}

And the system details:

{
    "system": "Linux",
    "memory": 31.06365203857422,
    "cpu": "12th Gen Intel(R) Core(TM) i5-1240P",
    "gpu": "unknown",
    "os_version": "\"NixOS 24.05 (Uakari)\"",
    "system_name": "Linux",
    "uuid": "d340a0a4-05cf-59b8-8bef-b37c560083d5"
}
5

I know this is in the 13 forum, but here you go, Framework 16. This was interesting:

-------Linux----------

No GPU detected.
{
    "mistral:7b": "41.25",
    "llama3:8b": "39.19",
    "phi3:3.8b": "51.70",
    "gemma:2b": "92.49",
    "gemma:7b": "36.76",
    "llava:7b": "43.84",
    "llava:13b": "4.35",
    "uuid": "bccd6f86-8b2d-5410-92fc-7120dbdd81f9",
    "ollama_version": "0.1.38"
}
----------
====================
-------Linux----------

No GPU detected.
-------Linux----------

No GPU detected.
{
    "system": "Linux",
    "memory": 58.668739318847656,
    "cpu": "AMD Ryzen 9 7940HS w/ Radeon 780M Graphics",
    "gpu": "Phoenix1",
    "os_version": "Arch Linux",
    "system_name": "Linux",
    "uuid": "bccd6f86-8b2d-5410-92fc-7120dbdd81f9"
}
----------
2 Likes
6

I haven’t noticed this is the 13 forum. In any case, thank you for the benchmark

7

Here’s my performance on Arch Linux, 13" AMD:

-------Linux----------
No GPU detected.
Total memory size : 30.66 GB
cpu_info: AMD Ryzen 7 7840U w/ Radeon  780M Graphics
gpu_info: no_gpu
os_version: "Arch Linux"
ollama_version: 0.1.41

Your machine UUID : a79f8302-d19d-5b65-a214-213535d44a16
-------Linux----------

No GPU detected.
{
    "mistral:7b": "6.09",
    "llama3:8b": "5.64",
    "phi3:3.8b": "9.54",
    "gemma:2b": "14.14",
    "gemma:7b": "4.85",
    "llava:7b": "6.21",
    "llava:13b": "3.43",
    "uuid": "a79f8302-d19d-5b65-a214-213535d44a16",
    "ollama_version": "0.1.41"
}

Any tips on getting some more performance out of it? I’m using the open source amdgpu drivers and have tried both ollama’s own install and the pacman/Arch-packaged version.

8

You’re number are strange, I also used the open source amdgpu drivers and ollama’s own install, maybe check your kernel version ?

9

According to this post, setting a CLI flag on the ollama server let’s rocm use the iGPU Enable AMD iGPU 780M in Linux, Create amd-igpu-780m.md by alexhegit · Pull Request #5426 · ollama/ollama · GitHub

One thing that would be really interesting is if framework updated the BIOS to let us set the GPU reserved RAM. It would be really interesting to set the GPU VRAM to 8GiB. Right now it’s capped at 4GiB. I have a laptop with 96GiB of RAM, and I would be happy to give it 16GiB so I could load more complex models.

I have a 6900HX mini pc that I gave 8GiB to the GPU and it works just fine.

Even if it’s somewhat slow, that would be an excellent use of large amounts of RAM for offline line and local LLM development.

2 Likes
10

Hey everyone, I decided to take this out for a spin with my new-to-me Ryzen Framework and see how it performs.

Lengthy details on how I got it working over on my “unboxing” blog post on my website, but TL;DR there’s an open PR to ollama that lets you unlock the full RAM of the system, not just allocated VRAM. Here are the results. Sadly it looks like the benchmark y’all were using completely changed out their models, so I can’t really compare to the others in this thread.

CPU inference only looks like this:

{
    "mistral:7b": "11.99",
    "llama3.1:8b": "10.81",
    "phi3:3.8b": "19.74",
    "qwen2:7b": "12.07",
    "gemma2:9b": "8.62",
    "llava:7b": "12.71",
    "llava:13b": "6.93",
    "uuid": "0f96d8fc-0390-5f84-bd51-3595976f0b2d",
    "ollama_version": "0.0.0"
}
{
    "system": "Linux",
    "memory": 27.205055236816406,
    "cpu": "AMD Ryzen 7 7840U w/ Radeon  780M Graphics",
    "gpu": "Phoenix1",
    "os_version": "Bazzite 41 (FROM Fedora Kinoite)",
    "system_name": "Linux",
    "uuid": "0f96d8fc-0390-5f84-bd51-3595976f0b2d"
}

While GPU acceleration with the experimental VRAM PR looks like this:

{
    "mistral:7b": "17.15",
    "llama3.1:8b": "10.84",
    "phi3:3.8b": "24.39",
    "qwen2:7b": "12.12",
    "gemma2:9b": "11.62",
    "llava:7b": "17.40",
    "llava:13b": "9.62",
    "uuid": "0f96d8fc-0390-5f84-bd51-3595976f0b2d",
    "ollama_version": "0.0.0"
}
{
    "system": "Linux",
    "memory": 27.205055236816406,
    "cpu": "AMD Ryzen 7 7840U w/ Radeon  780M Graphics",
    "gpu": "Phoenix1",
    "os_version": "Bazzite 41 (FROM Fedora Kinoite)",
    "system_name": "Linux",
    "uuid": "0f96d8fc-0390-5f84-bd51-3595976f0b2d"
}

Sadly, since I made that post, I’ve been running into some trouble; it looks like the system just fails to run properly if it’s under a fair bit of memory pressure. I can’t run a 14B model when using 14GB of RAM already, it just never finishes loading. But here’s hoping it improves and gets merged soon!

1 Like