LLM Benchmark (AMD 7840u)

Framework Laptop 13 Linux
1

For anyone interested, this is my LLM benchmark on my Framework 13 equipped with the AMD 7840u mainboard (32gb RAM), running KDE Neon 6 (based on Ubuntu 22.04 LTS), kernel is Linux fw 6.5.0-1020-oem, cpu-governor on balanced. (laptop plugged on AC)


-------Linux----------

No GPU detected.
{
    "mistral:7b": "12.12",
    "gemma:2b": "26.39",
    "gemma:7b": "9.75",
    "llama2:7b": "12.01",
    "llama2:13b": "6.82",
    "llava:7b": "12.67",
    "llava:13b": "7.02",
    "uuid": "c830e285-c9af-5bcf-b477-85e89d03a7bf",
    "ollama_version": "0.1.32"
}
----------
====================
-------Linux----------

No GPU detected.
-------Linux----------

No GPU detected.
{
    "system": "Linux",
    "memory": 30.65096664428711,
    "cpu": "AMD Ryzen 7 7840U w/ Radeon  780M Graphics",
    "gpu": "Advanced Micro Devices, Inc. [AMD/ATI]",
    "os_version": "KDE neon 6.0",
    "system_name": "Linux",
    "uuid": "c830e285-c9af-5bcf-b477-85e89d03a7bf"
}

benchmark tool used : https://github.com/aidatatools/ollama-benchmark

2

Interesting benchmark. Here are my results with same laptop!

-------Linux----------

No GPU detected.
{
    "mistral:7b": "12.41",
    "gemma:2b": "28.17",
    "gemma:7b": "9.97",
    "llama2:7b": "12.54",
    "llama2:13b": "6.87",
    "llava:7b": "12.85",
    "llava:13b": "7.03",
    "uuid": "dff15406-d2b5-5c70-bed1-xxxxxxxxxxxx",
    "ollama_version": "0.1.32"
}
----------
====================
-------Linux----------

No GPU detected.
-------Linux----------

No GPU detected.
{
    "system": "Linux",
    "memory": 58.662559509277344,
    "cpu": "AMD Ryzen 7 7840U w/ Radeon  780M Graphics",
    "gpu": "no_gpu",
    "os_version": "Fedora Linux 39 (Sway)",
    "system_name": "Linux",
    "uuid": "dff15406-d2b5-5c70-bed1-xxxxxxxxxxxx"
}
1 Like
3

Anyone with 16 + dGPU?

4

Here’s the results with my 13" 1240p. Running nixos with 32GB of RAM.

{
    "mistral:7b": "5.53",
    "llama3:8b": "4.98",
    "phi3:3.8b": "8.17",
    "gemma:2b": "10.44",
    "gemma:7b": "4.16",
    "llava:7b": "5.29",
    "llava:13b": "3.24",
    "uuid": "d340a0a4-05cf-59b8-8bef-b37c560083d5",
    "ollama_version": "0.1.38"
}

And the system details:

{
    "system": "Linux",
    "memory": 31.06365203857422,
    "cpu": "12th Gen Intel(R) Core(TM) i5-1240P",
    "gpu": "unknown",
    "os_version": "\"NixOS 24.05 (Uakari)\"",
    "system_name": "Linux",
    "uuid": "d340a0a4-05cf-59b8-8bef-b37c560083d5"
}
5

I know this is in the 13 forum, but here you go, Framework 16. This was interesting:

-------Linux----------

No GPU detected.
{
    "mistral:7b": "41.25",
    "llama3:8b": "39.19",
    "phi3:3.8b": "51.70",
    "gemma:2b": "92.49",
    "gemma:7b": "36.76",
    "llava:7b": "43.84",
    "llava:13b": "4.35",
    "uuid": "bccd6f86-8b2d-5410-92fc-7120dbdd81f9",
    "ollama_version": "0.1.38"
}
----------
====================
-------Linux----------

No GPU detected.
-------Linux----------

No GPU detected.
{
    "system": "Linux",
    "memory": 58.668739318847656,
    "cpu": "AMD Ryzen 9 7940HS w/ Radeon 780M Graphics",
    "gpu": "Phoenix1",
    "os_version": "Arch Linux",
    "system_name": "Linux",
    "uuid": "bccd6f86-8b2d-5410-92fc-7120dbdd81f9"
}
----------
1 Like
6

I haven’t noticed this is the 13 forum. In any case, thank you for the benchmark

7

Here’s my performance on Arch Linux, 13" AMD:

-------Linux----------
No GPU detected.
Total memory size : 30.66 GB
cpu_info: AMD Ryzen 7 7840U w/ Radeon  780M Graphics
gpu_info: no_gpu
os_version: "Arch Linux"
ollama_version: 0.1.41

Your machine UUID : a79f8302-d19d-5b65-a214-213535d44a16
-------Linux----------

No GPU detected.
{
    "mistral:7b": "6.09",
    "llama3:8b": "5.64",
    "phi3:3.8b": "9.54",
    "gemma:2b": "14.14",
    "gemma:7b": "4.85",
    "llava:7b": "6.21",
    "llava:13b": "3.43",
    "uuid": "a79f8302-d19d-5b65-a214-213535d44a16",
    "ollama_version": "0.1.41"
}

Any tips on getting some more performance out of it? I’m using the open source amdgpu drivers and have tried both ollama’s own install and the pacman/Arch-packaged version.

8

You’re number are strange, I also used the open source amdgpu drivers and ollama’s own install, maybe check your kernel version ?

9

According to this post, setting a CLI flag on the ollama server let’s rocm use the iGPU Enable AMD iGPU 780M in Linux, Create amd-igpu-780m.md by alexhegit · Pull Request #5426 · ollama/ollama · GitHub

One thing that would be really interesting is if framework updated the BIOS to let us set the GPU reserved RAM. It would be really interesting to set the GPU VRAM to 8GiB. Right now it’s capped at 4GiB. I have a laptop with 96GiB of RAM, and I would be happy to give it 16GiB so I could load more complex models.

I have a 6900HX mini pc that I gave 8GiB to the GPU and it works just fine.

Even if it’s somewhat slow, that would be an excellent use of large amounts of RAM for offline line and local LLM development.

2 Likes