would that work in rocm as well ? i quickly scanned the sources, but as the _CUDA_+ suggests, seems to work for CUDA based loads ? But maybe i misunderstood.
It doesn’t have any effect, really. It’s ignored when used with ROCM/HIP. I compiled my llama.cpp without this variable, and it works just fine and uses GTT (unified) memory.
it is a runtime env in llama.cpp not a build time config.
$ export GGML_CUDA_ENABLE_UNIFIED_MEMORY=ON
$ llama-server [...]
It work perfectly fine with llama.cpp. juste don’t know if ollama have remove it.
(llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu at master · ggml-org/llama.cpp · GitHub)
and @Pattrick_Hueper yes it work on rocm, llama.cpp did not have code dedicated for hip/rocm it use a build maping so like with pythorch it use CUDA tag
Hop we can have a try HIP/rocm backend and not this horrible mapping. (llama.cpp/ggml/src/ggml-cuda/vendors/hip.h at master · ggml-org/llama.cpp · GitHub)
1 Like
On my AI Max 395, it only reduces performance a bit and results in incorrect GTT usage being reported by amd-smi.
Fedora 43 Beta, ROCm 7.10-nightly, fresh llama.cpp build:
With GGML_CUDA_ENABLE_UNIFIED_MEMORY=ON:
$ amd-smi
+------------------------------------------------------------------------------+
| AMD-SMI 26.1.0+c9ffff43 amdgpu version: Linuxver ROCm version: 7.10.0 |
| VBIOS version: 023.011.000.039.000001 |
| Platform: Linux Baremetal |
|-------------------------------------+----------------------------------------|
| BDF GPU-Name | Mem-Uti Temp UEC Power-Usage |
| GPU HIP-ID OAM-ID Partition-Mode | GFX-Uti Fan Mem-Usage |
|=====================================+========================================|
| 0000:c5:00.0 Radeon 8060S Graphics | N/A N/A 0 N/A/0 W |
| 0 0 N/A N/A | N/A N/A 217/1024 MB |
+-------------------------------------+----------------------------------------+
+------------------------------------------------------------------------------+
| Processes: |
| GPU PID Process Name GTT_MEM VRAM_MEM MEM_USAGE CU % |
|==============================================================================|
| 0 39391 llama-bench 149.1 MB 60.7 MB 148.6 MB N/A |
+------------------------------------------------------------------------------+
| model | size | params | backend | ngl | n_ubatch | fa | test | t/s |
|---|---|---|---|---|---|---|---|---|
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | ROCm | 99 | 2048 | 1 | pp2048 | 974.71 ± 8.71 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | ROCm | 99 | 2048 | 1 | tg32 | 46.16 ± 0.01 |
Without:
$ amd-smi
+------------------------------------------------------------------------------+
| AMD-SMI 26.1.0+c9ffff43 amdgpu version: Linuxver ROCm version: 7.10.0 |
| VBIOS version: 023.011.000.039.000001 |
| Platform: Linux Baremetal |
|-------------------------------------+----------------------------------------|
| BDF GPU-Name | Mem-Uti Temp UEC Power-Usage |
| GPU HIP-ID OAM-ID Partition-Mode | GFX-Uti Fan Mem-Usage |
|=====================================+========================================|
| 0000:c5:00.0 Radeon 8060S Graphics | N/A N/A 0 N/A/0 W |
| 0 0 N/A N/A | N/A N/A 156/1024 MB |
+-------------------------------------+----------------------------------------+
+------------------------------------------------------------------------------+
| Processes: |
| GPU PID Process Name GTT_MEM VRAM_MEM MEM_USAGE CU % |
|==============================================================================|
| 0 39579 llama-bench 57.2 GB 371.1 KB 58.6 GB N/A |
+------------------------------------------------------------------------------+
| model | size | params | backend | ngl | n_ubatch | fa | test | t/s |
|---|---|---|---|---|---|---|---|---|
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | ROCm | 99 | 2048 | 1 | pp2048 | 993.10 ± 1.32 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | ROCm | 99 | 2048 | 1 | tg32 | 47.44 ± 0.03 |
It is not incorrect reported, it do not use GTT it use RAM alloc and so no need to change is boot config to use full 128Go of memory.
Highly unsophisticated testing using gpt-oss:120b and the same prompt on our ollama
without GGML_CUDA_ENABLE_UNIFIED_MEMORY=ON
ollama-1 | time=2025-10-21T07:19:56.058Z level=INFO source=server.go:700 msg=“gpu memory” PID=1 id=0 library=ROCm available=“60.6 GiB” free=“61.1 GiB” minimum=“457.0 MiB” overhead=“0 B”
total duration: 1m1.501427013s
load duration: 19.803030254s
prompt eval count: 78 token(s)
prompt eval duration: 314.893557ms
prompt eval rate: 247.70 tokens/s
eval count: 1316 token(s)
eval duration: 41.092212524s
eval rate: 32.03 tokens/s
with GGML_CUDA_ENABLE_UNIFIED_MEMORY=ON
ollama-1 | time=2025-10-21T07:24:38.158Z level=INFO source=server.go:700 msg=“gpu memory” PID=1 id=0 library=ROCm available=“60.8 GiB” free=“61.3 GiB” minimum=“457.0 MiB” overhead=“0 B”
total duration: 1m10.581739644s
load duration: 18.26998993s
prompt eval count: 78 token(s)
prompt eval duration: 317.190192ms
prompt eval rate: 245.91 tokens/s
eval count: 1677 token(s)
eval duration: 51.638464361s
eval rate: 32.48 tokens/s
so … no difference it seems
that said, i checked our code and this line
is in, that is the one i mentioned above having found the usage in source code
What version/install/container do you use for ollama?
GGML_CUDA_ENABLE_UNIFIED_MEMORY=ON did not have many change on speed (may be a litle) what change is were the llama.cpp buffer is allocated.
- with it it use RAM so (even on Windows?) it can use all available RAM
- without it is allocated on “device” in our case: on linux vRAM+GTT on windows vRAM (there is no GTT for them)
And something else ollama need some path to report the “correcte” memory size available.
(There is many issus report for iGPU on ollama, one of the last is Bug: inconsistent to use VRAM and GTT of iGPU of AMD Ryzen AI Processor · Issue #12342 · ollama/ollama · GitHub and the longest is Integrated AMD GPU support · Issue #2637 · ollama/ollama · GitHub)
Keep in mind that depending on the model, Ollama may not even use llama.cpp as a backend, as they are developing their own. Even when it uses llama.cpp, it’s outdated and modified fork, so the best way is to just forget about Ollama and use llama.cpp directly.
I don’t use ollama either, only llama.cpp (or llamafile, but it hasn’t been updated for a long time and doesn’t work with the framework…). I most often use llama.cpp.
So, I’m familiar with the llama.cpp configuration, and I’m trying to figure out the right way to create a backend for our APUs.
So, if I can help those who are using ollama, it’s always good to see if it can be configured correctly.
My basic configuration that did not need to change GTT config is like this for the rocm/hip backend :
export GGML_CUDA_ENABLE_UNIFIED_MEMORY=ON
export LLAMA_ARG_UBATCH=2048
export LLAMA_ARG_FLASH_ATTN=1
export LLAMA_ARG_NO_MMAP=1
export LLAMA_ARG_N_GPU_LAYERS=999
Oh, I meant to reply to @Pattrick_Hueper
That is I hope explained above. I am using a self build container based on the latest adaptations by rjmallagon, link tohis and my git is above.
I don’t quite understand why we are getting back to the “you should use llama.cpp over ollama” again. ollama fits our usecase perfectly fine and we are happy with it. It’s very well possible that llama.cpp may be more performant. But for us ollama is just fine.
The question I saw asked was wether the cuda unified memory environment variable that seems to have an effect in llama.cpp also has an effect in ollama using rocm. I tried it, it didn’t seem to have any for me. Question answered I believe. If you’d like me to try a specific model with the environment variable I’d be happy to try it as well. But I’m completely content with the default assigned GTT memory we currently have, all our needs are covered.
If we ever have to tweak and optimize for performance for a usecase in the future i might try llama.cpp and maybe the unified memory switch as well.
1 Like
FWIW, I don’t use the unified memory variable with llama.cpp and it works just fine. With it, memory reporting is off and it seems to perform worse on my system.
oups… I got it 
I’ll have a look on it…
I think I figured out how memory size is reclaimed. It seems they’ve changed their strategy and are now using ggml to retrieve GPU information.
So it might be simpler than I thought.
I’ll try to find some time to look into building ollama and see if my idea works.