I was wondering, are there any applications available to utilize the NPU on linux?
And not the kind for developers/tinkerers like running LLMs on it, but something more general user focused, e.g. OCR, Noise reduction, camera processing, presence detection, something that is useful in everyday workflow and would benefit from running a lower powered dedicated hardware.
From my findings i only found ppl trying to run LLMs for some reason and one project using it for TTS (which is what i am asking for).
P.S.: really need the tag “linux” be available for general discussions
1 Like
I know that lemonade is able to run AI models locally on the NPU. Here is a phoronix article on it: https://www.phoronix.com/news/AMD-Ryzen-AI-NPUs-Linux-LLMs
1 Like
Seems like the 1 very useful feature, STT, is not yet implemented with NPU for linux. But thank you for sharing.
P.S.: why is everyone trying to cram a LLM into a NPU, isn’t it massivly underpowered for anything but a novelty?
I actually tried it out. It was pretty decent for a local AI (I have zero frame of reference lol). I had to install lemonade-server on Arch, then follow its instructions to setup fastflowlm and xrt. Then, since linux kernel 7 isn’t out, I had to get a newer amdxdna kernel driver from amdxrna-dkms aur package. Then I could use the built in NPU.
To be honest, I think NPUs r kinda stupid. But, if you’re gonna use local AI, it’s pretty goated. Of course, you can always just use ollama-vulkan.
1 Like
Also noticed that lemonade only supports XDNA2 for now, so tough luck for my gen 1 no matter the OS.
Ye, i understand that the NPUs are very underpowered, but i thought of like having RNNoise analogue that uses the NPU, or having a model that mimics the IR sensors in the new OLED monitors to turn them off/lock but does that using the webcam, or macOS style build-in webcam background removal, etc.
All these applications don’t require high performance, but benefit from being always on and on a laptop that means being very power efficient. Thats why I don’t really see the use of CPU/GPU applicable because of power overhead.
But maybe I am missing a point somewhere.
I was looking into this a bit some time last year and I wasn’t able to find much of anything. Granted I’m a Windows user, but I kind of wanted a reason to use it (I did pay for it afterall). I don’t think I found anything other than LLMs to potentially run on it. I do like the idea of a denoiser (I think back to running RTX Voice on my old GTX 1060 laptop) for clearing up background noise for voice communications. Unfortunately, given how low the TOPS for the NPU are (I’m running the AMD Ryzen™ 7 7840HS) it seems like anything that could be used by it would rather run on anything other than the NPU (CPU/iGPU/dGPU). I tested out AMD’s privacy view a few times which is neat and works pretty well but still doesn’t run on the NPU. I really like the idea of a low power hardware to take care of small things that run all the time especially for laptops (like Intel’s efficiency cores). I would like to find something useful that I might actually use that would utilize it.
1 Like
Thank you for sharing.
The only thing I have found recently is Whisper models modified specifically for the AMD NPU. Seems that the XDNA is capable of running it in realtime, but none of the wrappers I have seen have implemented it.
Looks like there are options, but if you want to have it seemlessly integrated in your day-to-day workflow you gotta DIY.