You could try setting media.hardware-video-decoding.force-enabled to true in about:config. I found that in this post/thread on the Arch forms about AMD and hardware decoding.
Running firefox with the MOZ_LOG="FFmpegVideo:5" environment variable shows that va-api decoding is working with that setting enabled for me with the Ryzen 5.
Edit: I just did some more testing and it seems that changing media.hardware-video-decoding.force-enabled wasn’t what did it. It was setting media.ffmpeg.vaapi.enabled to true that actually fixed it.
I’ve also had a fair bit of trouble with video playback in Firefox on my FW 13 AMD 7840U. I’m running fedora 39 with the rpm package (seen it for a while, but currently on v121).
After changing the suggested config flag this is what I get:
Interesting to see that while HARDWARE_VIDEO_DECODING now shows as enabled due to it being forced on, other related items like H264_HW_DECODE and VP8_HW_DECODE still show as blocklisted.
Update: I reset the media.ffmpeg.vaapi.enabled config option to its default and ran through the instructions here: Howto/Multimedia - RPM Fusion. Both HARDWARE_VIDEO_DECODING and H264_HW_DECODEnow show as available, though VP8_HW_DECODE still shows as blocklisted. At least one site I had some trouble with appears to be working better now, although it’ll take some more time and testing before I’m confident this isn’t just a fluke.
So H264, H265 (HEVC), VP9 and AV1 is all I’d expect to see, with the exception that Firefox doesn’t support HEVC at all, software or hardware, I think for patient reasons.