Ah this was it! I remember enabling CONFIG_HID_SENSOR_HUB=y when futzing around trying to get my touchpad to work. However I forgot to disable it after it didn’t fix my touchpad. After I disabled this option and re-built the kernel my brightness keys work!
It was definitely a bit of a mystery for me because most people found just blacklisting the module worked. However because I set my option to y instead of m the blacklist didn’t work.
I added the knowledge from this thread to the Gentoo wiki