Update:
I’ve gotten around the kinks of it. I installed Fedora 36, switched to a rx 550 and started using the looking glass client for my vm. So far no issues! 
If anyone wants to start with pci passthrough and looking glass, Pavol has an awesome repo on github that automates everything and has videos showing you through it