Can I use custom Operating systems?

Can I add my own OS (that I developed :wink: )?

… sure, if it supports x86_64 and the drivers of your OS support it.

If you were building your OS based on Linux (so rather, you created a new distro, not an OS) and the kernel is pretty recent there is a good chance it will work.