As I'm currently thinking about also going this route I do have some questions, if I may. Could you elaborate what your hardware setup is? Do you have two screens, and two powerful GPUs? Or do you use a HDMI Switch for one screen? Basically, how do you cope with the hardware limitations of GPU Passthrough
(I briefly looked into using a single GPU for passthrough, which seems possible but involved rewriting the GPU rom and I'm not going to do that :)
One screen, two GPUs. For the screen I've got an LG 38GL950G, and the GPUs are a Radeon WX5100 (macOS) and RTX 2080 Ti (Windows). I run the Linux host (Ubuntu) headless.
I have the macOS side connected via HDMI and Windows via DisplayPort. Then I switch inputs when through the monitor's OSD for changing the OS (I think there's a way to do this through DDC/CI commands depending on the monitor, in which case you'd be able to write a script to do this, haven't looked into that yet). I have a USB switch for moving all device connections to the OS of my choice.
Also the board I have, an X299 Designare, has several USB controllers on their own IOMMU group so you can pass through each to their own VM.
Then start my VM. libvirt automatically unbinds it from the host and binds to the VM. You're right though, in the past this has been problematic. It is working fine though now at least on my Radeon WX 5100.
You don't need to rewrite the GPU rom! I never set it up fully due to other issues, but I had High Sierra + GPU passthrough working quite well in ESXI last fall.
I would still be using ESXI now, except my (unusual) goal was to run 10.9, and I couldn't for the life of me get GPU passthrough working in 10.9.
The neat thing about ESXI, though, is that it is officially supported by macOS. Only on Apple hardware, of course, but with ESXI the motherboard is all fake anyway, so it works identically regardless of the host's hardware. All you need to do is patch out the Apple hardware check, which is easy.
I don't! For whatever reason, there is precious little information on the internet for running macOS in ESXI. I got the original idea from another Hacker News comment: https://news.ycombinator.com/item?id=17766016 And then, this is what you need to enable macOS on non-Apple hardware: https://www.insanelymac.com/forum/files/file/964-macos-unloc... The rest is the same as setting up any other VM in ESXI with GPU passthrough, so you can read tutorials on that.
High Sierra and an nVidia card "just worked" for me on ESXI. (Sadly, Mavericks did not—I could not get the GPU to output anything—and that was the only OS I cared about—but you probably don't care.)
There's a lot more information on using KVM, so you may want to go that route instead. But of course, then then you need to use a Hackintosh bootloader like Clover or OpenCore. It was really cool having everything just install natively with ESXI.
(I briefly looked into using a single GPU for passthrough, which seems possible but involved rewriting the GPU rom and I'm not going to do that :)