Custom hardware? What does that mean? Did you literally build the hardware? If that is correct then it is your duty to write proper drivers and make sure that the hardware works.
Or does "custom hardware" mean that you bought hardware and assembled the PC on your own? If that is correct and you get the described experience that might more likely mean that you bought some hardware that doesn't work well together. Or some of the parts
that you bought are faulty.
I bought a set of hardware components recently too. And assembled them as my living room PC. Everything is brand new. Installed Windows 7. Had it go to Windows Update and install all the drivers. Everything works great. Solid and fast performance.
Windows should probably have something that says: Uppps... your hardware has issues. Please replace component X.
To add to that, there have been problems with certain combinations of hardware/software/OS/drivers/whatever forever. People see one Vista (or whatever) machine with the problem and someone jump to the conclusion that it's the OS when the same sort of problem
could happen on any OS, including Winodws XP, Linux, OS X...