I just did a VS 2012 reinstall for Silverlight work. This time I left Blend, LightSwitch, Foundation Classes for C++, Office Dev Tools, & SharePoint Dev Tools off and now the xaml designer works great! A little slow on first load still but otherwise everything is quick and responsive and all of the MANY issues are gone.
The funny thing is, Blend is still actually installed. I thought Blend was required for xaml work but decided to try leaving it out anyway because these xaml issues are such a major problem. Now if run change programs on VS 2012, Blend is checked - I uncheck it, it appears to uninstall, but it remains installed.
Maybe LightSwitch is the culprit? I tried it in VS 2010 and it was unusable slow so I never used it but left it installed and I always had issues with xaml designer in 2010 too.