@rhm: Cool to hear about SlimDX. I remember reading about it back when in was inbeta. Nice to see it's a mature product now.
I'm not so sure Silverlight is a as resource hungry on the phone as you might think (though i might be wrong on that). According to Scott Guthrie silverlight on the phone does automatic bitmap caching of UIElement. Such that when you scroll across in your panorama control, the text and images aren't being redrawn on each frame. Instead the fonts, images and other graphics are drawn once on start up and the gpu moves the bitmap cache around the screen. I understand Silverlight on wp7 has other proformance issues, such as when initialing UIElements but i don't think the rendering of the UI itself is a big bottleneck in most apps.
I don't see making calls to xna being a big deal within Silverlight. If a developer is smart enough to design Silverlight control to overlay an xna game that can benefit from bitmap caching then the overhead will be extremely low. Also, if for some reason the Silverlight controls do cause performance issues, the will not cause the xna game to freeze since the controls are composited together on a separate thread.