I gave up on the alternative method in the end. Ran into too many problems with controls drawing directly into the window, or directly manipulating the window (e.g. shifting parts of it around in when scrolling), over the top of what I was doing. That's what makes the DWM thumbnail method so great: It takes the rendering output or the windows and does things to it without affecting it at all. (It also lets me easily create effects on areas of the top-level window which can cross over child windows.)
So I went back to plan A. Almost ditched the idea completely but it looked so nice (to me ) that I kept at it a bit more, and I think I now have it working really well, albeit with some kludges. Long term testing will determine whether the feature sticks or is canned.
While playing with this stuff I discovered that Windows 7's DWM process will crash (and sometimes reset the desktop back to classic rendering) if the user invokes Aero Peek while a DWM thumbnail is being moved around in another window as Peek fades things out to sheets of glass.
Workaround is to set the new DWM window flag on the window the thumbnail is rendered into, telling DWM to ignore that window for Aero Peek. (Drawback is that it then stays visible while the other windows are faded to glass, but in my case that's fine as it's only visible for half a second anyway.)
I also noticed that my overlay window is considered part of the main window when minimising/restoring the window, and within Flip 3D. So it's only considered a separate window during the window-open animation, which seems weird. It's like the pop-up is only "attached" to its parent after that window-open animation has completed. (I checked that I'm not opening my pop-up before the parent window has been made visible, and I'm not.)