Some pretty important WPF performance updates just landed in .NET 4.5. I hope WPF performance updates keep comming because they can make or break some application scenarios. That update came many years after the initial launch of WPF though.
Our application is a code viewer and here we find that a "plain old (highly optimized) browser" is actually a great environment in terms of rendering performance and graphics flexibility. On the other hand, the more structured environment of .NET and WPF was appealing so we choose that. Had we been able to easily embed something like WebKit into the application we might have gone down that road.
There is a pretty sophisticated open-source WPF editor framework called Avalon - but in terms of flexibility with speed I still see the browser as being ahead of the game (but maybe I'm wrong, I'm still new to WPF). Obviously Visual Studio has a pretty performant WPF-based editor.
Since you've basically told nothing about what your little program does and what its UI requirements are, it's pretty hard to recommend anything.
As for using upper-case menu labels, I kind of fail to see the technical superiority of that but I guess you mean to imply that they were hip for doing it.
For simple applications I see nothing wrong with WinForms.