I also cannot tolerate Vista because of Explorer. Other things are tolerable, but Windows Explorer has changed so much, they should have given it a different name, that's what I think sometimes, it's not the same anymore. The experience I would call is RUDE. And because of Explorer, the whole UI of the OS changes, so the whole experience changes - Control Panel, (My) Computer, Recycle Bin, Network Connections, Network browsing. They've also spoilt Address Book and Add/Remove by making those 2 part of Explorer. Add/Remove although faster at populating apps than XP's Add/Remove is unbearable for me because of the new Explorer and it's ~!@#$ context-sensitive toolbar. Simply too many things are different.

1.  What can be the No.1 annoyance? Removed toolbar buttons and gave me stupid "Burn" and "Organize" buttons. Give me back my Cut/Copy/Paste/Properties/Copy To/Move To/Up. But this is largely solved now due to an addon I found. QTTabBar. It's a .NET addon however so it slows down Explorer startup. It also has some other goodies to change the behavior of the misbehaving rude Vista Explorer.
2.  Has complete amnesia. Doesn't remember a thing. XP's Explorer has partial amnesia.
3.  Back button doesn't behave the same.
4.  'Apply to all folders' doesn't really apply to ALL folders. There's a registry tweak I found for fixing this.
5.  Broken breadcrumb in FTP.
6.  No simple search for non-indexed files. MS says that Vista's search searches both, indexed as well as non-indexed, but only the indexed search is fast. The non-indexed search is much much much slower than XP's non-indexed search, so it takes sometimes a lot of time to find a single file in the same folder. This issue is also now sort of resolved because I use Agent Ransack on Vista for non-indexed search. And there are plenty of search utilities.
7.  Some commands like "Open command window here" only appear when Shift is pressed. Also, it doesn't appear in the left pane.
8.  The preview pane is too slow for videos, sometimes photos too. Large videos absolutely start killing the CPU if the preview "pain" is enabled.
9.  Although copying and replacing is improved, there is too much to read in the dialog and the dialog is too large and occupies a large area of the screen. I'm not a baby to press 3 large buttons when a file conflict occurs. Because the description/explaination is so long, I've to read it many times if I don't remember the position of the button.
10. Cutting and pasting across drives leaves empty folders on the original drives. It should remove them when moving is complete. Definitely a bug.
11. It tries to select the entire row in all views, even in list and details view. So if you click on a white space beside a file, the file gets selected! This is a huge pain when selecting multiple files. Thankfully, they added a relief in the form of check boxes.
12. Applies the wrong template when there are mixed files.
13. Renaming files selects only the filename minus the extension (Resolved by QTTabBar!)
14. Sorts in reverse, that is, when clicked on any column, first sorts in descending order.
15. Auto-refresh/autosort/autoarrange. This is my most hated feature of Explorer and is absolute rudeness to the user according to me. Suppose I paste 5 files, they're all sorted by name and scattered across.
16. Does not show the "free space" remaining in the status bar on all folders unlike XP Explorer. In fact, doesn't show different info which XP showed on the status bar.

Not to mention the slow file operations which are reportedly "fixed" in SP1.