It's not so much the desktop environment, as it is the fact that there are like 4 flawed choices of desktop environment, all of which are flawed in their own unique and annoying way. This is the only thing consistent right across everything in FLOSS/Linux; there are a whole bunch of choices given to you, none of which are any good. Why choose a distro when none of them are really adequate? Why can't there be just one good open-source Microsoft Excel clone, instead of several horrible options? Why does there need to be yet another god damned sound API, or another god awful 2D open-source graphics driver?
Another problem is the breakage that occurs pretty much constantly. A new kernel point release gets pushed through apt? Something’s going to break, and you're going to spend the requisite hour or two figuring out how to fix it, until you finally find some post on a forum (the 8th you’ve looked at) that solves your problem. Samba get updated? Well odds are the new version makes some small change to the way it reads config files, and suddenly your network shares are open to everyone on the network. Upgrading to a whole new release of Ubuntu of Fedora? Well you'd better hold onto your hats, and hope that wifi, sound, graphics, and flash don't all break at once.
How about Linux as a platform? How can a developer of a closed source application (say a game) be expected to target Linux, when the core bits in the system gets slaughtered with each point release? Not just the kernel, the core libraries and X11 do the exact same thing. Linux isn't so much a platform as it is a plate of spagatti, that might just run Firefox if it was custom compiled with a whole bunch of hooks for the specific point release of your distro, but don't you dare try to run the binary provided by Mozilla of a new version or you're sure to run into a world of hurt. Yes, if you completely lock down the system and make sure to only update everything very carefully on your own, like on a phone, an ebook reader, or a server, you can get a stable platform to target apps to, but that's nowhere near the reality of Linux on the desktop. Ever tried to run an old game (that is, more than a couple years old) on Linux? I don’t think I was ever able to get Doom 3 to run.
I abandoned Linux almost 2 years ago because I got tired of the constant fiddling and sacrifices I was making as a user; I had sh*t to do, and after 3 years on that garbage, I gave up, installed Vista, and have never been happier or more productive. The last thing I need is to watch OpenOffice, X, or my newly broken nvidia driver (thanks to some daily update somewhere) crash while I'm in the middle of writing a term paper -- again.