@kettch: There are devices available today, such as the YubiKey (http://www.yubico.com/yubikey) that take a simple approach. The problem is that they aren't integrated in the system, and so still provide enough friction as to cause most users to not be interested. The OS and software have to know about these two factor devices in order to remove this friction. Here's the UX necessary for users.
Sit down at your computer, where you're prompted to logon.
Tap your "key device" (phone or other device, doesn't really matter) for NFC, plug it in, let it communicate wirelessly or what ever. Details here aren't critical so long as what has to be done is easy. This would provide the users identity and a digital key (TOTP or otherwise... I'm not an expert on the best way to handle security here, I'm focusing on the UX problem).
The OS then prompts for a password (the second factor).
Authentication magic occurs and from that point on the OS knows who I am and has everything it needs to verify me for any other service. If I browse to my bank account the OS should have enough information already to identify and authenticate me, so I wouldn't be prompted again.
I'm not a security expert, so I can't give you the details on how best to implement things behind the scenes, but I know enough to know this is possible. The password and digital key on the hardware device (something I know and something I have) can be combined into a single key for encryption. By using a public/private key on the hardware (the system would be like SSH in this regard) and handshaking protocols it should be possible to make all of this (the initial logon and any subsequent authentication with other services) doable without ever storing a password anywhere. So you get all of the following benefits.
No password storage anywhere, so you never have to worry about stolen passwords. (Public keys would need to be stored, but those can't be hacked like passwords can.)
Two factor authentication, making it very hard for hackers to use social engineering or other tricks to gain access.
Usage is nearly as simple as today, but we get a truly universal single signon.
It's not full proof. There are attack vectors... I'm sure the security experts could point at several ways in which this could be attacked. However, it seems like this has to be more secure than what we have today.