For a cheap and easy setup, there is this USB Mic: http://www.samsontech.com/products/productpage.cfm?prodID=1810. It's a condenser mic as well, so is also great for vocals. I was able to get mine for under $100 IIRC.

I eventually stopped using it because the ASIO drivers conflicted with my audio card and forced me to have some ugly "hacks" to get it working in a low-latency environment. For your needs, I'm sure this won't be a problem though.

If you want to start "stepping it up", a good way to upgrade your experience is with a good digital audio card, a mixer (running into the audio card), and a real microphone into the mixer (you could even have a direct box for guitar if you wish). And a good quality and affordable DAW (Digital Audio Workstation) is SONAR. The Home Studio XL version of SONAR is quite powerful for hobbyist use and is affordable to boot.

Where I've sunk most of my money into is high quality sample libraries. MAN, those get 'SPENSIVE!!!!