Sorry to hear you're having some issues. First off: the toolkit assemblies that are publically available are built against .NET 4. They can be used with the new CTP (as I did at MIX), but you'd have to download the source from Codeplex and rebuild them against those assemblies. I didn't actually use any OData v3 features in the session, so my use of the CTP was pretty superflous.
The 500 error you're getting is a generic WCF error and could be from anything, and it's vague nature isn't because of the toolkit, that's a behavior of WCF. If you add the following attribute to your service, you should see a better error message:
Feel free to ping me if you continue having issues.