The WRT54Gv6 (actually, any version 5 and up) isn't really suitable for custom firmware-- as you suspect, it's a VxWorks router and doesn't have enough memory to run the full version of any of the custom firmwares you mention. OpenWRT, Tomato, and Gargoyle will not work on your router. DD-WRT has a "micro" version available that will work on your router, but it's missing most of the features that make running a custom firmware worth the risk.
That said, Tomato is worth the price of a WRT54GL (which, conveniently, is on sale @ Newegg for $60).