I had already checked the compatability tab to ensure it wasn't set to run as admin there, but I rechecked it and it's definately not that.
I tried logging in as a different user and that didn't help either.
I tried the US English version (I don't see how it's 'default', when I go to firefox.com it points me to the British version) that didn't work either.
I also tried uninstalling firefox and then manually deleting every firefox folder I could find (bits left behind in Program Files (x86), AppData, etc.) and then reinstalling, but that didn't work either.
Thanks for all the suggestions guys, but it looks like it might be time for me to go back to IE.