I've been doing some research, and there is a way to get around the bootstrapper flaw. It's a horribly convoluted way, but a way nonetheless.
As I suspected, when msbuild creates the bootstrapper based on the GenerateBootstrapper task, it simply dumps the required info into the resource section of the setup.exe file. Among these is the ApplicationFile property, which will be combined with the ApplicationUrl property and then apparently just launched by ShellExecuteEx. For ClickOnce from the web, this results in a http url getting passed to ShellExecuteEx, so it launches the default browser.
Since I can't modify the bootstrapper, and don't much feel like writing my own, I thought, what else can I do? If the ApplicationUrl property is omitted, the bootstrapper will simply try to launch ApplicationFile in the folder that the bootstrapper is in. Launching ClickOnce directly is out; I can't specify the path to rundll32.exe, and even if I could, I can't specify any parameters.
So there are two possibilities: place both the .application file and the manifest in the same folder as the bootstrapper, and simply omit the ApplicationUrl property. The bootstrapper will now launch the local .application file, which can find the (also local) manifest file, so it works. The other alternative is to use a simple .cmd file that launches ClickOnce directly (rundll32.exe dfshim.dll,ShOpenVerbApplication
http://www.example.com/someapp.application), and put that .cmd file as the ApplicationFile.
The downside of this is that you now need to include more than one file with the bootstrapper; so it must be zipped, extracted etc. ClickOnce just became ClickALot.
One way to get around this, which I've just tried and it does work, is to include both the setup file and the .cmd file in the resources of yet another bootstrapper (which I wrote myself). That bootstrapper will extract the real bootstrapper plus the .cmd file to the temp folder and run it.
It's convoluted, but it does seem to work, and provides a way to use the bootstrapper even if IE is not your default browser. As bad as this solution is, it's the best I can think of (short of rewriting the entire bootstrapper).
EDIT: I'll probably upload this stuff to the sandbox after I've tested it some more.