Logout the user (if you're using some kind of authentication) and then go back to the initial screen. There is no need to close the app.
If you deliberately try and circumvent rules in the store, all you will achieve is getting your app banned. Annoying the user because you don't like Microsoft's policy is unlikely to go down well either at Microsoft or with your users.
That sounds like a good solution.