Well you don't, you can just go back to the Start screen and click the app you want to switch to.
Then why have the 'swipe from left' task switching at all?
I agree with jgd12345 - the (dare I say it) iOS style of closing apps would be nice.
For my own app, I've been playing with a custom gesture - place all five fingers on the screen to grab the app, move your fingers closer together as though you are screwing up a ball of paper, then throw the app to the left or right will close it. Sounds horrible in explanation, but it's quite a natural thing to do. Of course, this won't work if someone is missing a finger or thumb, so it would fail in accessability roles!