You can ask a process nicely to finish what it is doing, but there is no guarantee it'll listen. And even if does finish what it doing, there is no guarantee that telling it to finish was a good idea.

You ought to look at SIGTERM and SIGKILL. This idea has been around since the proverbial dinosaurs.

The idea of the system over committing illusion is also pretty damn old. That new revolutionary technology is called virtual memory. You tell me how many people enjoy the experience of having most of the application's working data in swap. Not fun. Killing the process and removing all its state as an "alternative" to virtual memory isn't some magical workaround to this either, it's actually substantially more insane. That's why competently programed operating systems tend to use that capability as a last resort when even virtual memory fails.