Let's say there is 200 ms round trip latency to the gaming server. Also let's say two people are facing each other with shotguns. Whoever shoots first will kill the other one. Now if player A presses the fire button 50ms before player B, then player A should win, right? Well what really happens is that on both screens it looks like both players fired their guns, because the local game responds immediately to the input action (or as fast as possible given the local system delays). Then as far as the server is concerned, player A fired first resulting in player B losing. So even though player B will see their gun fire, he will still lose as if he didn't fire at all.
The only multiplayer shooter I play is Mass Effect 3. Its multiplayer uses a host/client architecture, with one of the players designated the host. The host is responsible for all game calculations, and the other players are essentially just seeing a simulation of what the host is telling them is happening. It does use the prediction system you're talking about, but this doesn't really mask latency very well.
If there is significant lag, symptoms include "rubber banding" (your PC and the host disagree about where you are so you constantly snap back and forth between the two locations), and a noticeable delay on actions (shoot an enemy: two seconds later, it dies). Some of the weapons and powers are not hitscan (instant hit) but use projectiles with travel time. In this case there is a delay between pulling the trigger and the projectile starting to fly, making it essentially impossible to hit moving targets with a projectile weapon unless you're the host. The delay is especially fun for things that are supposed to restore your shields. You think you used an ops pack in time to save your life, but unfortunately the host already thought you were dead so it doesn't work. And the game is evil enough that it will still subtract an ops pack from your inventory in that situation, even though it didn't do anything.
Another fun one is the magnet grab. A few enemies in ME3 are capable of "sync kills", which means they grab you and instantly kill you regardless of how much health and shields you had, and team mates cannot revive you if you've been killed that way. Each enemy capable of doing this has a certain range at which they can do this, and some conditions that must be met. The fun part is when you think you're nowhere near such an enemy, but the host thinks you are. You will get grabbed, dragged towards the enemy, and killed, with no way to stop it. On bad connections, people have been grabbed from up to 30 metres away, sometimes even through walls.
However, I rarely host and 90% of the time the lag isn't bad enough to affect gameplay. Strangely enough, on those rare occasions where I am the host it almost feels like actions have results before I perform them because I'm so used to the slight lag of playing off-host.
(And there is a glitch where if you're hosting, one particularly nasty enemy type (Phantoms) have 90% damage reduction during their dodge animation making them nearly impossible to kill with hitscan weapons on host, which is one of the reasons I actually prefer playing off host)
But, again, while there is noticeable lag in network games like that, I've never had any such delay on my own computer like what android was talking about, except in NFS3 which had notoriously slow control responses.