Is there still a chance that the unhandled exception change might be reverted to .NET 4 behavior? This new behavior sounds quite scary. In many (if not most) cases, if a Task is not observed nor waited upon, it is interesting for its side-effects. Such side-effect may or may not have completed, or worse, have only partially completed when the task aborts due to an exception.
It's clearly dev-unfriendly to observe that exception some non-deterministic (but potentially long) time later, but that's a hardly an argument for the change: it's much worse to get non-deterministic and potentially unnoticed state-corruption or deadlock.
Furthermore, it's inconsisent with normal behavior and encourages bad habits. You really don't want people throwing exceptions and ignoring them; that's the road to very-hard to debug code. If such code were to be called in a synchronous fashion it would occasionally fail, leading to an unintuitive behavior difference - and one which encourages just adding a try with an empty catch block to the synchronous variant to maintain parity; a situation that makes the code much harder to maintain in the long run.
Swallowing asynchronous exceptions by default strikes me as being a small, short-term gain in simple scenarios at the cost of higher maintenance costs in more realistic scenarios in the long run.
A final small detail: In your web caching scenario, you don't add to the cache until the Task completes. You could just use the more concise ConcurrentDictionaries GetOrAdd method with the added advantage of a much smaller window in which the cacher may make multiple identical requests.