Ok folks, so I think we might make some headway here. The issue is that we decided to use a CTP service to do smooth streaming out of Azure blob storage. This is not a support Azure CDN offering at this time, but they are letting us use some early prototype servers to do this. Sadly what that means, although all of our initial testing went well, is that when a server dies, the correct procedures to detect that and get it removed from rotation are not in place. For the normal production CDN services, this would not be an issue.

This is not the CDN team's fault, it is a CTP after all, but more our fault (specifically my fault as I made this decision) for deciding to dogfood this service in production. An issue was just discovered today that may improve the situation once it is resolved. I'll post here when it has been fixed and we will also look at alternatives to streaming through this not-yet-ready CDN service.