It took almost 30 sec for the MEF & Silverlight 4 Beta video to get totally clear then I tried it again later and it only took 15 sec before it jumped into good quality. I'm not sure if the other videos had the same problem because they don't have small text in them.
The http://www.iis.net/media/experiencesmoothstreaming demo only takes 5 to 9 seconds to get good and clear.
I have a 15mbit cable connection.
it all depends on the data streams getting sent. If the player thinks it isn't getting enough to sustain a certian bitrate stream, it will drop down.
For the two different experiences, the network connection could have been saturated at one point and not the another point in time.