It builds on the SmoothHD and IIS.net demos with a very nice GUI, and an awesome use of some of Silverlight’s RIA style-features. For example, A magnifying glass with user control over magnification level. Which is crazy cool, and actually kind of useful when doing demos.

It also offers manual stream selection so you can simulate the experience at different bitrates.

Anyway, it’s a blast. Give it a spin.

Live Smooth Streaming

Lots is going on at MIX, so as not to overwhelm with too many posts, I’ll try to triple-dip on this one.

First, as part of Scott Guthrie’s keynote yesterday (now with transcript), we announced that live streaming is coming for Smooth Streaming. The key advantages of Smooth Streaming are as applicable to live video as on-demand. And for high volume live events where scalability concerns can force users into “waiting rooms” or a lower tier of service, the offered scalability may be even more important yet. In particular, my personal take on the killer aspects of live streaming are:

Seamless adaptive bandwidth switching

With on-demand content, particular shorter duration, you can let users with slow connections buffer a while and then play.  But live is live; if you’re offering just a 1000 Kbps stream, a user isn’t going to get a decent experience if the the bandwidth available to the player drops to 800 Kbps for more than a couple of seconds. And even someone with a 5 Mbps connection may see that shared between multiple users and computers, and may have multiple bandwidth-consuming apps running at the same time; it’s a lot to ask that the bandwidth NEVER drops below 1000 Kbps for the duration of a long event. Thus, the managed code heuristics module running inside the Silverlight client can continuously measure available CPU speed, bandwidth, and even window size, and then give the user the best content they can use at that moment. And it can seamlessly switch without any pause in the video or “buffering” message to a lower stream if needed or a higher stream if usable.

My hope is that this can break live streaming out of the lowest common denominators used to maximize availability. With Smooth Streaming we can offer fallback rates down to the minimum experience appropriate for the content, and as high a rate as the content justifies, with each user getting the best experience they can get at the moment. I hope this can make consumer HD streaming a reality for a lot of viewers.

SmoothHD.com is a great demo of the seamless stream switching for on-demand; the live experience will be essentially identical.

Leveraging scalability of the web via proxy caching

In Alex Zambelli’s formulation, Smooth Streaming “adapts video to the web, instead of trying to adapt the web to video.” The content is delivered in a series of small files each containing a few seconds of video and audio. And each copy of each chunk has the same URL, so proxy caches handle this automatically. So all the CDNs (like our launch partner Akamai) with great web delivery technology are able to immediately leverage their huge network of proxy edge servers with Smooth Streaming. Moreso, all the ISPs and organizations with proxy servers (and that’s most of them) can have multiple people watching the same content with only a single copy of each chunk having to be sent to the proxy. So those horror stories about a company’s internet connection being brought to its knees by everyone watching the Olympics or March Madness at the same time? We think we’ve done a lot to make that much less of a problem, since the more popular the content, the more scalability it gets through the proxy caching. Hopefully this can make the waiting room a thing of the past.

There’s plenty of additional tuning the CDNs and others can do to further improve caching for Smooth Streaming specifically, but it gets a huge boost automatically by leveraging the existing infrastructure of the web.

Live PVR

Since on-demand Smooth Streaming is delivered as a bunch of small chunks, and live Smooth Streaming is delivered as a bunch of small chunks, we’ve eliminated the hard line between a live broadcast and the on-demand version of it that used to need to be published several hours later. Instead, what’s live is just the latest chunk that’s available, but every chunk is still there (and likely still in the proxy cache). So that means you can pause, rewind, skip to the beginning, skip back to live, all during the live stream. Think of it as a PVR in the cloud.

And it’s already in beta

Better yet, we’ve already got a public beta of it for download! It requires IIS 7.0, running on either Windows Server 2008 or Vista SP1 (you can play with it without installing 2008).

It can be installed via our cool new Web Platform Installer as well as traditional .msi files.

The package also includes a "simulated live encoder" which loops out the bits from a local file to the server just like an encoder would do. This enables server and player development, testing, and configuration without having to actually run a live encoder 24/7.

We also have a new IIS Media Services portal with information about all the IIS media delivery technologies. There’s a nice overview about the platform as well.

Inlet’s Live Smooth Streaming encoder

As part of the announcement, Inlet also announced they’ll be the first to market with 3rd party Smooth Streaming encoding, support on-demand in Armada and (in a live demo!) Live in Spinnaker. We’ve got lots of Windows Media and Silverlight customers using Spinnakers with great satisfaction already, so adding Live Smooth is going to be a great upgrade.

Inlet’s John Bishop is presenting along with IIS’s John Bocharov at MIX as I type this in fact. There should be more details about their encoders, and the on-demand version of the session should be available by tomorrow. Here’s the details for finding it in the archives easier.

Delivering Media with Internet Information Services 7 (IIS) Media Services and Microsoft Silverlight MIX09-T56F

By: John Bishop, John Bocharov Tags: Media | Servers

See how to deliver media with the best user experience in a cost-effective, scalable, and highly manageable way. Learn how to expand your reach and improve quality using Smooth Streaming, how to save on bandwidth, and how to maintain control when using Progressive Download. Understand how IIS Media Services and WMS light up the media ecosystem from encoding to playback.

Others have noted we’re racing along at un-Microsoft speeds with Silverlight and Smooth Streaming. It’s only six months since we shipped Silverlight 2 and we’re already at Silverlight 3 beta. And Smooth Streaming wasn’t even announced then and we’ve already got on-demand and live servers in beta. But hey, this is the internet, and that’s how fast we have to go.

And of course this is only possible with the great foundations of technologies like Windows Media, .NET, IIS, and Windows. And at least as important, with the great partners like Inlet we’ve been working with for years. We’re all really excited about what’s possible here, and pushing to get it out there so people can use it quickly. We’ve known what video on the web could become for more than a decade now, and it feels like we’ve finally entered that last lap toward the finish line.

And a great event to be focusing on would be…

Live Smooth Streaming for the 2010 Winter Olympics

It seems like I’m only just now catching up on my sleep from the 2008 Beijing Summer Olympics, but the 2010 Winter Olympics in Vancouver are less than 11 months away (staring.

And yesterday, as part of Scott’s keynote, Perkins Miller (senior vice president, Digital Media, for NBC Universal Sports and Olympics) announced we’ll be doing Vancouver with Silverlight and Smooth Streaming.  And building on the Bejing experience with 720p HD and much deeper interactivity and information. His whole comments are great, including some interesting numbers about Beijing viewership and NBCU’s vision for the 2010 experience .  I’ll just close out this already long post quoting his section from the transcript:

PERKINS MILLER: Thank you, Scott.

Hello, everybody! I was here about one year ago at this time, in fact, as we were preparing for the 2008 Beijing Olympics. How many people here watched some of the 2008 Beijing Olympics, those phenomenal athletes? (Applause.) I mean, it was just incredible. I mean, it was the single largest viewed audience that we had on television in history. It was also the single largest digital event in history. And we did that in partnership with Scott's team here, with the Silverlight group.

It was phenomenal. We did more than 50 million unique visitors, more than 1.3 billion page views. We streamed more than 70 million clips in the 17-day period of time. We served up 10 million hours of video.

And critically we served up 5,000 individual clips for consumers each and every day of the second week of the Olympics.

And what that really showed us is that the long tail really works. People in their desk, in their houses wanted to go watch rowing, wanted to go watch beach volleyball, wanted to watch all these sports at their time, at their leisure, and we enabled them to do it. It was an absolutely phenomenal partnership.

It really worked out well for us on the commercial side supporting all our interests. We were able to show for people who went to NBCOlympics.com and watched video and consumed content there, when they went back to watching their television, they watched twice as much television. Think about that, twice as much television for people who went and used a digital platform. It really brought to life the fact that we need to deliver as a media company a full 360-degree experience to our customers.

In addition, the people who watched video online during the time, if they were in the office or at their house, those folks who went and used the enhanced experience – you're seeing some of the examples here on the screen – these people who watched the enhanced experience like picture-in-picture, being able to watch four live feeds simultaneously, they watched three times as much video as those people who just chose to watch a single stream or a less enhanced experience.

What that told us is if we deliver a higher-quality experience, if we deliver something that's going to engage the audience, they will watch more. They will be more engaged by our product, and they will ultimately serve as a better audience to our product. So, it was really a tremendous experience.

Now, who knows, how many people can tell me where the next Olympics are? Anybody? OK, I didn't hear anything. Oh, come on, Vancouver, 2010. It's the winter Olympics. It was kind of a trick question; there are summer and winter games. So, the next Olympics are in Vancouver, Canada. They're roughly a year from now.

And I'm here to announce for the first time a renewal of our partnership with Silverlight. They will be providing the enhanced online video coverage for NBC's coverage of the 2010 Olympics in Vancouver, Canada. (Cheers, applause.)

And I am truly honored to be out here today because Scott's team has just done a phenomenal job coming up with what we think is going to be the ultimate product for a video event online. We're going to be streaming the Olympics fully in HD. This is going to be an adaptive, Smooth Streaming event, full 720p.

So, when you walk out of your house in the morning, and you go to your office, and you've left your beautiful 52-inch HD television at home, and you sit down at your desk and you want to grab some video from the Olympics that day, it's going to mirror that experience. We're going to be able to deliver you that continuous, high quality experience that you expect now as a consumer.

In addition, we're going to be able to bring you the DVR experience that you expect. So, just as Scott was illustrating earlier, you'll be able to pause the live stream, you'll be able to rewind the live stream, you'll even be able to go slow motion. So, for those of you who have been following the World Cup skiing this winter, there's a woman by the name of Lindsey Vonn who absolutely lit the world on fire. She won the World Cup overall. She is skiing phenomenally well. And if you want to see her fly off a 150-foot jump in the Vancouver Olympics – 50 meters for those from Canada – you'll be able to watch here in slow-mo online land those jumps and go through to the finish line.

We'll also be able to deliver to you full metadata overlays. What this means is that we'll be able to take the live results, the athlete biographies, the country information, all the information that gives you the kind of context for when the Olympics is, and deliver it to you as part of the enhanced Olympics experience. So, if you don't know who Lindsey Vonn is, you'll be able to find out who she is.

In addition, we've found that when you're watching the Olympics, and when you're trying to be engaged in this event, it's all-consuming. People want to know what's happening, when it's happening, and they want to consume the content where they are, whenever they are. This means that live video alerts are going to be critical. And for these games we'll deliver you not only live video alerts, we'll be able to give you real time feeds of those alerts. So, if you signed up to get the most popular clip of that moment, we'll deliver that to you. And again it will come through, depending on your platform, as a full HD experience.

Finally, because we need to commercialize this, for those of you that know the sports business, we do need to find ways to bring our partners to the table. We have the ability this time to do live ad insertion with our live streams. This functionality is going to be critical to give you as a consumer a fairly seamless experience, still be able to enjoy the live event, but allow our commercial partners access in order to find a way to associate with this great event that we'll be putting on together.

So, I can't tell you how excited I am. I'm thrilled to be here, I'm thrilled to be working with Scott's team. We have 331 days to go to the 2010 Olympics in Vancouver, and I hope you all watch, and thanks very much. (Applause.)

In closing, I’d like to add my own (Applause) as well. Great stuff.

Although it’s only 329 days now. Back to work…

A developer release, not a consumer release

Note that this is a developer release. Silverlight 2 was unique in scope, being the first release with managed code, so we did some consumer events using Beta 2, most notably the NBC Olympics coverage. We’re not planning on doing anything customer-facing with Silverlight 3 until the full release. So this is really a change for developers, designers, and media folks to get a head start on using the new features, and giving us feedback on any issues. End users are welcome to play around with it (I’ve got it installed on my home machines without any issues), but don’t expect to see much stuff on the web taking advantage of it until after it ships.

What’s Available

First off, here’s the Silverlight-specific downloads released today. The official Get Started page is at Silverlight.net.  And Tim Heuer has a great overview of everything.

• Silverlight 3 Tools for Visual Studio
• Expression Blend 3 Preview
• Silverlight 3 SDK CHM File
• Updated Silverlight Toolkit

MPEG-4 (H.264 and AAC-LC) support

First up, as we discussed back at IBC, we’re adding support for MPEG-4 content in Silverlight. Specifically, we’re supporting

• Self-contained .mp4 (including .f4v and .m4a) and .mov file formats (no reference movies or anything fancy like that).
• H.264 video in Simple, Main, and High 4:2:0 profiles (progressive scan only)
• AAC-LC audio mono or stereo (HE AAC will play back with lower fidelity, as in QuickTime)
• Local files or http progressive download.

Or, sliced another way, Silverlight 3 will be able to play pretty much all MPEG-4 files that would play back well in both QuickTime and Flash.

The H.264 and AAC-LC decoders are exposed via the MediaStreamSource used by Smooth Streaming, which means extending support to other file formats and protocols will work there as well. And of course, Smooth Streaming will support H.264 and AAC-LC as well.

Raw AV Pipeline

MediaStreamSource started with the ability to pass off VC-1 and WMA bitstreams to the decoder from managed code, enabling us to add protocol and format support within Silverlight. Most of Smooth Streaming on the client is implemented in C# as MediaStreamSource code. And we’re extending that to H.264 and AAC-LC. But there’s a huge number of other codecs out there that different markets are using and have asked to have in Silverlight. In fact, far more than we can reasonably build and test, and far more than we’d want to have to make everyone download in the installer.

So, instead we added three very simple but very powerful new outputs to MediaStreamSource: YV12, RGBA, and PCM.

Yep, managed code can now output uncompressed video frames and uncompressed audio straight to the media pipeline.

Most video codecs are going to use YV12, which is a flavor of the  a standard Y’CbCr 4:2:0 format used in most video codecs . We also wanted RGBA to enable video with alpha channels, and also any media formats that are natively RGB. And of course PCM is classic uncompressed audio.

Using a managed decoder is transparent to the user; it’s just another managed code .dll included in the .xap file. Since it’s running inside the Silverlight sandbox, there’s no download or other action required by the user to activate it; acts just like any other codec.

So, what’s codec performance like in managed code? It’s actually going to be fine for a variety of tasks. In our internal testing and prototyping, managed code offers about half the speed of a native compiled version. So for older codecs like MPEG-2, a managed decoder should be fine on a wide variety of hardware.

And remember, managed code doesn’t require C#. Existing decoders in C or C++ can be used as the basis for a managed decoder. The main restrictions are that native MMS/SSE and pointers can’t be used.

I can’t wait to see what other kinds of formats and codecs people wind up delivering with the Raw AV Pipeline. We’ve already seen some very neat demos.

GPU scaling and compositing

Silverlight 3 adds GPU support for scaling and compositing. Other products may call this something like “GPU accelerated video playback” which can be a little confusing – this isn’t GPU video decoding ala DXVA. What it does is take bitmap elements, including video, into the pipeline at their native size and then the GPU takes care of scaling each object to its final size, and then doing the final blending and compositing of the whole application. For media players, this essentially gives every player Fast Path performance. So while it does accelerate video playback, it’s not actually accelerating video decode per se. But on single-core machines, anything we can do to save CPU from other tasks to give more to the video decoder, the better. Again, the net effect is lower system requirements for any content that gets scaled; which should often improve the bitrate lower-end machines can play in Smooth Streaming.

There’s a few new facets to tuning a player for GPU playback, so it’ll need to be turned on explicitly in the application. But the performance gains for any video that’s scales are tremendous, so we’re going to help making sure everyone knows how to get it on and reap the benefits.

Full-screen GPU scaling/compositing will be supported on all Macs and all Windows machines with DirectX 9 or higher (which is XP SP1 and higher – that’s most machines and GPUs out in the wild today). Due to platform limitations, Silverlight on Mac won’t support GPU compositing inside the browser window, but will in full-screen mode. For performance-critical players (particularly in HD) a good compromise there is to set Stretch=”None” for the browser version and scale in full-screen mode, so that the windowed version is Fast Path and the full-screen is GPU composited for equivalent performance.

Improved logging

Robust and detailed logging has long been a critical feature for Windows Media to enable advertising and billing.

We’re extending that support to Silverlight 3, for both Windows Media and Smooth Streaming. I’ll provide a link to the full details when they’re posted.

Media pipeline improvements

We’ve done some under-the-hood work in Silverlight to optimize for silky-smooth video playback with as few dropped frames as possible. For the beta, we’ve focused mainly on single and dual-core systems, and you should see some significant improvements with the same player and content in SIlverlight 2 (and of course using the GPU when scaling will offer further performance gains).

Out of Browser Silverlight applications

Silverlight 3 will allow applications to be dragged out of the browser and onto the local system as a standalone, double-clickable app. This is useful for all kinds of applications, of course, but for media players it’s great to not have to launch the browser when all you want is an applet media player. This includes offline support, so a Silverlight media player could be used on an airplane or anywhere else without a network connection. And since it’s a real Silverlight app, it still uses the secure sandbox and so Admin rights on the machine aren’t required to install or run it. We’ve got nice integration for both Mac and Windows with Start Menu and Dock compatibility.  Ashish’s blog has lots of details.

I figured this was as good a time as go through a hands-on project for Smooth Streaming.

The Source

We’ve had a few years of good fun with the Blender Foundation’s Elephant’s Dream clip (as I used here, here, and here). They have a new Creative Commons licensed title up called Big Buck Bunny, which is a lot brighter, and with a lot of nice chewy grass for high frequency detail. Elephant’s Dream is a great test clip for handling gradients and dark detail (It’s all about differential quantization), but Big Buck Bunny is arguably a better test for general compression quality, and also has some nice sky gradients as well.

I downloaded the stereo audio as a FLAC file, and converted that to a 44.1 KHz 16-bit WAV file with Audacity. I then downloaded source as a series of PNG files, and then used After Effects to combine the PNG sequence and the WAV that into a 1920x1080 Lagarith RGB AVI file. Since PNG and Lagarith are both lossless, and I didn’t do any scaling or color space correction, I didn’t do any preprocessing, just assembling the content into a file Expression Encoder 2 can consume. I’m a big fan of the quality of EEv2’s Super Sampling scaling mode, so prefer to use that instead of other tools where I have less control over scaling quality.

Setting up the Encode

Since we had other encodes up at SmoothHD.com, I wanted to follow the same data rate/frames size/GOP length parameters we’d already used. For the other encodes in the series. Specifically, the below, which give us seven bands from 2500 Kbps to 364 Kbps:

Tweaking

Smooth Streaming as a feature came in very hot for EEv2 SP1, and there wasn’t a lot of time to tune the presets.

There are a couple of safe tweaks I always recommend as previously mentioned:

• Adaptive Dead Zone=Conservative
• Overlap=On
• Scene Change Detection=On
• Search Range=Adaptive

Those help Smooth Streaming (and web-rate VC-1 in general) quality with minimal impact on encoding time.

For quality-over-speed projects, I also like to use

• Complexity=4
• Chroma Search=Full True Chroma
• Match Method=Adaptive

All together, that’ll increase encode time about 2-3x for a nice but not overwhelming quality difference.

And because this is animation content, I’ll also use

• B-Frame Number=2

In general, the less noise in the content and the closer the matches between frames, the more likely 2 B-Frames will be more efficient than 1. I’ve gone up to 4 B-frames for Camtasia screen recordings.

Charts!

One cool thing about Smooth Streaming encoding is that the independent chunks make analysis and graphing of the content a lot easier. We can analyze each chunk as an independent unit, and compare quality between bitrates and versions quite directly. Do note that there is some allowed variability in chunk size; we target for a 5 second buffer, so that any three chunks in a row should average around the same amount, but if a particular chunk is harder to encode than its neighbors, it can have a somewhat higher bitrate.

Here’s the .XLSX if you want to check out the source data:

I’ve encoded four versions of the content

• Default: the stock EEv2 SP1 Adaptive Streaming settings
• Fast: Adding the “safe tweaks” above
• HQ: Adding the quality-over-speed settings
• Insane: Taking HQ and adding "Insane” modes that add a lot more in encoding time than quality
  • Single-thread encoding
  • Complexity 5
  • Hadamard Motion Match

Insane is there to show what happens when you burn as much CPU as possible.

To show the impact of these different encoding settings on the video, I used an in-house command-line tool to dump a log file of the size, type, etcetera of each frame, and then did made the Excel spreadsheet linked above to see what interesting data I could find.

First off, let’s take a look at just a bunch of frames. Specifically, the frames of Chunk 214, corresponding to the scene where the flying squirrel is flying over the forest floor right before the spikes pop up.

I’ve used contextual shading here to show the quantization parameter for each frame. In VC-1, a 1.0 is the least compressed and 31 is the most compressed. In general, QP is a decent proxy for the quality of a given frame, with values below 8 generally being pretty good and values above 8 getting increasingly blocky.

Since this is a 2-second chunk at 24 frames a second, we have exactly 48 frames. And since it’s a Closed GOP, it starts with an I-frame (keyframe).

The Insane, HQ, and Fast settings offer pretty similar results, due to using the 2 B-Frame pattern and Lookahead to detect scene and other changes. However, QP generally goes down as encode time goes up. The Default encode starts out with much higher QP since the previous scene had used more, and so needed to catch up a bit. By the end of the GOP, its QPs are going down while the “2 B” QPs are going up for a similar reason.

Interesting in itself but it’s definitely a trees-over-forest view.

For 95% of the 10+ minute clip, all the frames look fine; as typical with a CBR encode. It’s in the hard parts where the differences show up.

For the below, I’m going to focus on chunks 209-221 (6:56 to 7:22), which includes the very challenging section where the squirrel is flying over all the spikes jumping up.

In the below, we’re looking at a few metrics here:

• Mean QP per chunk:  The average QP of all frames in the chunk, as a proxy over overall quality.
• Max QP per chunk:  The highest QP in the chunk. Since it’s the ugliest parts of the video that stand out, that’s also an important flag for quality issues.
• Chunk size:  How many bytes in the chunk. Since Smooth Streaming is all about delivering video chunks, chunk size is the primary measure of data rate.

Here’s the data:

You can see there’s quite a bit of variability in chunk size; almost 3:1. That’s fine; we’d expect to see that in content that alternates between easy and hard shots. The swing between Mean and Max QP is worst for the Default settings; Lookhead is handy to help the codec know what’s coming down the pike. For Chunk 217, we see Default has a lower Mean QP but a higher Max QP compared to the others. In general, the Mean and Max QP is highest in the Default, as we can see in the below graphs:

Overall, we see that Default is the weakest, and Insane the strongest. But the gap from Default to Fast is the biggest change; in most chunks, Insane and HQ offer the same Max QP, and HQ has a better mean QP in a few cases. However, there are some chunks where Default does better than any of the other modes; since it’s a 1-pass CBR, one mode may have used more of its buffer by a given point than another, leaving fewer bits for the next part.

For reference, here’s that QP 31 frame in our four versions (click on the thumbnails to open at full resolution).

Default:

Fast:

HQ:

Insane:

So, what went wrong with that particular frame? It was a rate control failure of some sort. We can see from the below that the surrounding frames got more bits and lower QP in Default; it would have been better with a more consistent compression level. It’s also interesting to note the big improvement from Fast to HQ for these sequence of frames; it’s when the content is the most challenging that the slower, higher quality encoding modes make the most significant difference.

When you watch the video at full speed, the QP 31 frame isn’t really that noticeable as it’s right at a scene change.

Given the structure of Smooth Streaming the optimum way to allocate bits is quite a bit different than “classic” web encoding with the VC-1 Encoder SDK used in Expression Encoder was tuned for. Any compression engineer reading this has probably already thought of a dozen ways to improve encoders for chunked video. More about that later.

Lastly, here’s how long each version took to encode.

• Default: 59 minutes
• Fast: 59 minutes (1.0x Default)
• HQ: 158 minutes (2.68x Default)
• Insane: 268 minutes (5.4x Default)

As you can see, you rapidly get to the point of diminishing returns with slower encoding modes; the biggest improvement was from Default to Fast, which had no impact on encoding time at all.

First, install it already! It can be downloaded here, and probably most of you picked it up via Microsoft Update by now. James Clarke had a great overview blog post about it which I recommend.

This first post I’m going to focus on Smooth Streaming and multibitrate streaming in general.

“Adaptive Streaming” and Smooth Streaming

Here’s the big one: Expression Encoder can now do multibitrate encoding. Alex Zambelli has a good history of the technology.

When doing multibitrate encoding, EE does multiple simultaneous encodes from the same source file. While the big driver for that feature is Smooth Streaming (another overdue blog topic), we also support encoding a single multiplexed “Intelligent Steaming” WMV file or individual WMV files for each bitrate.

SmoothHD.com

And If you haven’t seen it yet, head on over to the SmoothHD.com demo site we do with Akamai’s  and check out the technology in practice. And yes, all those files were encoded with SP1.

SmoothHD.com in action. Those bars in the lower right show which data rate band you’re getting; mouse over for more details. Click on it to bring up a diagnostic menu and play around with scaling. And boy, that 20 Mbps DSL upgrade I’ve got on order can’t come soon enough!

Previewing Smooth Streaming with Expression Encoder

So, you want to play around with Smooth Streaming, but don’t want to have to upload your files to Akamai every time you want to do a quality check? Fair enough, but we don’t have media players that support the Fragmented MPEG-4 format used in .ismv files yet. Fortunately, EEv2 SP1 also includes a little localhost Smooth Streaming web server built into it. If you encode to a Silverlight 2 template and make sure that “Preview in Browser” is checked, you’ll get Smooth Streaming working, heuristics and all! It’s only on the local machine, and only when EE is running, but it’s a start.

To preview the video quality of Smooth Streaming, you can also make individual WMV files and watch them as normal. The output bitstreams are identical no matter what the file is wrapped in.

WMSnoop remains a great (and free) way to check out the GOP structure of WMV files. Below (click to zoom) you can see how Smooth Streaming produces a rock-solid cadence of a keyframe (in yellow) every 60 frames, while still being able to insert extra keyframes as needed for high quality scene changes.

Encoding for Multibitrate

Unlike the old FSDK-based MBR encoder, each stream is encoded in its own thread, which means a whole lot of cores can be saturated; my 8-core typically is running at nearly 100%, and if HD is involved, a 16-core would as well. The limiting factor can wind up being the speed of the source decoder past a certain point. Here’s a pretty typical load I’ll see when running from a Lagarith source file.

Multibitrate Video Options

Most of the parameters must be identical per stream; only frame size and bitrate can vary each. Other parameters like frame rate, audio, and codec settings are fixed. This is to facilitate Smooth Streaming by making sure that I-frames appear at the same point in each output stream. For technical reasons with the current VC-1 SDK, this also means only 1-pass CBR is supported for multibitrate encoding.

Multibitrate Audio Options

Only one audio option can be selected when doing Adaptive Streaming encoding. This isn’t the limitation it would have been in the WMA days, as WMA 10 Pro’s low bitrate audio quality is so good. We find 64 Kbps provides great quality for most content, and fits into most web delivery bitrates.

Multibitrate Container Options

There are three container choices

• ASF Single File. This gives you one WMV file for each video data rate, each including its own copy of the audio. This is an easy way to make multiple versions of the same source file.
• ASF Multiple File. This gives a single WMV file containing all copies of the video and one copy of the audio, ala Intelligent Streaming. This provides bitrate switching when used on Windows Media Services.
• IIS Smooth Streaming. This is for use with the forthcoming IIS Media Pack version that enables Smooth Streaming. To use this today, you’ll need to be part of Akamai’s beta program. That gives one output file per stream. There’s no way at the moment to play these files easily on the desktop, as they use the Fragmented MPEG-4 file format, which isn’t broadly supported in media player software; it’s mainly used inside of set top boxes like a TiVo.

Multibitrate Advanced Codec Settings

There’s a few required settings to get optimal Smooth Streaming compatible content in EEv2SP1. Smooth Streaming requires that all bitrates start have Closed GOPs (Group of Pictures – a keyframe and all the frames that reference it) starting on the same frame every few (typically 2) seconds. This means that the decoder gets a continuous sequence of frames to decode without any overlaps or missing frames, and without having to run two simultaneous decoders. These are compatible with (but not needed for) ASF, and using them gives the option to remux to Smooth Streaming at a future date.

• Maximum QP=31. This is a new feature in SP1, which specifies the maximum allowed Quantization Parameter that can be used. This is similar to the old WME “Smoothness” control which determines the tradeoff between preserving image quality versus frame rate. We’re trying to avoid a frame getting dropped in one data rate but not another, which could throw off GOP alignment. QP 31 is the most compressed a frame can be in VC-1, thus minimizing the chance of a skipped frame.
• Adaptive GOP=Off. Adaptive GOP tells the codec that it can “reset” the Key Frame Interval, making it into really “keyframe at least every.” That improves efficiency, but makes it possible that the different output bitrates might wind up starting new GOPs at different times, without any mechanism to resync them. When Adaptive GOP=Off, it’ll still insert an I-frame where appropriate to optimize quality, mainly when there’s a video cut in the middle of a GOP.
• Closed GOP=On. In a Closed GOP, the first frame of the GOP is always an I-frame (keyframe), and no frame in the GOP references any other GOP. That makes each GOP fully self-contained, as required for Smooth Streaming. If Closed GOP were off (Open GOP), B-frames can reference frames in other GOPs. This helps efficiency a little bit, but means you always need to have decoded most of the previous GOP to play the first frame of the current one. Even if Closed GOP=On, any additional I-frames inserted due to scene changes may be an Open GOP for improved efficiency.
• Output Mode: Elementary Stream Sequence Header. With an Elementary Stream Sequence Header, the VC-1 bitstream includes all the data the decoder needs to play back the chunk of video in that chunk. It’s required (and it’s a tiny bit of data; there’s no downside to using it).
• Insert Skipped Frames=On. With Skipped Frames, a frame doesn’t get encoded due to exceeding the Max QP or because it’s just a static frame without motion, consist of just a flag indicating “I’m the same as the frame before.” If Skipped Frames=Off, the frame before the dropped frame has its duration extended to cover the missing frame(s). The actual video playback is identical in either case; but this  ensures dropped frames don’t cause a GOP misalignment. There’s no real downside to using this for all files, Smooth Streaming or not.

Tips for multibitrate encoding

General Codec tweaks

These are the changes I made from the Adaptive Streaming default, implementing the typical “make it look good” settings I recommend for EE in general also apply to multibitrate compression. Just don’t mess with the settings up above!

Always On recommendations

These settings won’t have much impact on encode time, but can help quality significantly, particularly with

• Adaptive Dead Zone=Conservative . This tells the codec to bias toward softness over blockiness as a frame gets more compressed.
• Overlap=On. The Overlap filter softens the edges of blocks, reducing blocking and artifacts at higher compression rates. Web delivery is almost always at rates where the Overlap filter helps quality.
• Scene Change Detection=On. Scene Change Detection tells the codec to buffer ahead 16 frames to look for fades, flashes, and scene changes, and dynamically change the frame type based on that. This particularly helps with fades to/from black, and with flash frames (think raves and gunfights). While it can insert an I-frame, it won’t disrupt the pattern locked in by having Adaptive GOP=Off. Scene Change Detection never hurts quality, and often helps. Scene Change Detection maps to the Lookahead parameter used in the Format SDK.
• Search Range=Adaptive. Lets the codec dynamically adjust the search range based on the amount of motion in the content. Helps quality a lot when there’s more than 64 pixels horizontally or 32 pixels vertically of motion between two P-frames, without spending CPU cycles on frames with more sedate motion.

Quality over Speed recommendations

These are settings I use when I’m more worried about optimum quality than the speed of encode.

• Complexity=4. Complexity controls how hard the codec works on each frame, particularly the precision and thoroughness of looking at motion from frame to frame. Complexity 3 is a great default, since it provides most of the value of 4 and 5 at a higher speed; quality drops of quickly as you go to 2 on down. But 4 is a little better yet, particularly with complex motion like particle effects. Complexity 5 is overkill most of the time, but I wind up using it sometimes when I’m feeling more fussy than rushed, “just in case.”
• Chroma Search=Full True Chroma. Chroma Search looks for where the chroma (color) of the image changes differently that its luma (brightness).. This is particularly helpful with motion graphics and animation. Chroma is only a third of the data in the encode, so a full precision chroma search doesn’t add all that much to the encoding time, and with the right content can really help squeeze things down.
• Match Method: Adaptive. This switches between using Sum of Absolute Differences (SAD) and Hadamard motion matching on a per-macroblock basis. Don’t sweat what that means too much; just know it’s slower and a little faster than the default SAD-only. Full Hadamard is slower yet and sometimes lower quality than Adaptive.

Stretch, not Letterbox

Expression Encoder defaults to letterboxing to be used when the output frame size doesn’t match the source aspect ratio. However, even a slight variation in output aspect ratio from the source’s (typically to get the encoded height and width divisible by 16, the most efficient option) can result in a thin black line of letterboxing (top and bottom) or pillarboxing (left and right). If Resize Mode=Stretch and Mode=Profile Adaptive, Profile Adaptive will get the output frame size in the right ballpark while Stretch adjusts the video slightly to avoid those darn black lines. To make sure the output aspect ratio really is perfect, you can set the Video Aspect to that of the source, and on playback it’ll stretch the image pack to the exact original frame size and shape. Leaving it at the default of Square Pixel generally looks fine as well.

Note that you may need to reapply the Profile Adaptive setting if you apply a job preset to multiple files at once. That can reset to Mode to Custom.

Integer Framerates

By default, Adaptive Streaming profiles gives a keyframe every 2 seconds. However, that’s a precise 2.0000 seconds. So, if your source is a typical NTSC frame rate like 29.97 or 23.976, you’re 0.1% short of 30 and 24 frames a second, and so wind up getting a keyframe every 47 or 59 frames, not the 48 or 60 you might expect. If you set the frame rate to the whole number equivalent, the encode will actually maintain the source frame rate correctly (the value serves as an upper bound, but EE won’t insert duplicate frames when set higher than the source). But it will round up the keyframe cadence to the whole number, so you’ll get the keyframe every 48 or 60 frames like you’d expect.

Not really a big deal in practice I suppose, but I’ve long since internalized the multiples of 48 and 60, which makes it easier to me to figure out where my keyframes are.

Determining bitrates

Picking the optimum single frame size and data rate combination for a particular piece of content can already be a challenge. Having to determine optimal sizes for a bunch of different data rates. Fortunately, Alex Zambelli is as always ready with a handy utility for us – the Smooth Streaming Multi-Bitrate Calculator!

All you need to know is

• What bitrate works for the highest frame size for your content
• What minimum bitrate you want to go to
• How many steps to take

There’s actually some pretty deep math in there that knows how VC-1 bit-per-pixel requirements change as frame size changes, etcetera.