Lumia Imaging SDK v3 (No, it's not just for phones...)
This week, surprisingly enough, we're going to be doing a "Holly Cow, that was an awesome Hardware Event last week!" theme, focusing on related SDK's, projects and such.
Today, we're going to highlight a new Imaging SDK version, one that given the name you might think is for phones only, but it's not!
First Clint Rutkas, yes, that Clint, introduces us it in a recent post. Then we detail what's new, the cool Samples Repo and finally two posts Igor Ralic on upgrading to it.
The key feature introduced in Lumia Imaging SDK 3.0 is support for GPU based image processing through Direct2D. For most applications, this leads to significant performance improvement compared to the same application implemented with an earlier CPU-only version of the SDK. Where possible, exact implementation details of GPU support have been abstracted away from the API so that majority of the developers don’t need to be aware of what’s happening under the hood.
Another focus area in the 3.0 release is interoperability with other Windows 10 APIs: rendering to SwapChainPanel is supported, and Direct3DSurface, SoftwareBitmap and WriteableBitmap can be used both as image sources as well as rendering targets. With its high abstraction level and wide range of features, Lumia Imaging SDK 3.0 complements other Windows imaging APIs like Win2D and Windows Imaging Component (WIC).
Lumia Imaging SDK, now as version 3.0, makes technologies that Microsoft uses in its own imaging apps available to third-party developers. The features provide the ability to decode and encode images, apply effects, blend, crop, rotate, and resize. It gives developers easy access to world-class imaging algorithms and effortless GPU utlilization (utilizing Direct2D under the hood). It also allows the developers to extend the functionality for custom scenarios.
Note: Despite the name, the Lumia Imaging SDK actually works on any Windows 10 or Windows 10 Mobile device, as well as on any Windows 8.1 or Windows Phone 8.1 device.
The Lumia Imaging SDK is a rich set of image and video frame processing tools for developers to provide amazing visual experiences in their apps. It provides more than 60 pre-made effects, many of which can be applied without decoding the whole image. That means that you can apply effects to high-resolution images without worrying about your memory budget. You can add an effect to your existing project with just a few lines of C#. The library can also be called from C++ code. The effects can be chained to create the exact effect you're looking for.
- Quick Start, if you just want to get up to speed.
- Core concepts, explains the main concepts and functionality of the Lumia Imaging SDK.
- API reference documentation, quick jumplist to where the effects have been documented in more detail.
- Sample projects, alternatively, you can try out and quickly see the library in action.
Tip: Check out our open source project Lumia Imaging SDK Extras in GitHub. It will make it even easier for the developer to build amazing imaging apps. The source code project can be found in Lumia Imaging SDK Extras Git repository .
New in SDK 3.0
- GPU image processing - The performance has been improved by utilizing the GPU.
- Video Frame processing - Lumia Imaging SDK 3.0 provides a Microsoft Media Foundation Transform (MFT) making it possible to apply the effects to video frames.
- Windows interoperability - Lumia Imaging SDK 3.0 supports rendering to SwapChainPanel as well as rendering to or from the new Windows 10 imaging APIs SoftwareBitmap and IDirect3DSurface.
- Information about valid ranges for property values are now available at runtime through PropertyDescriptions. This allows the developer to easily match slider values to the supported range.
- Windows Phone 8.0 and Windows Phone 8.1 Silverlight applications are no longer supported.
Lumia imaging SDK has been updated with GPU capabilities and Universal Windows Platform (UWP) support. The development was focused on enabling these two scenarios with other improvements on the side.
The Lumia Imaging SDK version 3.0 is not directly backward compatible with earlier versions. The main difference is that we have changed the way the image processing graph is organized. We have moved away from the concept of filters; all filters are now instead effects. For information about how to upgrade from 2.0 to 3.0, see How to upgrade guide.
With SDK version 3.0, the Silverlight application framework is no longer supported. For Windows Phone 8.0 development, use Lumia Imaging SDK 2.0 that is available in NuGet.org.
New classes and Windows 10 interoperability
In Windows 10, new classes SoftwareBitmap and Direct3DSurface have been added to make it easier to work with images. We have added an image source and a renderer class for each of them. This allows the application to interact with the new Windows 10 types in a direct and efficient way.
New building block interfaces such as IImageProvider2 and IImageConsumer2 have also been added. These interfaces improve and extend the image provider model that the Imaging SDK exposes; however, the developers can continue using IImageProvider and IImageConsumer when not implementing custom types. For more information, see Custom sources, effects and effect groups.
Hardware accelerated effects
The effects in the list below are capable of being rendered by GPU, which vastly improves the rendering time.
Lumia Imaging SDK projects
- 1. Quick start
- 2. Sample projects
- 3. Lumia Imaging SDK Extras
- 4. Reference
1. Quick start
Quick Start is a sample project accompanying the tutorial that helps to get your first app that utilizes the Lumia Imaging SDK up and running. This sample implements the following basic tasks: picking an image from the camera roll, applying an effect to it, and processing the filtered image to be rendered and saved as a full resolution JPEG.
2. Sample projects
Edit Showcase is an example app that demonstrates the use of effects on still images. The photo is processed with the predefined effects. In addition, custom controls are implemented for manipulating the effect properties in real time. The processed image can be saved in the JPEG format into the camera roll.
Video Effect sample is an example app that demonstrates the capabilities and performance of the Lumia Imaging SDK by allowing the user to preview and apply a number of real-time effects to camera preview. The effects are applied to the stream received from the camera and shown in the viewfinder. The effects can be changed using the buttons in the application bar. This example app supports recording and capturing of videos and photos.
Lumia Imaging SDK and Win2D Demo
Lumia Imaging SDK and Win2D Demo is an example app that demonstrates the use of the Lumia Imaging SDK together with the Win2D API.
Image Sequencer is an example app that demonstrates the use of the Image Aligner and GIF Renderer APIs for creating Cinemagraph-style animations in the animated GIF format. There are also some example image sequences that can be used as a basis for the alignment and animation.
Custom Effect Sample
Custom Effect sample demonstrates how to create Custom Effects to do image manipulation both on the CPU and the GPU.
3. Lumia Imaging SDK Extras
This repository contains extra functionality and sample code for the Lumia Imaging SDK.
... [Go to the Repo]
Finally we have those two posts from Igor...
Lumia Imaging SDK 3 is finally out. The long awaited SDK that was blocking some devs from porting their Windows 8.1 apps to Windows 10 UWP is now available with great new performance improvements and new features, but also some breaking changes. Highlights of the release are definitely Windows 10 support and GPU image processing that brings the performance improvements, but many more changes were squeezed in - you can find more details in the documentation.
Silverlight apps are no longer supported though so if you want the latest magic from this SDK, porting to WinRT is necessary. If you're just migrating from v2 to v3, there's a great document for that called (believe it or not :)) How to upgrade from Lumia Imaging SDK 2.0 to 3.0. One thing that you'll notice right away is that filters are no longer called filters, but effects, but a more important (and breaking) change is that there is no
FilterEffect. This means that instead of chaining the filters by setting the IFilters array of
Imaging SDK 3 also brings a new interface called
IPropertyDescriptionsimplemented by effects that gives you information about the minimum, maximum and default values of an effect property at runtime. It's super useful when you want to use sliders to change property values, which is quite often with range effects, but don't want to hardcode the value. Let's create a quick demo to see how this works.
It's amazing to see Lumia Imaging SDK going forward. With Windows 10 support, performance improvements and even new features, there's no better way to manipulate images in Windows 10 apps. For range effects, PropertyDescriptions should be used to detect minimum, maximum and default effect values.
I remember when I built my Photo Light app, a photo editor for Windows Phone 7. There were no libraries like this one at the time, I pretty much had to do everything myself, pixel by pixel, taking care of memory, performance... Nowadays, all we need is Lumia Imaging SDK! :)
In my previous blog post about PropertyDescriptions in Lumia Imaging SDK 3 I wrote some code for rendering effect results in real-time that wasn't really that good and resulted in performance issues. To be fair, the blog post was about using PropertyDescriptions so rendering was used only for a more complete demo, but as I was reading the documentation I noticed a couple of ways to improve the code I wrote.
In that code, whenever the slider was moved to change the strength of the blur effect, I would create a new
WriteableBitmapRendererto render the result.
This is bad resource management - creating a new renderer has a performance cost so it should actually be kept alive and only disposed when it's no longer needed.
Doing some performance analysis proved that this code was bad and what happened when I loaded an image (3072x1728 pixels) and moved the blur effect slider around.
WriteableBitmapsshould be avoided as much as possible and renderers should not really be recreated all the time for real-time rendering as that causes serious performance issues. Instead, renderers should be kept alive and reused by the app if needed again. When it comes to interactive/real-time rendering, using a
SwapChainPanelRendereris recommended, and it should live as long as
A few of the past times we've highlighted the SDK
- Picture this, the Nokia Imaging SDK and you
- Imaging images with the Nokia Imaging SDK
- A new v1 power tool for your Windows Phone Image tasks, the Nokia Imaging SDK 1.0