Portable Image and Video processing with help from AForge.NET and Accord.NET
Last week I kicked off a mini-Windows Phone Image App Dev Series, highlighting the Nokia Imaging SDK, Picture this, the Nokia Imaging SDK and you. The thing is that the Nokia Imaging SDK was released months previous as was the post I highlighted.
- Forging Player Card Detection and Recognition program with AForge.Net
- Accord.Net makes it easy to add Image Stitching/Panoramas to your application
- "Contour Analysis for Image Recognition in C#"
- iSpy with my little... OSS security camera/audio software suite
Both authors though are talking about today's portable world...
A very popular open source C# project for image processing, neural networks, machine learning etc. is the AForge.NET Framework, developed by Andrew Kirillow. AForge.NET has been used in numerous applications. It also forms the basis for the Accord.NET Framework, developed by César de Souza, which substantially extends AForge.NET with additional image processing filters, machine learning algorithms, statistical distributions and much much more. Numerous examples of what Accord.NET Framework is capable of are available here.
AForge.NET was developed to run on the .NET Framework and uses the System.Drawing assembly as basis for image manipulation and processing. In particular the dependency on System.Drawing heavily limits the applicability of AForge.NET (and subsequently, Accord.NET) to more modern platforms, like WPF, Windows (for tablet and desktop, as opposed to desktop-only applications), and Windows Phone.
It seemed like a terrible waste that these frameworks would only be available to the outdated range of Windows Forms applications. So, to enable the invaluable functionality of AForge.NET and Accord.NET for modern Microsoft platforms, I started the project of porting these frameworks into Portable Class Libraries, PCL. These efforts are also open sourced and are available on Github: Portable AForge and Portable Accord.
It should be noted that Portable AForge and Portable Accord cover a subset of the assemblies in the pure .NET based frameworks. Video and robotics support is not included, and audio support is limited. Windows Forms UI controls have not been ported. Nevertheless, practically all image processing and scientific computing functionality is included in the portable frameworks.
To facilitate continuous inclusion of updated code for the two frameworks, I have striven to keep the original AForge.NET and Accord.NET code bases as untouched as realistically possible. To cover up for those parts of the framework codes that are not applicable in the portable class libraries, I have therefore developed two support assemblies called Shim and Shim.Drawing.
Shim, which contains simple implementations of non-UI types and methods unavailable in the PCL profiles, is maintained in a separate Github repository, since this assembly is also immediately applicable as a legacy code bridge in other modern target adaptations, such as the port of the fo-dicom library to the (modern) Windows and Windows Phone platforms.
Currently, Portable AForge and Portable Accord are available for the following target platforms:
- .NET Framework 4.5 and higher (for WPF applications)
- Windows 8 and higher (modern tablet/desktop applications)
- Windows Phone Silverlight 8 and higher
- Universal applications, i.e. Windows Phone 8.1 + Windows 8.1
Yes, you read it right! My most recent developments also incorporate the ability to target the Xamarin.iOS and Xamarin.Android platforms. The Xamarin assemblies are however not available as open source. If you are interested in developing AForge and Accord based applications for Xamarin.iOS and/or Xamarin.Android, please contact me directly at ...
And now, with AForge and Accord available on the three mobile platforms, cross-platform development using Xamarin.Forms has also become a reality! This means that it is practically possible to contain an entire C# code base in a PCL core library that uses AForge and Accord, and consume this core library from end applications for all three mobile platforms, Windows Phone, Xamarin.iOS and Xamarin.Android.
Just to show what can now be very quickly done, I copied ...
- Latest version: GitHub (demos)...
- NuGet packages: Imaging.GenericImage
- Help: Off-line - Imaging.* - unblock after download!
The .NET and C# computer vision libraries: AForge.NET, Accord.NET and EmguCV do not provide the truly portable image format.
UnmanagedImageis coupled with
System.Drawing.Bitmapand does not provide generic interface which is useful when dealing with image-processing algorithms which receive various color and channel depth types. EmguCV generic image provides the generic interface but is heavily coupled with OpenCV and does not provide non-generic interface which has basic data manipulation methods necessary for many algorithms. All mentioned image types do not provide a unified interoperability needed when mixing image-processing algorithms from different libraries.
The Accord.NET Extensions Framework (
Imaging.GenericImage) provides the slim generic image class and basic extensions (data handling, arithmetics). It provides multiple color spaces and conversions between them. The class can also be used in non-generic way for developers who prefer AForge's
UnmanagedImagestyle. To get compatibility for other image types just reference the appropriate extension assembly -
Generic image class is implemented as slim as possible thus making the library small, independent, and easily understabale. All other methods and many algorithms are implemented as extension methods which are part of other extension assemblies. Generic image implements the
IImagenon-generic interface which enables passing generic image instance where generic type is not possible or not desirable. The interface itself contains numerous extensions so even the unmanaged representation is quite useful. The base class
Imageis non-generic image representation capable of creating generic image from
TDepthtype where generic parameters are not possible. All other methods, including most Accord.NET image algorithms are done as extensions, so all you have to do is to include the right assembly :)" align="top"> .
The library has several benefits:
- provides the familiar EmguCV's generic image interface
The class signature is Image<TColor, Depth> where TColor is color type and TDepth is channel depth type. The serialized image format is also compatible with EmguCV generic image.
- is sleek and platform independent
The library only contains image representation and basic data handling functions including color conversion engine which are platform abstract.
- provides the non-generic interface
When needed, or if the user prefers, the generic image representation exposes non-generic interface
IImagewhich provides basic data handling functionalities.
- built-in extensible color conversion engine
The built-in color conversion engine based on graph-search will automatically chain color and depth conversion even if immediate color and depth conversion do not exist.
Color conversion. Color conversion engine uses directed graph where nodes are color, channel depth pairs and edges contain conversion function along with conversion complexity weight.
For example assume that the following conversions exist:
(Gray, Single) => (Bgr, Single) and (Bgr, Single) => (Hsv, Single) color conversions and Single => Double depth conversion.
If an user requests (Gray, Single) => (Hsv, Double) a most inexpensive conversion in terms of color conversion complexity is going to be selected, here:
(Gray, Single) => (Bgr, Single) => (Hsv, Single) => (Hsv, Double).
- type-safe generic extensions
The generic-image itself does not contain any methods which converts the image representations. This functionality is provided through extensions. Generic image is the foundation for all other image-processing algorithms and cross-library interoperability.
This article presents the portable generic image library which offers interoperability with image formats from other libraries: EmguCV, OpenCV, AForge.NET, Accord.NET and with the standard .NET types (Bitmap, BitmapData, 2D and 3D array). The code contains complete source as well as commented samples. The source and sample code are the part of Accord.NET Extensions Framework, a framework that brings many advanced algorithms primarily for image processing, object detection and tracking, all packed as fluent extensions and simple and intuitive generics, so do not forget to take a peek
- Latest version: GitHub (demos)...
- NuGet packages: Video.IO
- Help: Off-line - Video.IO - unblock after download!
Current image-processing and computer-vision libraries such as AForge.NET, Accord.NET or EmguCV bring many algorithms but they lack unified API for video, camera and image directory reading and writing. The AForge.NET and Accord.NET libraries relay on DirectX which is not portable and the EmguCV merely wraps the OpenCV file and video capture but does not provide the unified API.
The Accord.NET Extensions Framework provides unified API for IO video access: web-camera support, various video-format reading / writing, image-directory reader. All operations are stream-like and are abstracted therefore do not depend on actual video source. The library is made in platform-abstract fashion. If you have been using
System.IO.Streamclass then you will certain be familiar with the Accord Extensions Vision.IO library.
The figure below shows the unified architecture for video IO reading and writing, camera capture and directory reading.
The base class that all other classes inherit is
TImageis the generic image type. This class contains the base stream properties, but you probably will not be using it directly. The more interesting classes, which you are going to use much more often are:
ImageStreamWriter<TImage>. Those classes have very nice properties:
- the class members use the standard names for stream-like classes
By using familiar names developer does not have to learn anything new.
- they are enough for all basic video IO operations
if you just want shared IO operations which do not include specific type members the
ImageStreamWriter<TImage>classes are enough for all video IO operations. The provided abstraction layer enables you to write programs that do not handle video stream source types separately which makes the code clean and easy maintainable.
Now, here are some concrete samples!
Although the implementation uses the unmanaged libraries the provided NuGet package is platform abstract - for now (x86 /x64 - Windows). Depending on the running OS architecture the appropriate unmanaged library is loaded so the developer does not need to handle multiple OS architectures separately. The image below shows the pre-compiled library locations.
This article presents the portable video IO library which offers unified stream-like interface for file and camera capture and image directory reading. The library also enables video writing. The code contains complete source as well as samples with comments. The source and sample code are the part of Accord.NET Extensions Framework, a framework that brings many advanced algorithms primarily for image processing, object detection and tracking, all packed as fluent extensions and simple and intuitive generics, so do not forget to take a peek