But what if you don't have a Kinect? Or use of your hands? Or have limited movement or space?

You all know I have this thing for Emotiv. So lets put all this together.

What if you could...

Use your brain as mouse control (with EPOC by Emotiv)

Can you imagine that there is an easier way to control the mouse, than with your head? The answer could be: With your mind.

Electroencephalography (EEG) was already established in 1924. In the past it was used mainly for medical reasons. But now you can also use this technology, to control your computer.

Emotiv EPOC

Emotiv has developed a Neuroheadset called EPOC. It reads your brainwaves and on this ground it is able to detect your current feelings like boredom, frustration, meditation and excitement. Your mind sends also certain signal patterns when your face is showing facial expressions. So EPOC can detect facial expressions like smiling, lowering and raising your eyebrows, clenching your teeths or blinking your eyes. This is quite similar to the facial expression detection, which is provided by Kinect for Windows SDK. Furthermore EPOC has a Gyroscope, which recognizes your head movement very accurate. So we developed a version of FaceMouse especially for EPOC.

FaceMouse for EPOC

You can replace your mouse with Emotiv EPOC, the revolutionary brain computer interface. Within Emotiv Control Panel there is already a Mouse Emulator which uses the EPOC Gyroscope to control the cursor of your mouse. EmoKey, an app developed by Emotiv, provides basic functionality for clicking by using EPOC, but this method has several disadvantages:

• In EmoKey clicks are sometimes not executed correctly in special kinds of application windows.
• It is not possible to send a scroll command with EmoKey.

FaceMouse for EPOC sends mouseclicks directly to the operating system, independent from the currently active application window. Also it is possible to scroll or drag & drop, which is not possible with EmoKey.

Just activate the Mouse Emulator in Emotiv Control Panel and use this simple App. It is the easiest way to replace your mouse with EPOC.

FaceMouse

This is a small Windows application, which enhances the possibilities to use EPOC as mouse control. Within Emotiv Control Panel you find a Mouse Emulator which uses the EPOC Gyroskop to control the cursor of your mouse. EmoKey provides basic functionality for clicking by using EPOC, but this method has several disadvantages:

In EmoKey clicks are sometimes not executed correctly in special kinds of application windows. Also it is not possible to send a scroll command with EmoKey.

This application sends mouseclicks directly to the operating system, independent from the currently active application window. Also it is possible to scroll or point & click. FaceMouse uses data from Expressiv Suite.

Just activate the Mouse Emulator in Emotiv Control Panel and use this simple App. It is the easiest way to replace your mouse with EPOC.

Mouse control with your face, head and mind... That's just cool

New course: Building a Real Application with Kinect

John Sonmez has published a new course: Building a Real Application with Kinect

Have you played with Microsoft Kinect? Do you think it’s legit? Wouldn’t it be awesome if you could write your own Kinect enabled applications to take advantage of this awesome piece of technology? Well, I have great news… John has written a course so you can 

In this course he will walk you through the creation of a real application using the Kinect. You’ll be building a Fruit Ninja clone called Shape Ninja which will be capable of detecting chopping gestures and responding to audio commands. You will start off by learning a little bit about the Kinect itself and the Kinect SDK. Then, John shows you just how easy it is to get color image and depth data from the Kinect. You’ll make your application be able to detect and respond to a chopping gesture. He wraps things up by teaching you how to use the Microsoft Speech Platform SDK in combination with the Kinect’s audio sensors to implement real voice commands for our application.

If you’ve been waiting to check out the Kinect, but you didn’t know where to start, or perhaps you thought it would be difficult to learn; this course will get you up and running with the Kinect in no time. Click here to get started now. We hope you enjoy the course!

Project Information URL: http://blog.pluralsight.com/2013/05/01/new-course-building-a-real-application-with-kinect/, http://www.pluralsight.com/training/Courses/TableOfContents/building-real-kinect-application

Contact Information:

• Blog: http://blog.pluralsight.com
• Twitter: @pluralsight

Kinect No Need 4 Green – The Green Screen without a Green Screen

This little piece of software allows you to produce easily, quickly and cheaply the type of picture that you would normally need a proper green screen setup to create. The software uses a Kinect for Windows or Kinect for XBOX360 device connected to a Windows 7 or 8 pc.

All you need to do is stand in front of the device and it will cut you out. You can choose between different backdrops, which you can add to by copying your own pictures (JPEG or PNG) into the PICTURES sub-folder. You can zoom in & out and move the cut out image using the keyboard or by on-screen controls.  You can take snapshot pictures of what is displayed in the window and these pictures are saved into the SNAPSHOTS sub-folder.

The keyboard controls are as below:-
SPACE bar takes a snapshot photo
W,A,S,D keys move the cut-out image left, right, up and down
+ and – keys zoom the cut-out image in and out
UP and DOWN keys allow you to adjust the viewing angle of the Kinect Device
LEFT and RIGHT keys allow you to choose the backdrop picture
M toggles mirroring mode on and off
F11 toggles full screen mode on and off
C toggles depth cut off mode on and off. This mode changes the way Kinect cuts the image out, by cutting out based on the distance from Kinect, rather than trying to cut out individual people. When in this mode the < key and the > key allow you to adjust the cut off distance.

Project Information URL: http://drenton72.wordpress.com/2013/05/10/kinect-no-need-4-green-the-green-screen-without-a-green-screen/

Project Download URL: Click here to download No Need 4 Green version 1

Project Source URL: Click here to download No Need 4 Green version 1 with C# source code

Contact Information:

• Blog: http://drenton72.wordpress.com
• Twitter: @drenton72

Update: Since the writing of this post David has already released a v2!

Kinect NoNeed4Green v2

I've already made some changes to my NoNeed4Green, so here is version 2. The main addition being the facility to add foreground pictures. The foreground pictures have their own sub-folder called foreground and they use PNG files only, as you need images with transparent backgrounds for it to work. This allows you to put objects in front of the live cut-outs of people as well as having a background behind them. This lets you do things like putting someone behind the desk of the oval office or behind the desk of the BBC newsroom or on the bow of the Titanic. You can also now resize and move all 3 layers. Layer 1 is the background, layer 2 is the live cut-outs of people, while layer 3 are the foreground objects.

...

There is a few new keyboard controls as well :-

H toggles between hiding all on-screen buttons and revealing them. Keys 1,2,3 select layers 1,2 and 3 to allow you to move and resize them.

...

Project Information URL: http://drenton72.wordpress.com/2013/05/12/kinect-noneed4green-v2/

Project Download URL: Click here to download No Need 4 Green version 2

Project Source URL: Click here to download No Need 4 Green version 2

What?

NUIverse

NUIverse is an application, written to give me a sandbox for exploring “Natural User Interaction” with a large, multi-dimensional dataset. This build for the Samsung SUR40 with Microsoft PixelSense demonstrates multi-touch gestures, multi-user and multi-directional UI, and object-interaction with transparent Byte Tags. This site provides brief instructions for installing, using, and configuring NUIverse.

NUIverse is not intented as an accurate simulation of the solar system and surrounding universe, since it was written in my spare time and the focus was on interaction rather than accuracy. There are many existing applications available for this purpose, such as Microsoft WorldWide Telescope, Celestia, Space Engine etc.

The following video shows the application demonstrated on a Samsung SUR40 with Microsoft PixelSense at WPC 2012.

NUIverse for Windows

As previously discussed, my manipulation processor now supports WM_TOUCH messages, which means that I can do native multitouch on both Windows 7 and Windows 8. I have therefore updated NUIverse for a Windows-release, as shown below in Figure 1.

There are some key differences to the PixelSense-release, as follows:

• No support for tagged objects, since it does not use the Surface 2.0 runtime, nor require PixelSense hardware (though it will run on the latter outside of the Surface Shell).
• Since horizontal form-factor multitouch hardware is generally less-common than vertical form-factors, I have added a single-orientation configuration setting. This is true by default, since even if mounted horizontally, many touchscreens will not deliver the multitouch performance required for simultaneous multi-user interaction.
• Since the Surface Shell added chrome to close the application and by default the application runs full-screen, either drag in a menu control (see note below) and use the exit menu, or press ESC if a keyboard is present.

One of the current issues when running a full-screen desktop app on Windows 8 is that the operating system captures initial touches used for edge-swipes. If touch is maintained after an initial edge-swipe, further edge-swipes are not captured and therefore will add NUIverse controls to the screen. An alternative is to touch the screen and simultaneously edge-swipe, or to use two fingers when edge-swiping.

Several key configuration settings (in NUIverse.exe.config) are worth mentioning. Note that there is no graphical interface for these settings, and that the configuration file needs to be edited by hand (I would recommend saving a copy first):

• PixelWidth and PixelHeight control the resolution used for both windowed and full-screen mode.
• FullScreen controls whether the application runs full-screen (true) or windowed (false).
• For the configuration settings specified in mm to work correctly, set PixelsPerMm to the appropriate value, taking account physical screen size and either PixelWidth or PixelHeight (square pixels are assumed).

To install NUIverse for Windows, proceed as folows:

1. Install the XNA 4.0 runtime.
2. Download and extract NUIverse for Windows (2.75Mb) to a suitable location.
3. Low-resolution textures for several planets and moons are included, but extras can be created or downloaded from http://nuiverse.com.

Here's a snap of the app running on my Windows 8 notebook;

I said there was extensibility?

Extensibility Model Part 2

I previously mentioned that I had implemented an extensibility model, and thought it useful to discuss an example of adding a simple model to earth orbit, as shown below in Figure 1 (further images of which are in the gallery).

Figure 1. Model added to earth orbit. Colonial Raptor model (based on new TV series) by Coxxon.

The "extra" is defined as a folder containing the following items:

• A model in XNB format. XNA has built-in content importers for .x and .fbx (2009.1) formats.
• An optional pair of textures for both diffuse and emissive textures. These are standard image files.
• An XML file defining the "extra", in this case as shown below in Listing 1.

This configuration file specifies that the model should be added to the planetoid "earth" in the "solar" system, both of which are defined in system.xml configuration file.

In order to scale the model correctly, a scale factor is applied to normalize the model to unit length. This can either be applied in the XML scale attribute, or specified in the XNA content processor scale attribute, in which case the XML attribute can be set to 1. A size attribute then defines the maximum length of the model in km. The Colonial Raptor shown in Figure 1 was defined with a size of 8.6m.

The textures are defined in sub-folders "texture" and "emissive". If an emissive texture is not available, an all-black image (e.g. JPEG file) can be used.

The rotation period defines how long it takes for the model to rotate while orbiting the planetoid. If this is the same as the P orbital element, then the same face of the model is presented to the planetoid throughout the orbit. The remaining standard orbital elements specify that the model is in a circular equatorial orbit at an altitude of 500km (the earth has a radius of 6,371km).

Make sure you read through the rest of his NUIverse, and other, articles as they are very interesting and informative... (like his holographic Princes Leia recording post... that's just cool!)

Lose your Occlusion

Depth-sensing cameras like the Kinect give us the opportunity to mix physical environments and virtual environments, creating new immersive experiences. In this Second Story Labs experiment, we demonstrate how the use of multiple cameras helps solve problems with occlusion or “holes” the use of a single camera creates.

If you’ve ever worked with a Microsoft Kinect, you know that occlusion is no laughing matter. It’s responsible for gaping holes in people’s chests, disappearing necks, the noseless faces of zombies. Humans are full of convexities, and to a Kinect that means that we are also full of holes. We’ve dealt with this in lots of creative ways—filling in gaps with best estimates, “blurring” data, hiding holes with smoke and mirrors. But the best and perhaps most obvious solution to the dilemma of occlusion is simply to add more Kinects. A second gunman, if you will, shooting from an angle that will cover the first Kinect’s blind spot.

In this case, we are calibrating two Kinects in space about a meter apart and angled inward toward their subject. This way we can “see” our way around noses, arms, and other pesky occlusions. Then all we have to do is combine their data into a single mesh, and the rest is up to our imaginations.

At full resolution, we can actually get a pretty accurate model of a person’s face. And all of this is being rendered in real time, so that a user’s reality can be “augmented” while they interact. Here we have added three-dimensional models and rain particles to the virtual space to put the user into an imaginary landscape.

Project Information URL: http://blog.secondstory.com/2013/03/18/lose-your-occlusion-2/

from Vimeo.

Contact Information:

• Blog: http://blog.secondstory.com/
• Twitter: @2storypdx

Legacy software written for Windows 7 and prior targeted the traditional Windows API, often referred to as Win32. They resulting apps would be a combination of an executable, assets and libraries. The Windows 8 operating system introduces a new type of application, called a Windows Store app. The new architecture is called Windows Runtime, or WinRT for short. Windows Store apps are written exclusively against this new WinRT architecture.

Windows 7 and prior only had one mode of interacting with software, launching the application's executable from the desktop or file path. Windows 8 now defaults to a Windows Runtime start screen, where Windows Store apps are presented and can be accessed through their respective Live Tiles. An application's Live Tile is registered automatically when that app is installed from the Windows Store. If a user needs to access legacy software that hasn't been migrated to the new Windows Store format, they can still get to the desktop from the Start Screen.

Windows desktops and laptops have traditionally run on Intel processors, referred to frequently as x86 or x64 machines. However, in parallel with the release of Windows 8, an optimized version of the operating system designed for mobile ARM based devices was released called Windows RT. Windows RT devices, however, will not run legacy Win32 software, nor can you develop Win32 software for content.

So unlike a Windows 8 computer, which is optimized for the Windows Store apps written against WinRT but still provides access to Win32 software from the desktop, Windows RT devices do not.

What about the Microsoft Surface?

There are two versions of the Microsoft Surface. The first version runs the Windows RT operating system and uses an ARM CPU. The Surface Pro, however, is an x64 device and runs Windows Pro 8. Both devices are optimized for Windows Store apps, but the Surface Pro is a fully featured computer as well, allowing you to install Visual Studio, Photoshop, or whatever else you wish. You can think of the Microsoft Surface as a light, long lasting consumer consumption device, whereas the Surface Pro is a powerful, mobile content creation device and netbook equivalent.

What's up with Cinder and DirectX?

Cinder is a cross platform framework for C++ design engineering. In order to run and Apple, Android and Windows devices prior to Windows 8, the rendering libraries leveraged OpenGL as the open source, cross platform solution.

Even though DirectX has been Microsoft's preferred rendering solution for advanced graphics programming, Windows traditionally provided both DirectX and OpenGL support from the desktop. However, Windows 8 Store applications are now exclusively DirectX based, to maximize reusability across the entire domain of Microsoft experiences including Windows Phone, Windows RT, Windows 8, and gaming. This means that Cinder apps that leverage OpenGL for their rendering will not work as Windows Store apps.

To get a Cinder project running as a Windows Store app, the Cinder project needs to leverage a DirectX render instead of the default OpenGL renderer.

Both OpenGL and DirectX have undergone significant changes over the years, with DirectX migrating away from the simpler fixed function pipeline of OpenGL early on. As hardware and software continue to advance, both libraries continue to progress in their own distinct ways. Windows 8 Store apps leverage the most recent version of DirectX 11.1, which itself has some breaking changes from DirectX 11, most notably is the loss of D3DX, a very popular helper library.

As this document targets using Cinder for Windows Store apps, we will be using DirectX 11.1. Whereas you could have a DirectX renderer for desktop based Cinder experiences, because of the history and community support behind the existing OpenGL render, it would be a difficult choice to argue. However, to create a Windows Store app, DirectX 11.1 must be used as the underlying rendering technology in the Cinder framework.

What about Visual Studio?

Previous versions of Cinder were written to make it easy to use for Visual Studio 2010, and targeted Win32 applications. Since Windows Store apps require targeting the WinRT architecture, this has changed.

To develop software for Window Store 8, you need to use Visual Studio 2012. We will be walking through the setup assuming you have downloaded the free version of the software, Visual Studio Express 2012.

You should know, however ,that the professional versions of Visual Studio 2012 comes with several new features for professional graphics engineering, including a robust graphics debugger, DirectX support, visual shader editors, and tools for working with virtual models and objects.

Enough theory, lets get started

To create Windows Store apps, you need to have Windows 8 installed and running at least Visual Studio Express 2012 for Windows 8. Do not download Visual Studio Express 2012 for Web, Phone or Desktop. We are targeting Windows 8 here.

Setting up Visual Studio Express 2012 for Windows 8

Once you have a copy of Windows 8 running natively or on a virtual machine from another Operating System, you can get the bytes for Visual Express 2012 for Windows here:

http://www.microsoft.com/visualstudio/eng/products/visual-studio-express-products

When the installer has finished and you launch the application, you will be taken to a registration screen. If you choose not to register, Visual Studio Express 2012 will expire after 30 days. Go ahead and register the product.

Once registration is complete, Visual Studio will launch for the first time. In order to create a Windows Store app, you need to setup a Developer license. A developer license allows you to develop, install and test apps on your computer before submitting them to the Microsoft store for publishing and deployment to the community.

At this time, it would also be a good idea to go to the menu item Tools > Extensions and Updates and check for any updates to Visual Studio as well:

Grabbing the Git branch

Now that we have Windows 8 and Visual Studio Express 2012 setup, our next step is getting the correct Cinder bits on our machine. Since this is still under development, we need to pull from the correct developer branch of Cinder from Cinder's GitHub account.

The professional version of Visual Studio 2012 allows for rich Git integration, but the free Express version doesn't support this, so you will need to have a standalone Git client if you choose to use the Express version of Visual Studio. If so, we recommend downloading the GitHub client, which installs both itself and helpful Git utilities.

You can download the GitHub client here:

https://help.github.com/articles/set-up-git

After the GitHub client is installed, you may find it helpful to enter or setup your account. However, you should be able to pull the project if you just want to get up and running quickly using the Git Shell as explained further below.

As a community developed, open source project, Cinder separates the stable, official version of its Cinder framework from newer branches that developers setup as they program future functionality. WinRT support for Cinder is currently being developed off of one of these feature branches, with the goal of integrating in to a future iteration of the master project.

So we will need to clone the Cinder project locally, which defaults to the master branch, and then explicitly move to the dx_rt branch to get to the version under development that also includes support for Windows Store 8 projects.

To clone Cinder, either open your Git enabled Power Shell in the directory you want to use and type:

git clone https://github.com/cinder/Cinder.git

Or open a browser and go to the Cinder project page on GitHub.com. There you will see a Clone In Windows button that should launch the GitHub client automatically and start the local cloning process:

https://github.com/cinder/Cinder/tree/dx_rt

This should take several minutes to download the Cinder project and open it in your GitHub client. Once it appears as a line item in your GitHub client, you can right click the repository icon and select open a shell here. We will do the rest of the commands from the Power Shell directly.

If you look at your command prompt and you are still on the master branch, you can switch to the dx_rt branch by typing the following:

git checkout dx_rt

Or if you prefer to use the GitHub client you installed, you can switch from the master branch to the dx_rt branch from there as well.

If you are using the GitHub client, though, it would probably be a good idea to switch to a command shell for the next steps. You can do that from the client directly by right clicking on project's list item like so:

The master, dev and dx_rt branches all require you to also install certain Boost libraries to the Cinder project directory before you can compile the Cinder framework. However, on the dx_rt branch we will be using, this process has been greatly automated for you. So once you've switched to the dx_rt branch, type in the following two commands to download the required Boost libraries that are compatible with this particular branch of Cinder for Windows Store:

git submodule init

git submodule update

This may take a few minutes, but once it is done you should be ready to open the Cinder project and compile it for use.

Compiling Cinder for Windows Store apps

Once you have pulled the Cinder project from its GitHub repository, switched to the dx_rt branch that contains the bits for working on Windows Store apps, and run the submodules that download the additional Boost dependencies, you are ready to open up the Cinder solution to compile the library.

Go to where GitHub cloned the Cinder project, the default is Libraries\Documents\GitHub\Cinder. There you will find multiple version of the Cinder framework separated by development environment. You should see the xcode and vc10 solutions, but you will also three additional folders. Vc11 provides support for compiling Cinder for Visual Studio 2012, but still using OpenGL and thus limited to Desktop mode. VC11-DirectX is provides support for compiling Cinder for Visual Studio 2012 using DirectX 11 but also for Desktop mode. We will be focusing on the third option, the winrt folder. This is the solution that supports Windows Store apps and development.

Launch Cinder/winrt/cinder.sln and the Cinder project should open up in Visual Studio 2012 automatically. If you are using the professional version of Visual Studio 2012, you can Batch Build all the versions you need. Since we are using the Express edition in this walkthrough, though, we will Build individually the debug and release versions of the Cinder library we need.

We will build the debug version of the framework by making sure that our Solution Configuration in the toolbar is set to Debug and the Solution Platform is set to Win32.

To build the solution, simply go to the Build menu option and select Build Solution.

Some warnings may pop up, but there should be no errors. Once that is done, you will find a new file in the Cinder project directory under:

..\Cinder\lib\winrt\Win32\Debug\cinder.lib

This is the Cinder library your Windows Store projects will link to when you are programming and debugging your application. Now we need to rebuild this framework to provide a release version of this same framework. Going back to the Cinder project, and on the toolbar change the Solution Configuration option from Debug to Release and Build the solution again.

You now have a Debug and Release version of the Cinder framework and are ready to start creating Windows Store programs against it for any Windows 8 device, including the Surface Pro. However, any WindowsRT device, such as the non-Pro version of Surface, will be running on ARM and so you would need to follow the above steps targeting the ARM platform instead of just the Win32 platform.

Testing out the Library

At this point, you are ready to explore the collection of DirectX samples created specifically for Windows Store apps. You can find various projects for Windows Store in the DirectX folder inside the samples folder. For our initial test, we will open the following sample:

..\Cinder\samples\DirectX\_winrtBasic\winrt\winrtBasic.sln

When you open the solution in Visual Studio Express 2012, you will find basicApp.cpp in the Solution Explorer. This contains all the code we need to start using the Cinder framework.

The code in this sample illustrates a few basic tasks, including loading an image from the assets folder and using it as a texture, loading fonts and displaying them on the screen, handling mouse events, drawing various geometries and implementing a basic follower easing algorithm in the update() call.

Once you're looked over the code, you can press F5 or the Play button on the main toolbar to run the program on the machine. If you compiled both the Debug and Release versions of the framework, you can select either Solution Configuration. You will find a significant increase in speed in Release mode over Debug mode.

You will also notice that the program has automatically installed its own Live Tile on your Start Screen, which you can use to launch the program again whenever you wish.

 Congratulations! You should now be ready to start exploring Cinder for Windows Store apps.

@rickbarraza

You can learn more about this branch by going to the Cinder forums found here:

https://forum.libcinder.org/#Topic/23286000001540037

The system has two core components:

• A server application, which is a Win32 console application that handles incoming queries and returns data to the connected clients. This application runs on the desktop machine with the connected sensor.
• The Windows Phone application that sends requests to the target server and can trigger a variety of pre-programmed commands.

The Basics

Hardware and Communication Infrastructure

One of the most important parts of the project is the signal capture and replication hardware. For the purposes of this project, I decided to use a dual-mode Phidgets IR sensor. It supports both IR code capture and subsequent replication. From a user’s perspective, this device also eliminates a substantial code-learning overhead as well as the potential error rate. Instead of searching for a device-specific hexadecimal sequence that later has to be transformed in a working IR code, the user simply has to point his remote control at the sensor and press the button that he wants accessible from a mobile device. Given that the capturing software is running on the target machine, once the sensor detects that a code can be repeated within an acceptable precision range, it will be automatically captured and stored, with all required transformations worked out in the backend using the free Phidgets SDK.

Even though I can, I don’t have to handle the binary code content received through the sensor—the Phidgets .NET libraries carry built-in types that contain all the processed metadata that I will discuss later in this article.

This sensor is connected through a USB port to a machine that acts as a communication gateway. This server should have port 6169 open for inbound connections.

NOTE: The port number can be changed, but you have to keep it consistent between your server and client applications.

The communication between the phone and the computer running the client is performed via a TCP channel—sockets are used to perform the initial connections and serialized data transfer. You can see the generalized data flow between the devices that are involved in the procedure in the graphic below:

The server (desktop client) handles the local storage and release of all incoming IR codes. The mobile client has to know the location of the server—once specified and confirmed, it can send one of the pre-defined commands to it and either query the server for existing command groups (sets) or invoke one of the stored IR codes. When I pass data between devices, I use JSON for the serializable components. The data is also processed before being sent in order to speed-up the process—for example, on the server side the sets are serialized together with the associated codes. Like this:

[
{
    "Name":"batman",
    "IsList":false,
    "Commands":[
    {
        "Name":"test command",
        "Code":{
            "Mask":{
                "BitSize":12,
                "CodeData":"AAA="
            },
            "BitSize":12,
            "Encoding":2,
            "CarrierFrequency":38000,
            "DutyCycle":50,
            "Gap":44761,
            "Header":[
            2374,
            606
            ],
            "CodeData":"DJA=",
            "MinRepeat":5,
            "One":[
            1189,
            606
            ],
            "Repeat":null,
            "Trail":0,
            "Zero":[
            582,
            606
            ]
        }
    },
    {
        "Name":"turn off",
        "Code":{
            "Mask":{
                "BitSize":12,
                "CodeData":"AAA="
            },
            "BitSize":12,
            "Encoding":2,
            "CarrierFrequency":38000,
            "DutyCycle":50,
            "Gap":44770,
            "Header":[
            2360,
            613
            ],
            "CodeData":"DJA=",
            "MinRepeat":5,
            "One":[
            1169,
            613
            ],
            "Repeat":null,
            "Trail":0,
            "Zero":[
            585,
            613
            ]
        }
    }
    ]
}
]

The inherent problem with the JSON data above is the fact that the phone client does not need the information related to the code binary sequence and all the metadata that goes with it. So it is effectively stripped down and reduced to the names of the sets (when a list of sets is requested) and commands (when a list of commands is requested).

The Data Model

As you saw from the description above, the server organizes individual infrared codes in sets. A single set is a bundle of codes that may or may not be related to each other—ultimately, this is the user’s decision. A good example of using sets is organizing IR commands by rooms, devices or code types. Each set has a unique name on the server, therefore eliminating the possibility of a request conflict.

Each set stores individual commands built around the Command model:

namespace Coding4Fun.MissionControl.API.Models 
{ 
    public class Command 
    { 
        public Command() 
        { 
        } 
    
    public string Name { get; set; }     
    public SerializableIRCode Code { get; set; } 
    } 
}

Despite the obvious Name property, you can see that I am using a SerializableIRCode instance that is specific to each model. Before going any further, I need to mention that the Phidgets SDK offers the IRLearnedCode model to store code contents. I could have used it instead, but there is an issue that prevents me from doing that—there is no public constructor defined for IRLearnedCode, therefore there is no way to serialize it, either with the built-in .NET serialization capabilities or JSON.NET, which I am using in the context of the project.

Instead, I have this:

using Phidgets; 
namespace Coding4Fun.MissionControl.API.Models 
{     
    public class SerializableIRCode     
    {         
        public SerializableIRCode()         
        { 
            
        } 
        
    IRLearnedCode code;         
    public ToggleMask Mask { get; set; } 
    public int BitSize { get; set; } 
    public Phidgets.IRCodeInfo.IREncoding Encoding { get; set; } 
    public int CarrierFrequency { get; set; } 
    public int DutyCycle { get; set; } 
    public int Gap { get; set; } 
    public int[] Header { get; set; } 
    public byte[] CodeData { get; set; } 
    public int MinRepeat { get; set; } 
    public int[] One { get; set; } 
    public int[] Repeat { get; set; } 
    public int Trail { get; set; } 
    public int[] Zero { get; set; } 
    }     
}

It is an almost identical 1:1 copy of the original class, storing both the layout of the IR code and additional information related to its replication mechanism. You can learn more about each property listed in the model above by reading the official document on the topic.

ToggleMask, the identity bit carrier that helps marking the code as repeated or not, is also implemented through a built-in Phidgets SDK model, and it has the same problem as IRLearnedCode. I implemented this model to replace it in the serializable code:

namespace Coding4Fun.MissionControl.API.Models 
{ 
    public class ToggleMask 
    { 
        public ToggleMask() 
        { 
            
        } 

    public int BitSize { get; set; }         
    public byte[] CodeData { get; set; } 
    } 
}

I also needed an easy way to store all sets at once and carry all associated codes in a single instance retrieved from the storage. Here is the Set class:

namespace Coding4Fun.MissionControl.API.Models 
{     
    public class Set 
    { 
        public Set() 
        { 
            Commands = new List<Command>(); 
        } 
        
    public string Name { get; set; } 
    public bool IsList { get; set; } 
    public List<Command> Commands { get; set; } 
    } 
}

Notice that there is an IsList flag that allows me to specify how to display this specific list on the connecting device. This adds some level of flexibility for situations where the user wants to build a virtual remote for closely-related keys, such as digits. With that in mind, displaying those as a list might be inconvenient, wasting visual space on the client. But if the flag is set to false, the list can be displayed as a pad.

Also, when the server performs the data exchange, it provides a single “envelope” that allows the connecting device to easily understand what the server is trying to do:

namespace Coding4Fun.MissionControl.API.Models 
{     
    public class ServerResponse     
    {     
    public string Identifier { get; set; }     
    public string Marker { get; set; }     
    public string Content { get; set; }     
    } 
}

The Identifier property carries the server IP address. That way, when a device receives a response, it is able to either accept it, because it knows that a response is requested from a target location, or discard it because the user is no longer using the specific server.

Marker carries the command type of the sent command, therefore giving the Windows Phone application a hint as to what to do with the data. The server can send the following commands:

• SET_LIST – returns the list of sets that are currently available on the server.
• SET_COMMANDS:SET_NAME:IS_LIST – returns the list of commands that are associated with a given set that is currently stored on the server.
• NOTIFICATION – send a simple notification to the client; no further action is required.

Last but not least, Content is used to push the necessary data that is associated with the given Marker. It can be either a JSON-based string that lists the sets or commands, or a plain-text message that is used as an alert for the end-user.

Server Architecture

The server is the only component of this entire system that does all the heavy lifting. It learns commands, stores them and then generates new IR signal requests, as controlled from any of the connected clients. Let’s take a closer look at what happens behind the scenes—to start, I am going to document the network infrastructure.

The Network Layer

In order to be a reliable system, the server needs to be always ready to accept an incoming connection. For that purpose, it is possible to use the TcpListener class—an “always on” receiver that can handle incoming TCP connections. I integrated it in my CoreStarter class that is used to start the listener when the application is launched:

namespace Coding4Fun.MissionControl.API 
{ 
    public class CoreStarter 
    { 
        static TcpListener listener; 
        
        public static void LaunchSocket()     
        {     
            Console.WriteLine("Starting socket server on port {0}...", Constants.DEFAULT_PORT); 
            listener = new TcpListener(NetworkHelper.GetLocalIPAddress(), Constants.DEFAULT_PORT); 
            listener.Start(); 
            
            for (int i = 0; i < Constants.MAX_CONCURRENT_CLIENTS; i++) 
            {     
                Thread socketThread = new Thread(new ThreadStart(ListenForData));     
                socketThread.Start(); 
            } 
        } 
        
        private static void ListenForData() 
        { 
            Console.WriteLine("Listener thread started."); 

            while (true) 
            { 
                Socket acceptedSocket = listener.AcceptSocket(); 
                using (MemoryStream coreStream = new MemoryStream()) 
                {     
                    try 
                    { 
                        Console.WriteLine("Incoming connection: {0}", acceptedSocket.RemoteEndPoint); 
                        
                        using (Stream sourceStream = new NetworkStream(acceptedSocket)) 
                        {     
                            sourceStream.ReadTimeout = Constants.SOCKET_READ_TIMEOUT; 
                            
                            byte[] buffer = new byte[Constants.DEFAULT_BUFFER_SIZE]; 
                            int i; 
                
                            while ((i = sourceStream.Read(buffer, 0, buffer.Length)) != 0) 
                            { 
                                coreStream.Write(buffer, 0, i); 
                            } 
                        } 
                    } 
                    catch 
                    { 
                        string data = Encoding.ASCII.GetString(coreStream.ToArray()); 
        
                        CommandHelper.InterpretCommand(data, acceptedSocket.RemoteEndPoint.ToString()); 
                    } 
                } 
            } 
        } 
    } 
}

When LaunchSocket is called, the listener is activated on the current machine. As I mentioned above, the port number can be arbitrarily assigned, but has to be consistent between connecting apps in order for the TCP links to be established. Because I expect that more than one device will be connecting to the service at a time, the listener is set as active across a constant number of threads.

NOTE: By default, a there is a maximum limit of 5 simultaneous clients. Although this number can be adjusted, be aware of the requirements of each environment in which a limited number of potential devices can connect. Even though the performance footprint of each thread is minimal, it can have a negative effect if used in unnecessarily large instances.

ListenForData is used to read the incoming stream. When an inbound connection is accepted, the data is read with the help of a fixed content buffer. Then a read timeout is specified to prevent situations where the stream was completely read but the application still waits to pull non-existent data. Once the timeout milestone is hit, an exception is thrown, which marks the end of the stream—at this point, the plain text data that was received (remember that both the server and client exchange text data only) is passed to the command interpreter—CommandHelper, with a reference to the source of the command.

The commands from the device are passed as serialized key-value pairs (KeyValuePair<T, T>), the key being the command with any possible suffixes, and the value being the contents of the command itself that helps the server identify the specific item in the local storage.

InterpretCommand,in this case, does three things sequentially:

1. Deserialize the incoming string and create a KeyValuePair<string,string> instance.
2. Process the command and check whether it is recognizable.
3. Send a response to the client, if deemed necessary by the command type.

The serialization and deserialization is done via JSON.NET. You can install this package in your console managed Win32 project and the Windows Phone application project via NuGet:

The deserialization step is as simple as one line of C# code:

KeyValuePair<string, string> result = JsonConvert.DeserializeObject<KeyValuePair<string, string>>(rawCommand.Remove(0, rawCommand.IndexOf('{')));

The string is sanitized to ensure that only JSON content is being passed to the serializer.

Because of a relatively limited command set, I can put together the entire interpretation stack like this:

// Get the initial list of sets on the target server
if (result.Key == Constants.COMMAND_INIT)
{
    SendSets(sourceLocation);
}
// Create a new set on the target server
else if (result.Key.Contains(Constants.COMMAND_CREATE_SET))
{
    CreateSet(result, sourceLocation);
    SendSets(sourceLocation);
}
// Get the commands that are associated with a given set.
else if (result.Key == Constants.COMMAND_GET_COMMANDS)
{
    SendCommands(result.Value, sourceLocation);
}
// The client requested the server to learn a new command.
else if (result.Key.Contains(Constants.COMMAND_LEARN_COMMAND))
{
    LearnCommand(result, sourceLocation);
}
// The client requested one of the commands to be executed on the 
// target server.
else if (result.Key.Contains(Constants.COMMAND_EXECUTE))
{
    ExecuteCommand(result);
}
// The client has requested a set to be deleted from the target server.
else if (result.Key == Constants.COMMAND_DELETE_SET)
{
    DeleteSet(result.Value);
    SendSets(sourceLocation);
}
// The client has requested a set to be deleted from the target server.
else if (result.Key.Contains(Constants.COMMAND_DELETE_COMMAND))
{
    DeleteCommand(result);
    SendCommands(result.Key.Split(new char[] { ':' })[1], sourceLocation);
}

All commands are constants, declared in the local helper class:

public const string COMMAND_INIT = "INIT"; 

public const string COMMAND_CREATE_SET = "CREATE_SET"; 

public const string COMMAND_GET_COMMANDS = "GET_COMMANDS"; 

public const string COMMAND_LEARN_COMMAND = "LEARN_COMMAND"; 

public const string COMMAND_EXECUTE = "EXECUTE"; 

public const string COMMAND_DELETE_SET = "DELETE_SET";
public const string COMMAND_DELETE_COMMAND = "DELETE_COMMAND";

Notice that these are not the commands that the server sends back, but rather the commands it receives from connecting Windows Phone devices.

Let’s now take a look at the breakdown for each command.

SendSets:

/// <summary> 
/// Send the list of sets to the client that requested those. 
/// </summary> 
/// <param name="sourceLocation">The location of the requesting client.</param> 
private static void SendSets(string sourceLocation) 
{ 
    Console.WriteLine("Received an initial set query from {0}", sourceLocation);     
    ServerResponse response = new ServerResponse(); 
    response.Marker = "SET_LIST"; 
    response.Content = JsonConvert.SerializeObject(StorageHelper.GetRawSetNames()); 
    response.Identifier = NetworkHelper.GetLocalIPAddress().ToString(); 
    NetworkHelper.SendData(sourceLocation, JsonConvert.SerializeObject(response)); 
    Console.WriteLine("Sent the set list to {0}", sourceLocation);     
}

When this command is received, the server does not have to do much processing. It is only invoked when the client establishes the initiating link and needs to know what possible sets it can get from the target machine. The request is logged in the console and a server response is prepared that contains a serialized list of set names, which is later serialized as well and sent back to the source machine location.

StorageHelper and NetworkHelper will be documented later in this article.

CreateSet:

/// <summary> 
/// Create a new set and store it on the local server. 
/// </summary> 
/// <param name="result">The original deserialized command.</param> 
/// <param name="sourceLocation">The location of the requesting client.</param> 
private static void CreateSet(KeyValuePair<string,string> result, string sourceLocation) 
{ 
    bool isSuccessful = false; 
    string[] data = result.Key.Split(new char[] { ':' }); 
    
    Console.WriteLine("There is an attempt to create the {0} set from {1}.", result.Value, sourceLocation); 
    
    if (data[1].ToLower() == "list") 
        isSuccessful = StorageHelper.AddSet(result.Value); 
    else 
        isSuccessful = StorageHelper.AddSet(result.Value, false); 
    
    if (isSuccessful) 
        Console.WriteLine("The {0} set was successfully created.", result.Value); 
    else 
        Console.WriteLine("Something happened and the {0} set was not created.", result.Value);     
}

When a mobile device attempts to create a new set on the server, it sends a command in the following format:

CREATE_SET:list/pad, SET_NAME

CreateSet will get the type of the set that was created, will check whether a set with the same name already exists and will either create it or ignore the command altogether. No notification is sent to the connecting device, but either the failure or the success of the command is registered in the local console.

SendCommands:

/// <summary>
/// Send a list of commands that are associated with the pushed set.
/// </summary>
/// <param name="setName">The original deserialized command.</param>
/// <param name="sourceLocation">The location of the requesting client.</param>
private static void SendCommands(string setName, string sourceLocation)
{
    Console.WriteLine("There was a request to get the commands for the {0} set from {1}.", setName, sourceLocation);
    
    bool isList = StorageHelper.IsSetAList(setName);
    
    ServerResponse response = new ServerResponse();
    response.Marker = string.Format("SET_COMMANDS:{0}:{1}", setName, isList);
    response.Identifier = NetworkHelper.GetLocalIPAddress().ToString();
    response.Content = JsonConvert.SerializeObject(StorageHelper.GetRawCommandNames(setName));
    
    NetworkHelper.SendData(sourceLocation, JsonConvert.SerializeObject(response));
    
    Console.WriteLine("Command list for the {0} set were sent to {1}.", setName, sourceLocation);
}

Commands are sent in the same manner as sets—once the set is recognized, the names of the associated commands are retrieved and serialized inside a ServerResponse instance and then pushed back to the requesting device.

LearnCommand:

/// <summary> 
/// Learn a new command and store it on the target server. 
/// </summary> 
/// <param name="result">The original deserialized command.</param> 
/// <param name="sourceLocation">The location of the requesting client.</param> 
private static void LearnCommand(KeyValuePair<string,string> result, string sourceLocation) 
{     
    Console.WriteLine("[!] Server in COMMAND LEARNING MODE! Point the remote towards the sensor and send a command."); 
    
    string[] data = result.Key.Split(new char[] { ':' });     
    var set = StorageHelper.GetSingleSet(StorageHelper.GetSets(), data[1]); 
    
    if (set != null)     
    {     
        if ((from c in set.Commands where c.Name == result.Value select c).FirstOrDefault() != null) 
        {     
            Console.WriteLine("Cannot learn command {0} for the following set: {1}. Command already exists.", data[1], result.Value); 
            
            ServerResponse response = new ServerResponse();     
            response.Marker = "NOTIFICATION";     
            response.Identifier = NetworkHelper.GetLocalIPAddress().ToString(); 
            response.Content = "We could not save the following command - " + result.Value + ". It already exists in the set."; 
            
            NetworkHelper.SendData(sourceLocation, JsonConvert.SerializeObject(response));     
        } 
        else 
        { 
            if (sensor == null) 
                sensor = new IR(); 
            
            sensor.open(-1); 

            sensor.waitForAttachment(); 
    
            sensor.Learn += (sender, args) => 
            { 
                Console.WriteLine("[!] Server learned the command and is no longer in COMMAND LEARNING MODE."); 
                IRLearnedCode code = args.LearnedCode; 
                code.CodeInfo.MinRepeat = 5; 
            
                Command command = new Command(); 
                command.Name = result.Value; 
                command.Code = IRCodeWorker.GetSerializableIRCode(code); 
                
                StorageHelper.AddCommand(command, set.Name); 
                
                ServerResponse response = new ServerResponse(); 
                response.Marker = "NOTIFICATION"; 
                response.Identifier = NetworkHelper.GetLocalIPAddress().ToString(); 
                response.Content = "The following command has been stored: " + result.Value; 
                
                NetworkHelper.SendData(sourceLocation, JsonConvert.SerializeObject(response)); 
            }; 
        } 
    } 
}

Once a request was received that the server needs to learn a new command, an initial verification is done to make sure that the requested command name and set are not already taken. If neither the command nor the set exist, both will be created.

After the basic setup is complete, the IR sensor is activated and will be waiting for the command to be learned. The way it works is quite simple – the sensor will remain in learning mode until the point where it recognizes a command without error, being 100% sure that it can be reproduced internally. You will need to point your remote towards the sensor and hold the button you want captured for one or two seconds in order for the command to be learned.

NOTE: To ensure that a proper transmission is done, I manually set the minimal repeat value to 5. This is the number of times the sensor will fire the same code towards the target. That is the optimal value for a target device to receive the code if the remote is pointed directly at it without necessarily triggering the same command twice or more.

After the command is learned, the code is processed and transformed into a serializable instance. The connecting client is then notified about whether the command was learned.

ExecuteCommand:

/// <summary> 
/// Execute one of the commands currently stored on the local server. 
/// </summary> 
/// <param name="result">The original deserialized command.</param> 
private static void ExecuteCommand(KeyValuePair<string,string> result) 
{ 
    string[] data = result.Key.Split(new char[] { ':' }); 

    var set = StorageHelper.GetSingleSet(StorageHelper.GetSets(), data[1]); 
    
    if (set != null) 
    {     
        var command = StorageHelper.GetSingleCommand(StorageHelper.GetCommands(set.Name), result.Value); 

        IRLearnedCode code = IRCodeWorker.GetLearnedCode(command.Code); 
        
        if (sensor == null) 
            sensor = new IR(); 

        sensor.open(-1); 
        sensor.waitForAttachment(); 
        sensor.transmit(code.Code, code.CodeInfo); 
        sensor.close(); 
    } 
}

Command execution relies on the hardware sensor. The phone sends a command execution request in the following format:

EXECUTE:SET_NAME, COMMAND_NAME

Once the command is parsed out and found in the local storage, the IR code is transformed back to a model that is recognizable by the Phidgets SDK and transmitted towards the location where the sensor is pointed at the time of the execution.

DeleteSet:

/// <summary> 
/// Delete a single set and all the associated commands 
/// </summary> 
/// <param name="target">The name of the set.</param> 
private static void DeleteSet(string target) 
{ 
    var sets = StorageHelper.GetSets(); 
    var targetSet = StorageHelper.GetSingleSet(sets, target); 
    
    if (targetSet != null) 
    { 
        StorageHelper.RemoveSet(sets, targetSet); 
    } 
}

When deleting a set, only the name of the set should be specified. The user will get a warning on the client side that requires a confirmation of the deletion. The server will blindly execute the command.

DeleteCommand:

private static void DeleteCommand(KeyValuePair<string, string> result)
{
    var sets = StorageHelper.GetSets();
    string setName = result.Key.Split(new char[] {':'})[1];
    var targetSet = StorageHelper.GetSingleSet(sets, setName);
    var command = (from c in targetSet.Commands where c.Name == result.Value select c).FirstOrDefault();
    
    if (command != null)
    {
        targetSet.Commands.Remove(command);
        StorageHelper.SerializeSets(sets);
    }
}

Not only can the user remove entire sets, but he can also target specific commands from a given set. Once a DELETE_COMMAND directive is recognized, the set name is parsed out from the original string, that follows the DELETE_COMMAND:SET_NAME, COMMAND_NAME format, and a simple LINQ query extracts the command instance, removes it and stores the set content on the local hard drive.

Notice that for some commands, particularly for set creation, deletion and command deletion, the server will return a list of the remaining items. The contents will be automatically updated on the devices, which will be waiting for that response. This measure was deliberately introduced to minimize the chances of a user triggering a command that was already deleted or trying to query a previously removed set.

Transforming Codes

You might have noticed that I am using IRCodeWorker.GetSerializableCodeType to transform a Phidgets SDK native IR code model into a serializable one. This is a helper function that performs a field copy of the existing object. Because of the differences in the model structure, it has to be done manually:

public static SerializableIRCode GetSerializableIRCode(IRLearnedCode code) 
{ 
    SerializableIRCode sCode = new SerializableIRCode();     
    sCode.BitSize = code.Code.BitCount;     
    sCode.Encoding = code.CodeInfo.Encoding;     
    sCode.CarrierFrequency = code.CodeInfo.CarrierFrequency;     
    sCode.CodeData = code.Code.Data;     
    sCode.DutyCycle = code.CodeInfo.DutyCycle;     
    sCode.Gap = code.CodeInfo.Gap;     
    sCode.Header = code.CodeInfo.Header;     
    sCode.MinRepeat = 5;     
    sCode.One = code.CodeInfo.One;     
    sCode.Repeat = code.CodeInfo.Repeat;     
    sCode.Trail = code.CodeInfo.Trail;     
    sCode.Zero = code.CodeInfo.Zero;     
    sCode.Mask = new ToggleMask()     
    {         
        BitSize = code.CodeInfo.ToggleMask.BitCount,         
        CodeData = code.CodeInfo.ToggleMask.Data         
    }; 
    
    return sCode;     
}

The reverse process is easier because I can pass each of the existing properties to the IRCodeInfo constructor. The only difference is the fact that I need to use Reflection to create an instance of IRLearnedCode because there is no public constructor defined and a dynamic object has to be created:

internal static IRLearnedCode GetLearnedCode(SerializableIRCode serializableIRCode) 
{ 
    IRCode code = new IRCode(serializableIRCode.CodeData, serializableIRCode.BitSize);     
    IRCodeInfo info = new IRCodeInfo(serializableIRCode.Encoding, serializableIRCode.BitSize, serializableIRCode.Header,     
    serializableIRCode.Zero, serializableIRCode.One, serializableIRCode.Trail, serializableIRCode.Gap, serializableIRCode.Repeat, 
serializableIRCode.MinRepeat, serializableIRCode.Mask.CodeData, IRCodeInfo.IRCodeLength.Constant,
serializableIRCode.CarrierFrequency, serializableIRCode.DutyCycle); 
    
    object[] parameters = new object[] { code, info }; 
    
    BindingFlags flags = BindingFlags.NonPublic | BindingFlags.Instance; 
    object instantType = Activator.CreateInstance(typeof(IRLearnedCode), flags, null, parameters, null); 

    return (IRLearnedCode)instantType; 
}

Command and Set Management

Looking back at the code that I put together for the command interpreter, there is one class that does all local content manipulation—StorageHelper. This is a simple class that performs LINQ queries on set as well as command collections, and makes sure that all the changes are preserved in the sets.xml file in the application folder that is used as the only storage place for all the content that is being manipulated by the server.

namespace Coding4Fun.MissionControl.API.Helpers 
{ 
    public class StorageHelper 
    { 
        
        /// <summary> 
        /// Lists all available sets that are currently stored on the server.         
        /// </summary> 
        /// <returns>List of sets on the machine.</returns>         
        internal static List<Set> GetSets()         
        {             
            List<Set> sets = null; 
            
            string rawContent = GetRawSets();             
            sets = JsonConvert.DeserializeObject<List<Set>>(rawContent); 
            
            return sets;             
        } 
        
        /// <summary>         
        /// Returns the list of commands that are associated with the given set.         
        /// </summary>         
        /// <param name="setName">The name of the target set.</param>         
        /// <returns>List of commands associated with the given set.</returns>         
        internal static List<Command> GetCommands(string setName)         
        {             
            List<Command> commandList = null; 
            
            var sets = GetSets(); 
            
            Set singleSet = null;             
            if (sets != null)             
                singleSet = (from c in sets where c.Name == setName select c).FirstOrDefault(); 
            
            if (singleSet != null)         
            {         
                commandList = singleSet.Commands;     
            } 
            
            return commandList; 
        } 
        
        /// <summary>     
        /// Gets the list of names for the commands in the requested set. 
        /// </summary> 
        /// <param name="setName">The name of the target set.</param> 
        /// <returns>List of commands associated with the given set.</returns> 
        internal static List<string> GetRawCommandNames(string setName) 
        { 
            List<Command> commandList = GetCommands(setName); 
            
            List<string> stringSet = null; 
            
            if (commandList != null) 
            {     
                stringSet = commandList.Select(x => x.Name).ToList();     
            } 
            
            return stringSet;     
        } 
        
        /// <summary> 
        /// Get the list of names for all sets on the local server. 
        /// </summary> 
        /// <returns>List of sets on the machine.</returns> 
        internal static List<string> GetRawSetNames() 
        {     
            List<Set> sets = GetSets(); 
            
            List<string> stringSet = null; 
    
            if (sets != null) 
            {     
                stringSet = sets.Select(x => x.Name).ToList();     
            } 
            
            return stringSet; 
        } 
        
        /// <summary> 
        /// Get the raw string contents of sets.xml. Should only be used in the 
        /// context of this class. 
        /// </summary> 
        /// <returns>JSON string representing stored sets and commands.</returns> 
        internal static string GetRawSets() 
        { 
            string sets = string.Empty; 
            
            if (File.Exists("sets")) 
            { 
                using (StreamReader reader = new StreamReader(File.OpenRead("sets"))) 
                { 
                    sets = reader.ReadToEnd(); 
                } 
            } 
            else 
            { 
                FileStream stream = File.Create("sets.xml"); 
                
                stream.Close();     
            } 
            
            return sets; 
        } 
        
        /// <summary> 
        /// Check whether a set is marked with a IsList flag. 
        /// </summary> 
        /// <param name="setName">The name of the target set.</param> 
        /// <returns>TRUE - set is a list. FALSE - set is not a list.</returns> 
        internal static bool IsSetAList(string setName) 
        { 
            bool isList = true; 
            var sets = GetSets(); 
            Set set = null; 

            if (sets != null) 
                set = (from c in sets where c.Name == setName select c).FirstOrDefault(); 
            
            if (set != null) 
                isList = set.IsList; 
            
            return isList;     
        } 
        
        /// <summary> 
        /// Serialize the set collection to sets.xml 
        /// </summary> 
        /// <param name="sets">Collection to be serialized.</param> 
        /// <returns>true if sets are serialized.</returns> 
        private static bool SerializeSets(List<Set> sets) 
        { 
            try 
            { 
                using (StreamWriter writer = new StreamWriter("sets.xml", false)) 
                {     
                    string data = JsonConvert.SerializeObject(sets); 
            
                    writer.Write(data); 
                } 
                
                return true; 
            } 
            catch 
            {     
                return false;     
            } 
        } 
        
        /// <summary> 
        /// Add a new set to the existing global set collection. 
        /// </summary> 
        /// <param name="name">Set name.</param> 
        /// <returns>true if successfully added set.</returns> 
        internal static bool AddSet(string name, bool isList = true) 
        { 
            var sets = GetSets(); 

            if (sets == null) 
                sets = new List<Set>(); 
            
            var singleSet = GetSingleSet(sets, name); 

            if (singleSet == null) 
                sets.Add(new Set { Name = name, IsList = isList }); 
            
            if (SerializeSets(sets)) 
                return true; 
            else 
                return false; 
        } 
        
        /// <summary> 
        /// Retrieves a single set from a collection that has a specific name. 
        /// </summary> 
        /// <param name="sets">The source collection from which to extract the set.</param> 
        /// <param name="name">The name of the set to get.</param> 
        /// <returns>An instance of the found set, if any.</returns> 
        internal static Set GetSingleSet(List<Set> sets, string name) 
        { 
            if (sets != null) 
                return (from c in sets where c.Name == name select c).FirstOrDefault(); 
            else 
                return null; 
        } 
        
        /// <summary> 
        /// Add a IR command to an existing set. If the set is not found, it will be created. 
        /// </summary> 
        /// <param name="command">The command instance to be added.</param> 
        /// <param name="targetSet">The name of the target set.</param> 
        /// <returns>true if the command was successfully added.</returns> 
        internal static bool AddCommand(Command command, string targetSet) 
        { 
            var sets = GetSets(); 
            
            if (sets == null) 
                sets = new List<Set>(); 
            
            var singleSet = GetSingleSet(sets, targetSet); 
            
            if (singleSet == null) 
                singleSet = new Set { Name = targetSet }; 
            
            var singleCommand = (from c in singleSet.Commands where c.Name == command.Name select c).FirstOrDefault(); 
            
            if (singleCommand == null) 
            { 
                singleSet.Commands.Add(command); 
                
                if (SerializeSets(sets)) 
                    return true; 
                else 
                    return false; 
            } 
            else 
                return false; 
        } 
        
        /// <summary> 
        /// Retrieve a single command instance from one of the sets on the local server. 
        /// </summary> 
        /// <param name="commands">Original list of commands.</param> 
        /// <param name="name">Name of the command to be retrieved.</param> 
        /// <returns>An instance of the command, if found. NULL if not.</returns> 
        internal static Command GetSingleCommand(List<Command> commands, string name) 
        { 
            if (commands != null) 
                return (from c in commands where c.Name == name select c).FirstOrDefault(); 
            else 
                return null; 
        } 
        
        /// <summary> 
        /// Remove a set from a local machine. 
        /// </summary> 
        /// <param name="sets">Original list of sets.</param> 
        /// <param name="targetSet">Name of the set to remove.</param> 
        internal static void RemoveSet(List<Set> sets, Set targetSet) 
        { 
            sets.Remove(targetSet); 
            
            SerializeSets(sets); 
        } 
    } 
}

Sending Data Back to the Client

SendData in the NetworkHelper class handles all outbound connections. Here is its structure:

/// <summary> 
/// Send data to the target network machine. 
/// </summary> 
/// <param name="destination">The target machine IP.</param> 
/// <param name="data">Data to be sent, in string format.</param> 
/// <param name="sanitizeIp">Determines whether to remove the port from the given IP string.</param> 
public static void SendData(string destination, string data, bool sanitizeIp = true) 
{ 
    using (Socket client = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp)) 
    {     
        string completeIp = string.Empty; 
        
        if (sanitizeIp)     
            completeIp = destination.Remove(destination.IndexOf(":"), destination.Length - destination.IndexOf(":")); 
        
        client.Connect(completeIp, 6169); 
        client.Send(Encoding.UTF8.GetBytes(data));     
    }     
}

A new stream socket is created in order to connect to the target machine over the TCP pipe. If IP sanitization is enabled, the port is stripped from the address in order to pass a valid IP. A Socket instance cannot directly handle IPs of the format:

255.255.255.0:PORT_NUMBER

Later, in a synchronous manner, a connection is established and the data is sent.

At this point, you can see that the barebones service offers a flexible way to manage content. It can be accessed by any application type as long as the server can be accessed and the application can send commands in the pre-defined format and the content requested is actually located on the target server. This allows for high levels of extensibility and interoperability, as the server usage is not limited to a single platform. If I decide to create a Windows Store application that would allow me to control my TV, I simply need to add socket connection layer that will send plain strings to the machine where the IR sensor is connected.

Similarly, if some functionality needs to be added, it is possible to do so without ever touching the client applications. A modification in the endpoint will be reflected with no direct effect on all connection applications as long as all handled returned and requested values are preserved. The only additional requirement is that if the client applications want to take advantage of newly introduced capabilities, they need to have an updated command transmission layer for the new command types.

In Program.cs, I simply need to start the server through the CoreStarter class:

namespace Coding4Fun.MissionControl.API 
{ 
    class Program     
    {         
        static void Main(string[] args)         
        {             
            Console.WriteLine("Coding4Fun MissionControl Server");             
            CoreStarter.LaunchSocket();             
        }         
    }     
}

Mobile client overview

The mobile client does not have the capability to send commands directly to the IR sensor. Instead, it connects to a remote machine that has the IR sensor plugged in and attempts to invoke a command from the list returned by the service. A single mobile client can support control over multiple servers.

NOTE: Make sure that at the time of working with the Windows Phone client, the server is actually running on your local machine. To make it easier to test, also open port 6169 for incoming connections in Windows Firewall.

When building a Windows Phone application, make sure you have the proper version of the SDK installed, as well as a SLAT-compatible machine if you plan on testing the application in the emulator.

Networking Infrastructure

The Windows Phone application also relies on a network infrastructure somewhat similar to that of the server. There is a TCP listener that is created when the application is started:

// Code to execute when the application is launching (eg, from Start) 
// This code will not execute when the application is reactivated 
private void Application_Launching(object sender, LaunchingEventArgs e) 
{ 
    ServiceSerializer.DeserializeServices(); 
    
    listener.OnClientConnected += listener_OnClientConnected; 
    listener.Start(6169); 
}

Here, listener is an instance of TcpSocketListener—a custom class designed to handle incoming network connections:

namespace Coding4Fun.MissionControl.WP.Network 
{ 
    public class TcpSocketListener : SocketConnectorBase     
    {     
        StreamSocketListener coreSocket; 
        
        public async void Start(int port)     
        {     
            coreSocket = new StreamSocketListener();         
            coreSocket.ConnectionReceived += coreSocket_ConnectionReceived; 
            
            try     
            {         
                await coreSocket.BindServiceNameAsync(port.ToString());         
            }     
            catch (Exception ex) 
            {     
                Debug.WriteLine(ex.Message); 
                
                coreSocket.Dispose();     
                coreSocket = null; 
                OnConnectionCompleted(new ConnectionEventArgs { IsSuccessful = false, DeviceID = string.Empty }); 
            } 
        } 
        
        async void coreSocket_ConnectionReceived(StreamSocketListener sender, StreamSocketListenerConnectionReceivedEventArgs args) 
        { 
            Debug.WriteLine("Connection received!"); 
            
            DataReader reader = new DataReader(args.Socket.InputStream); 
            
            try 
            { 
                while (true) 
                { 
                    StringBuilder builder = new StringBuilder(); 
        
                    uint actualLength = 1; 

                    while (actualLength > 0) 
                    {     
                        actualLength = await reader.LoadAsync(256); 
                        builder.Append(reader.ReadString(actualLength)); 
                    } 
                    
                    OnConnectionCompleted(new ConnectionEventArgs 
                    { 
                        Socket = args.Socket, 
                        IsSuccessful = true, 
                        DeviceID = args.Socket.Information.RemoteHostName.DisplayName, 
                        Token = builder.ToString() 
                    }); 
                    break; 
                } 
            } 
            catch (Exception exception) 
            { 
                Debug.WriteLine(exception.Message); 
                
                OnConnectionCompleted(new ConnectionEventArgs { IsSuccessful = false }); 
            } 
        } 
    } 
}

A StreamSocketListener is used for the connection core. When a connection is received, a continuous loop reads the entire contents of the incoming stream. OnConnectionCompleted is declared in the base class—SocketConnectorBase.

namespace Coding4Fun.MissionControl.WP.Network 
{ 
    public class SocketConnectorBase 
    { 
        public event EventHandler<ConnectionEventArgs> OnClientConnected; 
        public virtual void OnConnectionCompleted(ConnectionEventArgs connectionArgs) 
        { 
            if (OnClientConnected != null) 
            { 
                OnClientConnected(this, connectionArgs); 
            } 
        } 
        
        public event EventHandler<bool> OnSendCompletedEvent; 
        public virtual void OnSendCompleted(bool succeeded) 
        { 
            if (OnSendCompletedEvent != null) 
            { 
                OnSendCompletedEvent(this, succeeded); 
            } 
        } 
    } 
    
    public class ConnectionEventArgs : EventArgs 
    { 
        public StreamSocket Socket { get; set; }     
        public string DeviceID { get; set; }     
        public string Token { get; set; }     
        public bool IsSuccessful { get; set; }     
    } 
}

ConnectionEventArgs here is used to identify the content that is passed to the client. DeviceID gives access to the source IP, IsSuccessful tells the developer whether the established connection is active and the Token carries the raw string if any was received.

Sending data is simplified to the maximum with the help of the SocketClient class, which relies on a StreamSocket instance that handles outbound connections and writing to the output stream:

namespace Coding4Fun.MissionControl.WP.Network 
{ 
    public class SocketClient : SocketConnectorBase 
    {     
        StreamSocket _socket; 
        
        public SocketClient() 
        {     
            _socket = new StreamSocket();     
        } 
        
        public SocketClient(StreamSocket socket) 
        {     
            _socket = socket;     
        } 
        
        public async void Connect(string hostName, int portNumber) 
        { 
            try 
            { 
                await _socket.ConnectAsync(new HostName(hostName), portNumber.ToString(), SocketProtectionLevel.PlainSocket); 
            
                OnConnectionCompleted(new ConnectionEventArgs { IsSuccessful = true }); 
            } 
            catch (Exception ex) 
            {     
                Debug.WriteLine(ex.Message); 
                
                OnConnectionCompleted(new ConnectionEventArgs { IsSuccessful = false });     
            } 
        } 
        
        public async void Send(string dataToSend) 
        { 
            try 
            {     
                using (DataWriter writer = new DataWriter(_socket.OutputStream)) 
                { 
                    // Write the length of the binary data that is being 
                    // sent to the client. 
                    writer.WriteUInt32((UInt32)dataToSend.Length); 
                    
                    writer.WriteString(dataToSend); 
        
                    // Send the actual data. 
                    await writer.StoreAsync(); 
                    
                    writer.DetachStream(); 
                    
                    OnSendCompleted(true); 
                } 
            } 
            catch 
            { 
                _socket.Dispose();     
                _socket = null;     
                OnSendCompleted(false);     
            }     
        }     
    }     
}

As with the listener class, SocketClient supports OnConnectionCompleted to notify the application that the connection attempt completed.

Back in App.xaml.cs, the data from the incoming connection captured by the TcpSocketListener instance is passed to the ResponseHelper class:

void listener_OnClientConnected(object sender, ConnectionEventArgs e) 
{ 
    ResponseHelper.HandleIncomingResponse(e.Token); 
}

This class reads the possible three commands sent by the server and interprets them, creating internal collections from the raw data if the current server IP matches the one obtained in the ServerResponse (the same model in the desktop application):

using Coding4Fun.MissionControl.WP.Models;
using Coding4Fun.MissionControl.WP.ViewModels;
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Windows;

namespace Coding4Fun.MissionControl.WP.Misc
{
        public class ResponseHelper
        {
                public static void HandleIncomingResponse(string rawResponse)
                {
                        if (rawResponse != null)
                        {
                                ServerResponse response = JsonConvert.DeserializeObject<ServerResponse>(rawResponse);
                
                                if (response.Marker == Constants.COMMAND_SERVER_NOTIFICATION)
                                {
                                        Deployment.Current.Dispatcher.BeginInvoke(() =>
                                        {
                                                MessageBox.Show(response.Content, "Server Response", MessageBoxButton.OK);
                                        });
                                }
                                else
                                {
                                        if (CommonViewModel.Instance.IsWaiting)
                                        {
                                                if (response.Identifier == CommonViewModel.Instance.CurrentServer.Location)
                                                {
                                                        // returns the list of sets that are associated with the current server.
                                                        if (response.Marker == Constants.COMMAND_SERVER_SET_LIST)
                                                        {
                                
                                                                List<string> items = JsonConvert.DeserializeObject<List<string>>(response.Content);
                                
                                                                if (items != null)
                                                                {
                                                                        List<Group<string>> groupedItems = Group<string>.CreateGroups(items,
                                                                        CultureInfo.CurrentCulture, (string s) => { return s[0].ToString(); }, true);
                                                                        SetsPageViewModel.Instance.Sets = groupedItems;
                                                                }
                                                                else
                                                                {
                                                                        SetsPageViewModel.Instance.Sets = new List<Group<string>>();
                                                                }
                                
                                                                Deployment.Current.Dispatcher.BeginInvoke(() =>
                                                                    {
                                                                            CommonViewModel.Instance.IsWaiting = false;
                                    
                                                                            if (!App.RootFrame.CurrentSource.ToString().Contains("SetsPage"))
                                                                            {
                                                                                    App.RootFrame.Navigate(new Uri("/Views/SetsPage.xaml", UriKind.Relative));
                                                                            }
                                                                    });
                                
                                                        }
                                                        // returns the list of commands associated with a given set.
                                                        else if (response.Marker.Contains(Constants.COMMAND_SERVER_SET_COMMANDS))
                                                        {
                                                            string[] data = response.Marker.Split(new char[] { ':' });
                                                                if (data[1] == CommonViewModel.Instance.CurrentSet)
                                                                {
                                                                        bool isList = false;
                                                                        bool.TryParse(data[2].ToLower(), out isList);
                                    
                                                                        if (isList)
                                                                        {
                                                                                CommonViewModel.Instance.CurrentSetType = "list";
                                                                        }
                                                                        else
                                                                        {
                                                                                CommonViewModel.Instance.CurrentSetType = "pad";
                                                                        }
                                    
                                                                        CommandsPageViewModel.Instance.Commands = new System.Collections.ObjectModel.ObservableCollection<string>(JsonConvert.DeserializeObject<List<string>>(response.Content));
                                    
                                                                        Deployment.Current.Dispatcher.BeginInvoke(() =>
                                                                            {
                                                                                    CommonViewModel.Instance.IsWaiting = false;
                                                                                    App.RootFrame.Navigate(new Uri("/Views/CommandsPage.xaml", UriKind.Relative));
                                                                            });
                                                                }
                                                        }
                                                }
                                        }
                                }
                        }
                }
        }
}

If the response comes from a server that is different than the one that is currently active, the data is discarded as the user no longer needs it. Also, for specific commands, the mobile application will be on standby, waiting for a response (unless the user decides to cancel the request) – the IsWaiting flag is an application-wide indicator that a pending server action is in the queue.

Same as with the server, the commands in the Windows Phone application are represented through pre-defined constants:

public const string COMMAND_SERVER_SET_LIST = "SET_LIST"; 

public const string COMMAND_SERVER_SET_COMMANDS = "SET_COMMANDS"; 

public const string COMMAND_SERVER_NOTIFICATION = "NOTIFICATION";

Let’s now take a closer look at how it is handled internally to build the visual layer.

Handling the Data

The first thing users will see when the application is launched is the list of registered servers:

This is ServiceListPage.xaml. The list of servers that were added is retrieved from the isolated storage on application startup, with the help of the standard serialization routine implemented in the Coding4Fun Toolkit—specifically, its storage subset (you can get it via NuGet):

The one-liner that initializes the internal server collection is as follows:

MainPageViewModel.Instance.Servers = Serialize.Open<ObservableCollection<Server>>(Constants.SERVERS_FILE);

Here, the SERVERS_FILE constant is equal to servers.xml. It is a good idea to use constants for file names in order to be able to later modify the location through a single change instead of digging through the many source files in a solution to find references to the old location.

The user can define an unlimited number of servers, as long as he can access those. There is no restriction on the location of the server itself—it can work with the desktop in your room just as well as with a PC on the other end of the world (yes, this was tested).

When adding a new server, the user is redirected to AddServicePage.xaml, where he can fill in connection details, as well as the location of an image that would help him identify that specific item in the general list:

Once data entry is complete, it is validated internally to make sure that the server is not already registered with the same name and location. If the validation step passes, the server is added to the list of local access points and the user is returned back to the server selection page:

private void AttemptAddService()
{
        if (!string.IsNullOrWhiteSpace(txtName.Text) && !string.IsNullOrWhiteSpace(txtLocation.Text))
        {
                Server server = new Server
                    {
                            Name = txtName.Text,
                            Location = txtLocation.Text,
                            ImageURL = !string.IsNullOrWhiteSpace(txtAvatar.Text) ? txtAvatar.Text : string.Empty
                    };
        
                if (!CollectionHelper.CheckServerExists(server))
                {
                        MainPageViewModel.Instance.Servers.Add(server);
            
                        Serialize.Save(Constants.SERVERS_FILE, MainPageViewModel.Instance.Servers);
                        NavigationService.GoBack();
                }
                else
                {
                        Alert.Send("The service with this name or location is already registered.");
                }
        }
        else
        {
                Alert.Send("The service needs a name and a location.");
        }
}

When a server selection is made by the user, it is necessary to show SetsPage.xaml. However, it is necessary to also check whether the server is active or not prior to the actual navigation. With the help of internal bindings, I am doing it through a RelayCommand:

public RelayCommand SelectServerCommand { get; private set; }
private async void SelectServer(object server)
{
        CommonViewModel.Instance.IsWaiting = true;
        CommonViewModel.Instance.CurrentServer = (Server)server;
    
        bool result = await CommonViewModel.Instance.CommandClient.SendCommand(CommonViewModel.Instance.CurrentServer.Location, 
        Constants.COMMAND_SERVER_HELLO, string.Empty);
        if (!result)
        {
                Alert.Send(Constants.MESSAGE_SERVER_CONNECT_FAIL);
                CommonViewModel.Instance.CurrentServer = null;
        
                CommonViewModel.Instance.IsWaiting = false;
        }
}

COMMAND_SERVER_HELLO represents the initial handshake command that I mentioned earlier—it requests the list of sets on the target server. To streamline command processing, CommandClient is used and wraps around the SocketClient class, giving me the possibility to call SendCommand with the command metadata without having to explicitly handle socket interactions in my views:

namespace Coding4Fun.MissionControl.WP.Network 
{     
    public class CommandClient     
    {     
        private SocketClient client; 
        
        public Task<bool> SendCommand(string key, string value, Action<bool> onCompleted = null) 
        {     
            var taskCompletionSource = new TaskCompletionSource<bool>(); 
            
            client = new SocketClient();     
            client.OnClientConnected += (s, args) =>     
            {     
                if (args.IsSuccessful)     
                {     
                    string data = JsonConvert.SerializeObject(new KeyValuePair<string, string>(key, value));     
                    client.Send(data);     
                } 
            
                taskCompletionSource.SetResult(args.IsSuccessful); 
                client = null; 
            }; 
            
            client.Connect(Binder.Instance.CurrentService.Location, 6169); 
            return taskCompletionSource.Task;     
        } 
    } 
}

From here on, ResponseHelper is once again involved, grouping all the data alphabetically—remember this call:

// returns the list of sets that are associated with the current server. 
if (response.Marker == Constants.COMMAND_SERVER_SET_LIST) 
{     
    List<string> items = JsonConvert.DeserializeObject<List<string>>(response.Content);     
    if (items != null) 
    {     
        List<Group<string>> groupedItems = Group<string>.CreateGroups(items,     
    CultureInfo.CurrentCulture, (string s) => { return s[0].ToString(); }, true);     
        Binder.Instance.Sets = groupedItems;     
    } 
    else 
    {     
        Binder.Instance.Sets = new List<Group<string>>();     
    } 
}

The grouped collection is later bound to a LongListSelector instance:

For each handshake call to the server, the set collection will be re-initialized, in case the server was updated by another device while the user was not taking any actions.

Adding a set takes the user to AddSetPage.xaml, where the user input is once again validated and the appropriate command sent to the currently selected server:

private async void AttemptAddSet()
{
        if (!string.IsNullOrWhiteSpace(txtName.Text))
        {
                this.Focus();
        
                bool commandSent = await CommonViewModel.Instance.CommandClient.SendCommand(CommonViewModel.Instance.CurrentServer.Location, 
                    string.Format(Constants.COMMAND_CREATE_SET, ((ListPickerItem)lstType.SelectedItem).Content.ToString()), txtName.Text);
                if (!commandSent)
                {
                        Alert.Send(Constants.MESSAGE_SERVER_CONNECT_FAIL);
                }
                else
                {
                        CommonViewModel.Instance.IsWaiting = true;
                        NavigationService.GoBack();
                }
        }
        else
        {
                Alert.Send(Constants.MESSAGE_NO_NAME_FAIL);
        }
}

The end-user is also able to specify whether the new set is a list or a pad. Since the server does not explicitly define the type of a set beyond marking whether it’s a list, it is possible to have an arbitrary type here.

To give you an idea of what it looks like in the current release of MissionControl, here is the pad representation of a set of commands:

It is a convenient way to display buttons for typical actions, such as channel switching through digits. Since we can safely assume many of those will be tapped sequentially, a list would be inconvenient to scroll through.

On the other hand, some remote control commands work well with a list because no sequences are invoked most of the time:

If the pad is not desired, it can easily be swapped with another design and internal template - the appearance is swapped dynamically and is not hard-bound to a string value.

Once a set is selected, a connection attempt is made to the current server in order to check whether there is still a communication channel available with the resource that fetched the initial list of commands. If a connection is established, the server will also return a set of commands that are available in the set at the time of the request.

private async void AttemptLoadCommands()
{
        bool commandSent = await CommonViewModel.Instance.CommandClient.SendCommand(CommonViewModel.Instance.CurrentServer.Location,
            Constants.COMMAND_GET_COMMANDS, CommonViewModel.Instance.CurrentSet);
        if (!commandSent)
        {
                Alert.Send(Constants.MESSAGE_SERVER_CONNECT_FAIL);
                CommonViewModel.Instance.IsWaiting = false;
        }
}

You’ve probably already noticed that both for commands and sets, the initial routine verifies the connection to the server. The server might go dark after the set list is loaded, therefore rendering any attempt to process other commands impossible. To avoid scenarios in which the user is waiting for a response from a server that doesn’t run, the user is notified before being redirected to the subsequent view, if the connection fails. That way unnecessary navigation passes are out of the picture.

If the user selects a command from one of the lists demonstrated above, an EXECUTE directive is issued via the CommandClient class:

private async void lstCommands_SelectionChanged(object sender, SelectionChangedEventArgs e) 
{ 
    if (lstCommands.SelectedItem != null)
    { 
        string selectedItem = lstCommands.SelectedItem.ToString(); 
        
        CommandClient commandClient = new CommandClient(); 
        bool commandSent = await commandClient.SendCommand(string.Format(Constants.COMMAND_EXECUTE, 
        
        Binder.Instance.CurrentSet), selectedItem); 
    
        if (!commandSent) 
        {     
            Alert.Send(Constants.MESSAGE_SERVER_CONNECT_FAIL);     
        } 
        
        lstCommands.SelectedItem = null; 
    } 
}

Once the server receives the command, it will send it to the target without additional notifications being released to the connecting client.

When it comes to learning a new remote control code in LearnCodePage.xaml, the procedure is exactly the same as with any other part of the server communication process — a LEARN_CODE command is sent to the server with the associated set and new command name, and the server will wait for incoming IR input, leaving the connecting device free (no waiting lock is issued):

private async void AttemptLearnCode()
{
        if (!string.IsNullOrWhiteSpace(txtName.Text))
        {
                CommonViewModel.Instance.IsWaiting = true;
                this.Focus();
        
                bool commandSent = await CommonViewModel.Instance.CommandClient.SendCommand(CommonViewModel.Instance.CurrentServer.Location,
                    string.Format(Constants.COMMAND_LEARN_NEW, CommonViewModel.Instance.CurrentSet), txtName.Text);
                if (!commandSent)
                {
                        Alert.Send(Constants.MESSAGE_SERVER_CONNECT_FAIL);
                }
                else
                {
                        Alert.Send(Constants.MESSAGE_COMMAND_LEARN_WAIT);
                        NavigationService.GoBack();
                }
        
                CommonViewModel.Instance.IsWaiting = false;
        }
        else
        {
                Alert.Send(Constants.MESSAGE_NO_NAME_FAIL);
        }
}

Once the server learns a new command — if, and only if, the user still works in the context of the same server — an alert will be displayed, telling the user whether the command was successfully learned.

For convenience purposes, I also implemented a quick launch panel, where frequently-used commands can be placed. Whenever a user wants to add something here, he will tap-and-hold on an existing command in any of the sets that are available for any given server, and select the "add to quick launch" option. Once completed, the stored commands will be available on the main page, even when the user is not directly connected to the server that carries the command:

Because this interaction layer is placed outside the boundaries of a single server or set, I needed to create a special data model to store the quick commands and the related connection information, that would let me call the server even when it is not the currently selected one:

namespace Coding4Fun.MissionControl.WP.Models
{
        public class Favorite
        {
            public string CommandName { get; set; }
            public string ParentSet { get; set; }
            public string ServerLocation { get; set; }
        }
}

Same as with the list of servers, the list of favorites is deserialized on application startup:

MainPageViewModel.Instance.Favorites = Serialize.Open<ObservableCollection<Favorite>>(Constants.FAVORITES_FILE);

Logically, we would also need to have a way to eliminate trailing commands for servers or sets that have been removed, since those can no longer be invoked or might have a different meaning on servers that were added and have the same IP as the previous owner. This is easily done with a simple LINQ expression that is passed to RemoveTrailingFavorites in the CollectionHelper class:

internal static void RemoveTrailingFavorites(Func<Favorite,bool> predicate)
{
        var favorites = MainPageViewModel.Instance.Favorites.Where(predicate).ToList();
    
        if (favorites.Count() > 0)
        {
                foreach (var favorite in favorites)
                {
                        Deployment.Current.Dispatcher.BeginInvoke(() =>
                            {
                                    MainPageViewModel.Instance.Favorites.Remove(favorite);
                            });
                }
        
                Serialize.Save(Constants.FAVORITES_FILE, MainPageViewModel.Instance.Favorites);
        }
}

A typical usage scenario is reflected in the server removal snippet:

public static bool RemoveServer(Server server)
{
        try
        {
                RemoveTrailingFavorites(x=> x.ServerLocation == server.Location);
        
                MainPageViewModel.Instance.Servers.Remove(server);
        
                Serialize.Save(Constants.SERVERS_FILE, MainPageViewModel.Instance.Servers);
        
                return true;
        }
        catch
        {
                return false;
        }
}

Because an ObservableCollection<T> is used for both the list of servers and quick launch commands, the view will be instantly updated to reflect the changes.

Improvements to the project

This specific project relies on a hybrid IR transmitter and receiver, which is not exactly cheap. As a step forward for this project, it can be adapted to use a central microcontroller that acts as a server (e.g. Netduino) and a series of IR emitters (instead of using a composite receiver/emitter) connected to it. Reduced cost for the IR infrastructure is key, as not every single component needs the capability to learn IR commands. You can have a single command capturing endpoint and multiple transmitters. This will also eliminate the need for a desktop client, since the server on the microcontroller can be built to be accessible via a web-browser.

Another important aspect not covered in this article is security. With the current workflow, anyone who has direct access to the server IP is able to do anything he wants with the data handled by the server. I am basing my writing on the assumption that you are testing the application on a secure local network and that the the odds of something like this happening are close to zero. However, for other environments where tampering with a server might be unacceptable, consider implementing a layer of security between the server and the client.

Conclusion

With affordable microcontrollers and sensors, home and office automation can be a nice bonus resulting from little investment. This article covers the implementation of a proof-of-concept server and application that can be easily extended and adapted to a variety of environments and devices.

Windows 8 Platformer Game Starter Kit

Platformer Game Starter Kit(includes 2 HTML5-based examples)

Don't start from scratch. Our Platformer Game Starter Kit for Windows 8 will get you going with full code & free game art.

How

---

Feel free to take the examples, mix them up and build your own great game for the Windows Store following these steps:

1. Download your free Visual Studio for Windows 8 for all the tools you will need.
2. Download the Platformer Game Starter Kit.
3. Mix it up and build your own epic game.
4. Open a Windows Store Developer Account.
5. Let us know about it and we can even give you free advice on making it great and help pass store certification.
6. Publish it to the Windows Store.

We know everyone is not a game artist, so we have provided you with some free game art and other places you can find royalty-free art for your game.

Ludus Game Starter Kit

An HTML5 Platformer Game Starter Kit for Windows8 Store

by Christer (McFunkypants) Kaitila http://mcfunkypants.com

Source: https://github.com/mcfunkypants/ludus

Demos: http://www.mcfunkypants.com/ludus

Welcome to Ludus, brave adventurer!

Ludus is a dirt-simple game engine that uses html5 canvas. It was designed specifically for mario/sonic style platformer games but could be used for any genre with minimal changes. It boasts great performance, and requires only free tools.

The word Ludus means PLAY/GAME/TRAINING in latin. The Ludus engine is designed to be a great way to learn to make games.

Sure, there are bigger and more complex game engines.

This one is designed to be easy to play around with.

Why? It was created to run real-world games that have a beginning, middle, and end. It was optimized to run on less powerful systems and touch-screen tablets. It is simple, but goes beyond the level of a "tech demo" to encapsulate all the required functionality you might need to make a polished game, such as GUI and sound.

...

IT'S POSSIBLE TO MAKE A GAME USING LUDUS IN A WEEKEND.

You could make your own game just by changing the artwork: it is possible to make a new game without ever touching a line of code. All you would need to do is modify the .png and .mp3 art assets as well as the level#.js level data files.

The best tool to create in-game worlds is TILED (http://www.mapeditor.org) which exports data in .json format.

Check out the example .tmx source files and see if you can make your own game! Once you've made a level, export as .json and then run the _buildlevels.bat file to convert JSON to JSONP data to a new level#.js file.

The one important thing to remember is that each layer is important. The order matters. The bottom-most layer is for regular world tiles that the player will collide with. The rest are for pickups, bouncy platforms, dangerous spikes, and so on.

Additionally, you should modify the game settings in the MAP MENU -> MAP PROPERTIES dialog. Gravity, speed and the player's starting position are all stored there.

HACKING THE SOURCE CODE

The only source code file you will want to modify, if you choose to start coding, is ludus.js. Inside, you'll notice all sorts of variables at the top which you can modify to your heart's content. Note that some of these variables are overwritten by whatever is defined inside the level data.

With regard to the structure of the code, the most important thing to know is that the game is controlled by STATE objects.

Each state (title screen, in-game, and between level transitions) is a class object that has a .setup(), .update() and .draw() function. You could add new states for things like boss battles, inventory screens or a high score table.

See these class constructors: function TitleScreenState() function LevelTransitionScreenState() function PlayState()

HAVE FUN!

...

Let's take a peek at the two included example games (In Visual Studio of course!)...

Both games ran for me the first time with no issues and were fun (in a platformer kind of way). Sure there's only 3 levels, but hey, if there was too much done it would take all the coding fun away!

So either start from scratch or copy off of these, this kit and the provided free artwork, framework and code will have you building your next great Windows 8 game...

So Philips recently introduced their Hue Connected Bulbs: an easy-to-use set of LED light bulbs and Wi-Fi connected bridge which allows you to dynamically change the color of your home lighting using their iOS or Android app. What’s particularly cool is that the bridge has a web API which you can access to set the colors of each bulb with your own app.

We at untitled network developed our own Philips Hue app called Oni: light Control, which is currently available on the Windows Phone Store. In addition to allowing you to set the color of your home lighting with defined “Moods”, the app also allows you to do the same using your phone’s built in voice commands or inexpensive NFC stickers. Here’s a demo of the Oni: Light Control in action:

I’m going to show you how you can develop your own app using the color picker from the Coding4Fun Toolkit for Windows Phone and the Json.Net library from Newtonsoft. These libraries can be found on NuGet, but you’ll obviously need the Philips Hue Connected Bulbs kit to test things out.

Getting Started:

The Philips Hue API uses a restful JSON interface you can access using any http client. Documentation on all of the supported methods by the Philips Hue bridge can be found at http://blog.ef.net/2012/11/02/philips-hue-api.html.

To get started, you’ll need authorized access to your bridge’s API. Once the bridge has successfully established a network connection with your router, discover its internal IP address via the URL: http://www.meethue.com/api/nupnp

You should get a response similar to:

[{"id":"ffss00fffe123456","internalipaddress":"192.168.1.100","macaddress":"0aa:bb:cc:dd:00:11"}]

Note the internalipaddress value and use the IP to access the bridge’s API directly the with the URL http://192.168.1.100/api

Now, since we haven’t registered a user for authorization, attempting to access the hub will return an error from the bridge:

[{"error":{"type":1,"address":"/","description":"unauthorized user"}}]

To register a new user, we’ll first POST the username we wish to use.

var client = new WebClient();
 
//our uri to perform registration
var uri = new Uri(string.Format("http://{0}/api", HostnameTextBox.Text));
 
//create our registration object, along with username and description
var reg = new
{
    username = UsernameTextBox.Text,
    devicetype = "Coding4Fun Hue Light Project"
};
 
var jsonObj = JsonConvert.SerializeObject(reg);
 
 
//decide what to do with the response we get back from the bridge
client.UploadStringCompleted += (o, args) =] Dispatcher.BeginInvoke(() =]
{
    try
    {
        ResponseTextBox.Text = args.Result;
    }
    catch (Exception ex)
    {
        ResponseTextBox.Text = ex.Message;
    }
});
 
//Invoke a POST to the bridge
client.UploadStringAsync(uri, jsonObj);

Note the response we get back from our hub will be

[{"error":{"type":101,"address":"","description":"link button not pressed"}}]

This is because bridge requires you to first push the link button before new registrations can be made. After pushing the button and invoking the registration function again, you should receive the following result from the bridge:

[{"success":{"username":"coding4fun"}}]

Setting the Bulb Color:

We should now be able to access all methods on the bridge. You can get all of the configuration details, including all of the bulbs and their statuses with the same base url: http://192.168.1.100/api/coding4fun

Now comes the fun part. There are three color modes in which you can use to set the color of your bulbs:

• hue & sat: ‘hue’ is a color range between 0-65535 which represent 182.04*degrees, ‘sat’ is saturation with a range of 0-254
• xy: are coordinates in the CIE 1931 space
• ct: is a color temperature expressed in mireds from 154 to 500, coolest to warmest respectfully

Source: http://rsmck.co.uk/hue

We’ll be setting the colors of our bulbs using hue & saturation parameters. Luckily, the Coding4Fun Toolkit for Windows Phone has three awesome color picker controls and some useful color extensions which makes setting the bulb colors a breeze. We start by building our state object – a list of parameters we want our bulb to be set to. Then we use the PUT verb to set the light with the following URL: http://{BRIGE-IPADDRESS}/api/coding4fun/lights/1/state

The “1” in the url is the 1-based index of the bulb you want to set.

//Get the HSV Value from the currently selected color
var hsv = LightColorSlider.Color.GetHSV();
 
//build our State object
var state = new
{
    on = true,
    hue = (int)(hsv.Hue * 182.04), //we convert the hue value into degrees by multiplying the value by 182.04
    sat = (int)(hsv.Saturation * 254)
};
 
//convert it to json:
var jsonObj = JsonConvert.SerializeObject(state);
 
//set the api url to set the state
var uri = new Uri(string.Format("http://{0}/api/{1}/lights/{2}/state", HostnameTextBox.Text, UsernameTextBox.Text, LightIndexTextBox.Text));
 
var client = new WebClient();
 
//decide what to do with the response we get back from the bridge
client.UploadStringCompleted += (o, args) => Dispatcher.BeginInvoke(() =>
{
    try
    {
        ResponseTextBox.Text = args.Result;
    }
    catch (Exception ex)
    {
        ResponseTextBox.Text = ex.Message;
    }
     
});
 
//Invoke the PUT method to set the state of the bulb
client.UploadStringAsync(uri, "PUT", jsonObj);

That’s it! Be sure to check out all of the other functions the (other functions the Hue API supports) at http://developers.meethue.com/index.html

Bio:
Jarem Archer is a self-taught Software Developer and UX Designer. He’s part of a small team at untitled network who have a passion for video games, digital motion, entertainment and gadgets. Follow him on Twitter at http://twitter.com/unt1tled.

Introduced at //build/ 2012, Maelstrom is Coding4Fun's latest creation.  Step up to the podium and battle against your opponent in full-on stereoscopic 3D!  Learn how it was made here.

How to Play

Each player stands at a podium containing a 27" touch-screen, 2 cameras mounted side-by-side for stereoscopic 3D, a rumble floor, and LED lighting.  After some smack-talking to each other, players tap their screen to begin.  After the countdown, a ball is launched into the arena.  Players move their fingers around on the touch-screen to move the paddle in the 3D arena and return their opponent's volley.  If the ball hits one of the glowing blue tiles, it will split into a maelstrom of 3 separate balls which will be tricky to handle.  A player can also charge up their paddle by moving it quickly across the screen to send the ball flying back at their opponent in a quick burst.  The first to score 7 points wins!  If you can imagine what "ridiculously over-engineered 3D Pong" might look like, you can imagine Maelstrom!

The Hardware

Each game podium contains a Lenovo A720 27" touch screen PC used to control the player's paddle.  Lighting along the side rails reacts to the gameplay.  Additionally, the platform under each player's feet rumbles thanks to a Sparkfun MP3 Trigger and bass boosters controlled with a Netduino Plus.  The game is projected to a huge screen with an Optoma GT750E 3D projector and 3D shutter glasses.  Each podium also contains two Axis M1014 MJPEG cameras pointed at the player to complete the 3D experience.

The Software

The game is a Windows Store DirectX 11.1 game written in C++.  DX11.1 provides full stereoscopic 3D support for any 3D-capable display device, be it a projector, an LCD TV, or anything else.

The game controller is a C# / XAML Windows Store application which connects to the "game console" via a standard socket connection.  The Netduino Plus corrdinates all of the hardware with a C# .NET Micro Framework application.  All of these tie together into a seamless multiplayer experience!

Source code and an article on how we built it all will be coming soon on Coding4Fun...

Video

You can find a very quick video of the game being played at //build/ 2012 here.

Photos

Attributions

• Music track - "FatLoad- The Bullet(no master)" by FreaK NeoS
• Sound effects + music editing - David Wallimann
• DirectX shaders - Michael McLaughlin

The OBD-II Manager project is the OBD-II sub-system and instrument cluster application from the Project Detroit Mustang we built with West Coast Customs.  If you're not familiar with this project, you can find out more about it here and here.

OBD stands for On-Board Diagnostics.  In the world of cars, this can be thought of as the computer which has a variety of data points which can be queried, such as speed, fuel level, and even trouble codes that relate to the check engine light.  If you've ever taken your car in for service, your service center has hooked up the car to a computer via the OBD port to get the status of the car and what may not be working properly. 

This project contains a library you can use to integrate OBD data into your application, as well as a modified version of the actual Instrument Cluster application used in the Project Detroit car which shows how the library can be used.

More Information

• Coding4Fun Article
• Project Detroit
• OBD-II to USB cable (ELM323/327-based OBD-II to USB cable)
• More information on OBD-II
• ELM323/327 Datasheet

Instrument Cluster Screenshots

At the SXSW Interactive Opening Party hosted by Frog Design and Microsoft, the Coding4Fun team has unveiled their latest invention, BoxingBots!

With BoxingBots you are the controller except this time, forget the screen. You’re controlling a 6-foot tall metal robot that’s trying to punch out your opponent’s 6-foot tall machine. It’s Real Steel in real time.

Video Demo

Watch Laura Foy's video of the BoxingBots in action!

BoxingBots combines Kinect’s motion sensing technology with pneumatic robots in a 6-foot square ring. Two players face off to try to knock each other’s blocks off.

Each robot has

• 1 On-board laptop PC
• 70 programmatic LEDs in the head and chest controlled by a Phidgets LED controller board
• 2 2.5 gallon tanks of compressed air @ 150 PSI
• 8 12-volt 17amp batteries
• 2 Roboteq motor controllers that power the four omni-directional wheels
• 2 5" stroke cylinders that punches/retracts the arms
• 1 6' stroke cylinder that lifts the head when the robot is knocked out
• 2 button switches on the chest that are triggered by punches and a Phidget 8/8/8 interface board that controls the relays and buttons switches

How gameplay happens

• Each robot's on-board PC receive commands over the network from the Admin PC for movement, punches, score, etc
• Each player stands in front of a Kinect for Windows-enabled PC that interprets their gestures (punch, move in a direction, strafe, rotate) and sends that as a command to the Admin PC
• The Admin PC manages the game, including pausing, reset, starting, sending commands to the robots and receiving damage data from the robots when the robot has been punched

Photos