Kinect, HTML5, WebSockets and Canvas

Description

The theme for this week seems to be working out to a "let's code" week. Today's project is one that's pretty unique and something I've seen only a couple times and I think this is the first that used the Kinect for Windows SDK and included the source too...

Kinect & HTML5 using WebSockets and Canvas

Kinect defined Natural User Interaction. HTML5 redefined the Web. Currenty, there are various tutorials describing how to interact with a Kinect sensor using Windows Forms or WPF for the user interface. But what about using a web interface for handling Kinect data? Trying to combine those two hot, cutting-edge technologies, I came up with a pretty and open-source solution, which I am going to describe in this blog post.

I am going to use the official Kinect SDK for my demo...

The project consists of two sub-projects: A server-side application which uses Kinect SDK and a client-side web page displaying the skeleton joints on an HTML5 canvas.

...

Tutorial

Here is, step by step, a way to achieve the above functionality:

Step 1: Server application

The server application's job is straightforward: Detect the users' joints, pack the data and send them to the clients using web sockets.

...

After collecting, packing and transmitting the data, clients can now consume and process them accordingly.

Step 2: Client application

Time for HTML5 bits! We have the server set up, so let's add the web page clients. HTML5 spec recommends web socket support, currently implemented by Internet Explorer 10, Google Chrome and Mozilla Firefox. Web sockets are great for direct message communication between server and clients.

All we need is some JavaScript for receiving the server data and a canvas for drawing the points. Here is the JavaScript event handler for getting the data and drawing the joints to a <canvas> element:

...

Step 3: Mixin it up!

Now run the server application and then open the HTML file in Internet Explorer 10 or Google Chrome. Stand in front of a Kinect sensor and may the force be with you.

Project Information URL: http://studentguru.gr/b/vangos/archive/2012/01/01/kinect-amp-html5-using-websockets-and-canvas.aspx

Project Download URL: http://studentguru.gr/cfs-file.ashx/__key/communityserver-components-postattachments/00-00-13-43-27/KinectHtml5.zip

Project Source URL: http://studentguru.gr/cfs-file.ashx/__key/communityserver-components-postattachments/00-00-13-43-27/KinectHtml5.zip

image

image

static void Nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{

    List users = new List();

    foreach (var user in e.SkeletonFrame.Skeletons)
    {
        if (user.TrackingState == SkeletonTrackingState.Tracked)
        {
            users.Add(user);
        }
    }

    if (users.Count > 0)
    {
        string json = users.Serialize();

        foreach (var socket in _sockets)
        {
            socket.Send(json);
        }
    }
}

_sockets = new List();

var server = new WebSocketServer("ws://localhost:8181");

server.Start(socket =>
{
    socket.OnOpen = () =>
    {
        Console.WriteLine("Connected to " + socket.ConnectionInfo.ClientIpAddress);
        _sockets.Add(socket);
    };
    socket.OnClose = () =>
    {
        Console.WriteLine("Disconnected from " + socket.ConnectionInfo.ClientIpAddress);
        _sockets.Remove(socket);
    };
    socket.OnMessage = message =>
    {
        Console.WriteLine(message);
    };
});

socket.onmessage = function (evt) {
    status.innerHTML = "Kinect data received.";

    // Get the data in JSON format.
    var jsonObject = eval('(' + evt.data + ')');

    context.clearRect(0, 0, canvas.width, canvas.height);
    context.fillStyle = "#FF0000";
    context.beginPath();

    // Display the skeleton joints.
    for (var i = 0; i < jsonObject.skeletons.length; i++) {
        for (var j = 0; j < jsonObject.skeletons[ i ].joints.length; j++) {
            var joint = jsonObject.skeletons[ i ].joints[ j ];

            // Draw!!!
            context.arc(parseFloat(joint.x), parseFloat(joint.y), 10, 0, Math.PI * 2, true);
        }
    }

    context.closePath();
    context.fill();
};

Contact Information:

The Discussion

  • User profile image
    3lawi_m

    i like kinect but i'am not diving into it. is it possible to avoid using web socket and extend phonegap to do the interaction with kinect so its able to the run on mobile paltforms?

    i think its possible. am i right?

Comments closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to send us feedback you can Contact Us.