Coffeehouse Post

Single Post Permalink

View Thread: I love you, Kinect.
  • User profile image

    Alright, if this public Skydrive folder thing works, here's the source in!212

    The way the application works is that it displays whatever the RGB camera sees, and tracks the skeletons of up to two people in the frame, and checks which of their joints are closer than a certain distance to the Kinect sensor. Those joints should cause a ripple in the video image, so the joints 3D coordinates are converted to 2D pixel coordinates corresponding to the joint's position in the video image.

    At these 2D coordinates, a ripple pixel shader effect is displayed. Pixel shaders are small C procedures that run on the GPU, and I have no idea how they work exactly, so I downloaded Shazzam and let it generate a nice C# wrapper class for me. Shazza also allows you to test the shaders, so I played around with the provided Ripple pixel shader's properties. I decided that each ripple effect would start with a certain frequency (the number of ripples in the effect) that would over time decrease to a frequency of zero meaning no ripples. This would give the impression of the ripples calming down again.

    Each ripple effect also has an amplitude, which deterines how 'deep' the ripples are. A light touch should cause a light ripple and a deep punch should cause a wild, deep ripple, so the closer the joint is to the sensor, the higher the ripple's amplitude.

    The biggest problem was applying multiple pixel shader effects to the same image. The WPF Image control, like many others, has an Effect property, but that only takes a single pixel shader effect. The only way I could find to apply two or more pixel shaders to the image was by wrapping the Image in a container (I chose a border but I guess it could be  a Canvas or whatever) and setting the Effect property of that container. Then, when another ripple needs to be added, we create another border to wrap the first border (which wraps the Image), set the new border's Effect, and so on. This results in a deep nested tree of borders, the very deepest of which cointains the Image control. As soon as a ripple animation as finished, I remove the border it was applied to. The application stays remarkably performant, despite the fact that, if you go crazy waving your arms and legs, it can contain up to 50 nested borders, each with animating pixel shader, wrapping the single image of you in front of the Kinect.

    I haven't had time to clean up the code yet, I basically wrote it from scratch non-stop in one evening, so some bits are kind of dodgy, but it still works. If anyone has any further questions, let me know.