Fact: using multiple pixel shaders in WPF is harder than tracking a freaking skeleton in three dimensions.

Edit: source code provided below.

But now everyone knows what you really look like ....

Herbie

But now everyone knows what you really look like ....

And that he has exactly the same Ikea chair as I do.

Aye captain!

7 hours ago, Bas wrote

Fact: using multiple pixel shaders in WPF is harder than tracking a freaking skeleton in three dimensions.

Which Kinect library are you using? And is the RGB video from the Kinect device exposed as a DirectShow stream or something else?

So far all the MIDI controller apps use OpenNI and some of the earlier "hacks" so I haven't been able to do a whole lot with it. Of course I've only had it for 3 days

@Sven: heh, I've seen that chair in so many places. When I first got my Roomba it got stuck on the legs, so I figured that seeing how many of those chairs I've seen, there must be someone else out there with a Roomba/Poäng combination, and sure enough as soon as I had typed "Roomba poa" in Google a whole bunch of threads about the problem rolled out. I love IKEA.

@Maddus: that's so uncanny! When I saw it I thought "hey, I don't remember this picture at OH MY GOD!"

@W3bbo: I'm using the official Kinect SDK. The RGB video is exposed as a bunch of PlanarImage objects for each frame, that you can poll or subscribe to an event that returns one as soon as it's ready. The PlanarImage isn't specific to any framework, but you can get to the bits and other data easily and convert it to whatever image format you need. It's not entirely convenient, but an extension method is easily written and it's still pretty versatile this way.

@dentaku: I still want to clean up the code a bit, and I'm too busy this weekend, but with a bit of luck I can post something later tonight. Otherwise probably monday.

@cbae: disco!

9 minutes ago, Bas wrote

@Ian2: sheer force of will.

@Sven: heh, I've seen that chair in so many places. When I first got my Roomba it got stuck on the legs, so I figured that seeing how many of those chairs I've seen, there must be someone else out there with a Roomba/Poäng combination, and sure enough as soon as I had typed "Roomba poa" in Google a whole bunch of threads about the problem rolled out. I love IKEA.

Poäng? Hey, I think I have that chair too!

16 minutes ago, Bas wrote

@W3bbo: I'm using the official Kinect SDK. The RGB video is exposed as a bunch of PlanarImage objects for each frame, that you can poll or subscribe to an event that returns one as soon as it's ready. The PlanarImage isn't specific to any framework, but you can get to the bits and other data easily and convert it to whatever image format you need. It's not entirely convenient, but an extension method is easily written and it's still pretty versatile this way.

Did you use the Coding4Fun libraries? I think I remember seeing some extension methods to simplify some of that stuff being demonstrated in Dan Fernandez's videos.

The way the application works is that it displays whatever the RGB camera sees, and tracks the skeletons of up to two people in the frame, and checks which of their joints are closer than a certain distance to the Kinect sensor. Those joints should cause a ripple in the video image, so the joints 3D coordinates are converted to 2D pixel coordinates corresponding to the joint's position in the video image.

At these 2D coordinates, a ripple pixel shader effect is displayed. Pixel shaders are small C procedures that run on the GPU, and I have no idea how they work exactly, so I downloaded Shazzam and let it generate a nice C# wrapper class for me. Shazza also allows you to test the shaders, so I played around with the provided Ripple pixel shader's properties. I decided that each ripple effect would start with a certain frequency (the number of ripples in the effect) that would over time decrease to a frequency of zero meaning no ripples. This would give the impression of the ripples calming down again.

Each ripple effect also has an amplitude, which deterines how 'deep' the ripples are. A light touch should cause a light ripple and a deep punch should cause a wild, deep ripple, so the closer the joint is to the sensor, the higher the ripple's amplitude.

The biggest problem was applying multiple pixel shader effects to the same image. The WPF Image control, like many others, has an Effect property, but that only takes a single pixel shader effect. The only way I could find to apply two or more pixel shaders to the image was by wrapping the Image in a container (I chose a border but I guess it could be  a Canvas or whatever) and setting the Effect property of that container. Then, when another ripple needs to be added, we create another border to wrap the first border (which wraps the Image), set the new border's Effect, and so on. This results in a deep nested tree of borders, the very deepest of which cointains the Image control. As soon as a ripple animation as finished, I remove the border it was applied to. The application stays remarkably performant, despite the fact that, if you go crazy waving your arms and legs, it can contain up to 50 nested borders, each with animating pixel shader, wrapping the single image of you in front of the Kinect.

I haven't had time to clean up the code yet, I basically wrote it from scratch non-stop in one evening, so some bits are kind of dodgy, but it still works. If anyone has any further questions, let me know.

By the way. If I install VS2010 Express will it mess with the Windows Phone 7 tools I already have installed. Any stories of incompatibilities?

I'm guessing it's VS2010Express1.iso that you install?

@dentaku: Performance kind of depends on where you run it, I suppose. The pixel shaders run on the GPU so it kind of depends on what kind of graphics card you have. It doesn't slow down noticably on my desktop, but I kind of see a drop on my laptop. You can mess with the maxRipples constant in MainWindow to decrease the number of ripples (and thus the number of nested Border elements, which I think are the main culprit) on the screen at one time and see if that helps. You can also increase minRippleDistance but I feel that it gets kind of noticable if you force them to be more than five pixels apart from each other.

Installing VC# Express shouldn't mess with the Windows phone tools, I have both running on my laptop. No idea which iso though, I always just download it from http://www.microsoft.com/express.

Installing VC# Express shouldn't mess with the Windows phone tools, I have both running on my laptop. No idea which iso though, I always just download it from http://www.microsoft.com/express.

OK, it's just that the webpage has a bunch of downloads listed on it. http://www.microsoft.com/visualstudio/en-us/products/2010-editions/express
The bottom link is an .ISO with everything included in it.

I have a Geforce 8800 GTS and a Core 2 Duo 6800 so I'm not running it on ancient hardware
It plays Portal 2 with 8x anti-aliasing at 1680X1050 just fine for example.
Once I have VS Express installed I'll mess with it a bit and see what effect lowering the ripples has.

I haven't updated my nvidia drivers in forever. Maybe I'll try that.

1 hour ago,CKurt wrote

@dentaku:Looks like you do not have enough RAM and the Hard Drive is used to extend it? Or you have a bad video driver.

I wonder if it's my C,F,G or H drive that's doing it?

20 hours ago,CKurt wrote

@dentaku: I'm wondering why there might be so much hard disk activity. The activity is only when the effect needs to be applied, not when the app is running normally?

 

C:\$LogFile (NTFS Volume Log) gets written to allot while the drives are going. That's the only consistent thing.

Herbie

13 minutes ago,Dr Herbie wrote

@SlackmasterK: Congrats on the exposure.  I saw Kinnect NUI on Codeplex but haven't given it a whirl yet. Can you get a version that works when the user is sitting at the desktop? I want to use by PC as normal, but having the Kinect as an augmentation to keyboard and mouse while I'm sitting there (e.g. wave away a notification).

Herbie

uhh, maybe?  The project relies on the SDK and Kinect to provide skeletal information, and it tends to go all wonky when you sit down. Especially with a desk blocking you.  Haven't tried it without a desk in the way.

1 hour ago,Bas wrote

I've found that it works reasonably well if you sit down, but only when it has seen you stand before.

I think the problem is it needs to see the whole skeleton before it can see the skeleton.

Come to think of it, this gadget might my useful for SlackmmasterK's workspace.

4 hours ago,SlackmasterK wrote

Looks like my own Kinect project made it onto Engadget.  Unfortunately they took the video I made for Chris Pirillo and not one of my own... but oh well.

That link no goes directly to the Engadget home page ... I don't see your video on there.   I have checked out yourYouTube videos, however, and am subscribed.  Good work so far.  Can't wait to see what's coming. 

http://www.engadget.com/2011/06/27/kinectnui-enables-minority-report-style-interaction-in-windows-s/

Herbie

maybe a kinect that has become self aware :O