Project Natal + Windows

aL_ said:

Bas said:

*snip*

i think 3dv built the depth maera, they apparently had a some sort of 3d webcam lined up but then ms bought them (see http://en.wikipedia.org/wiki/ZCam) in any case, i hope licensing wont be a problem

the face recognition login would be really great, but also think of what you could do with the POV/head tracking.. you could make sure messages appear where the user is (or is not) looking [on a multimon system for example] you could "peek" outside the edge of your desktop by meaning to the opposite direction

also, because you sit much closer to your pc than your tv, the depth resolution (in x/y) should be higher so maybe you can have it track your hands and fingers.. if thats possible, imagine the a 3d modeling app where you actually use your hands to manipulate your model just beeing able to do minority report type manipulation on the pc would be great

maybe i shouldnt get to exited yet though the word is still out on accuracy and resoultion i suppose

i think 3dv built the depth maera, they apparently had a some sort of 3d webcam lined up but then ms bought them (see http://en.wikipedia.org/wiki/ZCam) in any case, i hope licensing wont be a problem

Hmm... It would seem that MS does own Natal entirely -- who knows... IANAL...

But that does bode well for having a cheap full-body motion controller for Windows...

Bass said:

I just discovered that Natal is like, not coming out for a long time. Seriously what is it with Microsoft and annoncing things like years in advance? When Apple announces a product, it often comes out the next day.

I actually prefer having this knowledge. Yes, it's like waiting for the candystore to open but if you want ignorance, just turn off the PC

Bas said:

aL_ said:

*snip*

I'm guessing it's like the wiimote, which has an IR camera, but you have no way of getting the raw IR image. There's an onboard processor that analyses the image from the IR camera, does blob tracking and stuff like that, and then gives out an image with up to four dots. I'm guessing Natal has a similar processor that gives you the individual joints it sees and perhaps an ID for the face it recognizes (as well as a simple RGB image, see the video chat or the reflection in the water in the Milo demo), but not the actual raw grayscale depth-image.

possebly allthough you can actually get the ir image from the wiimote, you can see it in the wii when you adjust the brightness of the wiimote camera.

it seems likely that they would make the blob identificationon the device using an fpga or something but i wonder if they do the joint calculations of there..

come on microsoft give us more info you cant drop a bomb like that on a nerd like me and then not provide more info i wonder if this was the secret thing beckman was working on or if that is something diffrent

Minh said:

Minh said:

*snip*

Have you guys seen Sony's answer to the motion controller question?

OK, first of all, augmented reality is cool...!!!

Secondly, I think Sony's solution has a better chance of creating good games than Natal,  at least in the near-term... It's really like a WiiMote... with much greater pointing range... since w/ the WiiMote, the camera inside it must see the IR lights in the sensorbar.

I think Natal is going to miss having a trigger... Unless all Natal games will be of the Wii Fit variety...

I'm hoping what we're seeing in Natal isn't the final version

I can't believe that there won't be something like the sony/wii controller being added to Natal.  People like props, and you'll need something extra for really fine precision, but I think the the Natal technology is a better base to build on.

PerfectPhase said:

Minh said:

*snip*

I can't believe that there won't be something like the sony/wii controller being added to Natal.  People like props, and you'll need something extra for really fine precision, but I think the the Natal technology is a better base to build on.

In case of stuff like racing games.. why not just hold something round? I mean, the Wii racing wheels are really just plastic wheels with a controller in them. With natal you wouldn't even need the controller, just the plastic wheel.

Bas said:

PerfectPhase said:

*snip*

In case of stuff like racing games.. why not just hold something round? I mean, the Wii racing wheels are really just plastic wheels with a controller in them. With natal you wouldn't even need the controller, just the plastic wheel.

I remember way, way back I used to play Need for Speed with the Microsoft Sidewinder Freestyle Pro (basically a gamepad with sensors in it so you could use it as an analog controller by moving it around). It didn't work so well, mainly because there's no feedback, no resistance, no indication when you are steering as much as possible. Steering wheels don't really work for me unless they're attached to something.

Not to mention that for arcade racing games, a regular d-pad controller or even the keyboard will almost always be easier and allow you to be faster. Realistic controls are only suited for realistic games.

My guess is it could be to avoid patent issues, although there are other companies that make depth cameras too.

One thing I couldn't really find an answer to is what part of the Milo demo was Milo and what part was Natal. The way it could see your expression, for instance.. was that all processing of the raw camera image by the Lionhead guys or was part of that processing done for them by natal, like the facial recognition and joint tracking?

I wonder if the tech could be scaled down to work on handheld devices.  Multi-touch is all very nice, but it would be great if you didn't have to touch the screen for it work. 

Ray7 said:

I wonder if the tech could be scaled down to work on handheld devices.  Multi-touch is all very nice, but it would be great if you didn't have to touch the screen for it work. 

Unless they are using some really unconventional algorithm then I say no, because the issue with machine vision algorithms tend to be memory size and memory bandwidth (and also CPU to an extent), and handheld's current resources don't cut it for most things except the most trivial.

I think even from Project Natal they are using some of the Xbox 360's resources anyway, because it would seem very expensive to put all this processing on a ASIC. This isn't like a Wiimote which is doing something very trivial algorithmically. But if Natal is done in software it would make writing a driver quite hard.

Really the demo they did makes it look like what they are doing is easy, but it's really really hard. TBH I wouldn't be suprised if it didn't work nearly as well as does in the demo when it's finally released.

Bass said:

Ray7 said:

*snip*

Unless they are using some really unconventional algorithm then I say no, because the issue with machine vision algorithms tend to be memory size and memory bandwidth (and also CPU to an extent), and handheld's current resources don't cut it for most things except the most trivial.

I think even from Project Natal they are using some of the Xbox 360's resources anyway, because it would seem very expensive to put all this processing on a ASIC. This isn't like a Wiimote which is doing something very trivial algorithmically. But if Natal is done in software it would make writing a driver quite hard.

Really the demo they did makes it look like what they are doing is easy, but it's really really hard. TBH I wouldn't be suprised if it didn't work nearly as well as does in the demo when it's finally released.

if not something like a cell phone, maybe a netbook? a netbook with a somewhat powerful gpu [assuming gpu acceleration here] sould be able to handle the natal..

im also slightly worried about the push for have no controller at all by microsoft.. [as mentioned earlier in this thread] i think you do need some sort of artifact. in real life you're not waving your arms around in this air when driving, you got a wheel, when using a gun you have a trigger etc, its actually less realistic to just have nothing at all

sony in their presentation mention that there are just some thing you need a button for and i think thats more realistic in terms of how games actually work.. from looking at the videos of natal it looks like theres like 250-500ms lag [totaly unscientific and a pure guesstimate] and thats not good enough for shooting games..

then again there is absolutly nothing stopping people from using the natal in conjunction with a regular controller it also remains to be seen if the natal can track other things that humans, like wheels or swords or whatever, it a tad unsetteling that the joint tracking demos doesnt include a hand joint.. finger joints may be to much to ask for but not tracking the hand joint would make sword swinging a lot less cool :/ compare that with the sony setup where basically all they track is the hand joint. its far less advanced but looks more responsive and accurate.. i do hope im wrong about that though

aL_ said:

Bass said:

*snip*

if not something like a cell phone, maybe a netbook? a netbook with a somewhat powerful gpu [assuming gpu acceleration here] sould be able to handle the natal..

im also slightly worried about the push for have no controller at all by microsoft.. [as mentioned earlier in this thread] i think you do need some sort of artifact. in real life you're not waving your arms around in this air when driving, you got a wheel, when using a gun you have a trigger etc, its actually less realistic to just have nothing at all

sony in their presentation mention that there are just some thing you need a button for and i think thats more realistic in terms of how games actually work.. from looking at the videos of natal it looks like theres like 250-500ms lag [totaly unscientific and a pure guesstimate] and thats not good enough for shooting games..

then again there is absolutly nothing stopping people from using the natal in conjunction with a regular controller it also remains to be seen if the natal can track other things that humans, like wheels or swords or whatever, it a tad unsetteling that the joint tracking demos doesnt include a hand joint.. finger joints may be to much to ask for but not tracking the hand joint would make sword swinging a lot less cool :/ compare that with the sony setup where basically all they track is the hand joint. its far less advanced but looks more responsive and accurate.. i do hope im wrong about that though

I think a low-mid powered GPU, with about 256 MB of very high bandwidth memory (typical of what is in video cards anyway) should be sufficent if dedicated to this kind of work. It's not something you'd find in a cellphone today, but maybe in a few years. Actually with these kind of intelligent algorithms proves we still need Moore's law to continue working if we want to advance software.

aL_ said:

Bass said:

*snip*

if not something like a cell phone, maybe a netbook? a netbook with a somewhat powerful gpu [assuming gpu acceleration here] sould be able to handle the natal..

im also slightly worried about the push for have no controller at all by microsoft.. [as mentioned earlier in this thread] i think you do need some sort of artifact. in real life you're not waving your arms around in this air when driving, you got a wheel, when using a gun you have a trigger etc, its actually less realistic to just have nothing at all

sony in their presentation mention that there are just some thing you need a button for and i think thats more realistic in terms of how games actually work.. from looking at the videos of natal it looks like theres like 250-500ms lag [totaly unscientific and a pure guesstimate] and thats not good enough for shooting games..

then again there is absolutly nothing stopping people from using the natal in conjunction with a regular controller it also remains to be seen if the natal can track other things that humans, like wheels or swords or whatever, it a tad unsetteling that the joint tracking demos doesnt include a hand joint.. finger joints may be to much to ask for but not tracking the hand joint would make sword swinging a lot less cool :/ compare that with the sony setup where basically all they track is the hand joint. its far less advanced but looks more responsive and accurate.. i do hope im wrong about that though

I do not know why the demos showed the skeletal tracking with only those few points. Supposedly it tracks 48 points per body (just Bing "Project Natal 48 points"). So I was trying to figure out what those 48 points might be, and it seems like it should include the fingers, or at least the hand joint.  Unless this includes the face (for facial expression).  Then again if it can go down as fine as facial expression, I can't see why they can't also include fingers.  Maybe the prototypes at this point only track as many points.

I also don't see why they can't scan any object of your desire, like a tennis racquet, sword, whatever, and allow you to use that as part of the tracking.

But the lag will need to be fixed. I think with some smart algorithms they can actually reduce lag to the point where it becomes insignificant.  Here is one example I was thinking of: Since Natal contains both an RGB camera as well as the depth camera, they can be made to complement each other.  The RGB camera frames are 2D and require much less post-frame processing, whereas the depth camera requires much more post-frame processing (presumably).  Let's say it is a boxing game.  If the player has their fist close to their body, the skeletal mapping will clearly show the elbow sticking out far from the fist (if the player is facing the camera).  Internally, the 3D skeleton is mapped to points on the RGB 2D image. Now, let's say the player gives a fast stab with their fist. The elbow starts moving "towards" the fist and eventually they are in line, with the elbow now "behind" the fist. Let’s say it takes some time for the post 3D image to be processed to determine this.  But the RGB camera captured this change as well. Using much simpler (faster) 2D image processing algorithms, it can track the fact that the elbow is moving "towards" the fist.  Since Natal knows the dimensions of the player's skeletal parts, it knows that the only way this can happen is if the fist actually moved towards the camera.  So using that knowledge, it can make assumptions about how parts of the body are moving before the depth algorithms are completed. Once the 3D processing completes, it can correct for any errors in the 2D processing, but it should be quite accurate if the 3D to 2D mapping algorithm is doing a good job.

Then again, the final product's processing might be fast enough, and the algorithms might be refined enough that it is fast enough to begin with.

I don't necessarily see this as completely replacing the 360 controller, more like adding additional input.  For instance, let's say in COD6, there is a menu option that becomes enabled if it detects Project Natal.  In that menu, you can select to map certain body motions to certain functions.  Like lean left/right to strafe left/right, or a sudden fist jab to simulate a melee attack, etc. You still play with your controller, but now you can have some additional control.

I was thinking of complete FPS control like this:

• Walk forward/back: Lean slightly forward/back
• Run forward/back: Lean left/right more
• Strafe left/right: Lean slightly left/right
• Turn left/right: Rotate your shoulders left/right.
• Look left/right: Rotate your head slightly left/right
• Look up/down: Tilt your head slightly up/down
• Aim (zoom): Lift your hands slightly up
• Fire: Flick your fingers (not 100% sure about that one)
• Reload: Fist over fist (a rolling motion with two fists)
• Switch weapons: Right hand over right shoulder
• Switch grenades: Left hand over left shoulder
• Melee: Stab/swing with your fist. This could actually mirror what you are doing: Left/right stabs, uppercuts, smash from above, etc

Notice that you can do all of these while still sitting on your couch. Of course you could still use your controller and just replace some of the controls with the motion versions. And also notice that it is optional, so that those that don't have Natal can still play the game.

BitFlipper said:

aL_ said:

*snip*

I do not know why the demos showed the skeletal tracking with only those few points. Supposedly it tracks 48 points per body (just Bing "Project Natal 48 points"). So I was trying to figure out what those 48 points might be, and it seems like it should include the fingers, or at least the hand joint.  Unless this includes the face (for facial expression).  Then again if it can go down as fine as facial expression, I can't see why they can't also include fingers.  Maybe the prototypes at this point only track as many points.

I also don't see why they can't scan any object of your desire, like a tennis racquet, sword, whatever, and allow you to use that as part of the tracking.

But the lag will need to be fixed. I think with some smart algorithms they can actually reduce lag to the point where it becomes insignificant.  Here is one example I was thinking of: Since Natal contains both an RGB camera as well as the depth camera, they can be made to complement each other.  The RGB camera frames are 2D and require much less post-frame processing, whereas the depth camera requires much more post-frame processing (presumably).  Let's say it is a boxing game.  If the player has their fist close to their body, the skeletal mapping will clearly show the elbow sticking out far from the fist (if the player is facing the camera).  Internally, the 3D skeleton is mapped to points on the RGB 2D image. Now, let's say the player gives a fast stab with their fist. The elbow starts moving "towards" the fist and eventually they are in line, with the elbow now "behind" the fist. Let’s say it takes some time for the post 3D image to be processed to determine this.  But the RGB camera captured this change as well. Using much simpler (faster) 2D image processing algorithms, it can track the fact that the elbow is moving "towards" the fist.  Since Natal knows the dimensions of the player's skeletal parts, it knows that the only way this can happen is if the fist actually moved towards the camera.  So using that knowledge, it can make assumptions about how parts of the body are moving before the depth algorithms are completed. Once the 3D processing completes, it can correct for any errors in the 2D processing, but it should be quite accurate if the 3D to 2D mapping algorithm is doing a good job.

Then again, the final product's processing might be fast enough, and the algorithms might be refined enough that it is fast enough to begin with.

I don't necessarily see this as completely replacing the 360 controller, more like adding additional input.  For instance, let's say in COD6, there is a menu option that becomes enabled if it detects Project Natal.  In that menu, you can select to map certain body motions to certain functions.  Like lean left/right to strafe left/right, or a sudden fist jab to simulate a melee attack, etc. You still play with your controller, but now you can have some additional control.

I was thinking of complete FPS control like this:

• Walk forward/back: Lean slightly forward/back
• Run forward/back: Lean left/right more
• Strafe left/right: Lean slightly left/right
• Turn left/right: Rotate your shoulders left/right.
• Look left/right: Rotate your head slightly left/right
• Look up/down: Tilt your head slightly up/down
• Aim (zoom): Lift your hands slightly up
• Fire: Flick your fingers (not 100% sure about that one)
• Reload: Fist over fist (a rolling motion with two fists)
• Switch weapons: Right hand over right shoulder
• Switch grenades: Left hand over left shoulder
• Melee: Stab/swing with your fist. This could actually mirror what you are doing: Left/right stabs, uppercuts, smash from above, etc

Notice that you can do all of these while still sitting on your couch. Of course you could still use your controller and just replace some of the controls with the motion versions. And also notice that it is optional, so that those that don't have Natal can still play the game.

I think you are making an incorrect assumption saying 3D image processing algorithms are more computationally more difficult. Edge detection is can be done with 3D data without almost any effort because of the added Z dimension, but with 2D data requires convolutions of spatial frequencies. Convolutions are very, very slow.

With a real 3D camera, you have more information to make correct assumptions with less processing. This assumes the hardware actually includes a 3D camera, a piece of hardware which actually returns 3D vectors, because in reality such hardware is very expensive, out the realm of consumer electronics. 

Bass said:

BitFlipper said:

*snip*

I think you are making an incorrect assumption saying 3D image processing algorithms are more computationally more difficult. Edge detection is can be done with 3D data without almost any effort because of the added Z dimension, but with 2D data requires convolutions of spatial frequencies. Convolutions are very, very slow.

With a real 3D camera, you have more information to make correct assumptions with less processing. This assumes the hardware actually includes a 3D camera, a piece of hardware which actually returns 3D vectors, because in reality such hardware is very expensive, out the realm of consumer electronics. 

Project Natal includes both a 2D RGB camera as well as a 3D time-of-flight camera. This is what makes it so amazing for something that is expected to cost around $200 (not much more than a Guitar Hero set). From your post it sounds like you don't believe that this is the case (that it is a true 3D camera where each pixel contains the true depth at that point). This fact has been pretty much confirmed. This is not a bad place to start:

http://procrastineering.blogspot.com/2009/06/project-natal.html

Also, I realize that for the grayscale depth image returned by the 3D camera, it is computationally easy to find edges.  But this is not the only thing Natal does. It maps that info into multiple skeletal points, and that is where the heavy processing is.  What I was proposing was that if that part of the processing is what is really causing the lag, that using additional, simpler 2D/3D overlay tracking could help to reduce lag since the software can make assumptions about how the skeleton is moving (from the 2D image) before the heavy 3D processing is able to even calculate the true position of the skeleton.

BitFlipper said:

Bass said:

*snip*

Project Natal includes both a 2D RGB camera as well as a 3D time-of-flight camera. This is what makes it so amazing for something that is expected to cost around $200 (not much more than a Guitar Hero set). From your post it sounds like you don't believe that this is the case (that it is a true 3D camera where each pixel contains the true depth at that point). This fact has been pretty much confirmed. This is not a bad place to start:

http://procrastineering.blogspot.com/2009/06/project-natal.html

Also, I realize that for the grayscale depth image returned by the 3D camera, it is computationally easy to find edges.  But this is not the only thing Natal does. It maps that info into multiple skeletal points, and that is where the heavy processing is.  What I was proposing was that if that part of the processing is what is really causing the lag, that using additional, simpler 2D/3D overlay tracking could help to reduce lag since the software can make assumptions about how the skeleton is moving (from the 2D image) before the heavy 3D processing is able to even calculate the true position of the skeleton.

At the bottom it says : "Product vision. Actual features may vary." or something similar to that.

If Microsoft has a 2D camera that estimates 3D data, that's not the same thing. In practice if you want accurate 3d data you need a 3d range finder, a piece of hardware that returns 3d vectors directly, based on lasers and what not.

Trust me I would like nothing more then to be able to purchase a REAL 3d range finder for $200. I have a lot of ideas personally that could benefit from a $200 range finder. If Microsoft could somehow make it happen with some crazy economics of scale or whatever the hell that would be amazing, but the fact of the matter is right now they cost thousands of dollars. That would be a massive drop in price. Seems almost unrealistic.

Bass said:

BitFlipper said:

*snip*

At the bottom it says : "Product vision. Actual features may vary." or something similar to that.

If Microsoft has a 2D camera that estimates 3D data, that's not the same thing. In practice if you want accurate 3d data you need a 3d range finder, a piece of hardware that returns 3d vectors directly, based on lasers and what not.

Trust me I would like nothing more then to be able to purchase a REAL 3d range finder for $200. I have a lot of ideas personally that could benefit from a $200 range finder. If Microsoft could somehow make it happen with some crazy economics of scale or whatever the hell that would be amazing, but the fact of the matter is right now they cost thousands of dollars. That would be a massive drop in price. Seems almost unrealistic.

It seems like you think know everything, except you choose to ignore the technology in Project Natal. Why would you do that? Instead of aguing with me about it and stating as fact that it does not contain a 3D camera, why don't you do the basic research? You bothered to post a reply here, why not also do a quick search on the subject to get more up to speed?

I realize you are very anti-MS, but that is no reason to be in denial about the technology in Project Natal.  It doesn't take much searching to find all sorts of information about the camera technology in Natal, which proves you 100% wrong.  Since you refuse to do that little bit of reaserch, I'll do it for you, and hopefully you will stop being in denial.

"The 3D sensor itself is a pretty incredible piece of equipment providing detailed 3D information about the environment similar to very expensive laser range finding systems but at a tiny fraction of the cost. Depth cameras provide you with a point cloud of the surface of objects that is fairly insensitive to various lighting conditions allowing you to do things that are simply impossible with a normal camera.

But once you have the 3D information, you then have to interpret that cloud of points as "people". This is where the researcher jaws stay dropped. The human tracking algorithms that the teams have developed are well ahead of the state of the art in computer vision in this domain. The sophistication and performance of the algorithms rival or exceed anything that I've seen in academic research, never mind a consumer product."
http://procrastineering.blogspot.com/2009/06/project-natal.html

"The device features an "RGB camera, depth sensor, multi-array microphone, and custom processor running proprietary software", which provides full-body 3D motion capture, facial recognition, and voice recognition capabilities. The Project Natal sensor's microphone array enables the Xbox 360 to conduct acoustic source localization and ambient noise suppression, allowing for things such as headset-free party chat over Xbox Live.

The depth sensor consists of an infrared projector combined with a monochrome CMOS sensor, and allows the Project Natal sensor to see in 3D under any ambient light conditions. The active depth-sensing range of the depth sensor is adjustable, with the Project Natal software capable of automatically calibrating the sensor based on the gameplay and environment conditions, such as the presence of couches."
http://en.wikipedia.org/wiki/Project_Natal

This link shows a video demo of the depth camera that was created by 3DV Systems, which is the company MS bought to get the hardware for Project Natal. BTW, research into Project Natal was going on years before they bought 3DV Systems. It is much more than just the 3D camera.
http://www.3dvsystems.com/gallery/movies/Face.mpg

An interview with Shane Kim, corporate vice president for strategy and business development at Microsoft’s game division:
"It uses an RGB camera (image sensor), a 3-D depth camera (which determines how far away an object is from the camera), and a multi-array microphone."
http://venturebeat.com/2009/06/02/microsoft-games-executive-describes-origins-of-project-natal-game-controls/