But also, id like to start a petition to get windows drivers for this thing.. i dont own a xbox 360 but i'd buy the natal in a heartbeat if i could program against it and i know im not alone..

yes i know the natal is not even beta yet, but the sooner we make our voices heard the better

--edit--

in case you havent heard, billy G himself reckons natal is heading to windows

http://news.cnet.com/8301-13860_3-10286309-56.html

I mean, MS didn't invent this technology, they are licensing it. And the licensing fees usually depend on how widely the gadget is used...

If MS paid $X to have it on the Xbox, then MS would have to pay $X + $Y to also have it be used on Windows...

This is different than the Xbox joypad controller that MS made available on the PC also because the joypad is as commoditized as it gets -- even though they did have to shell out some more dough for rumble...

We'll just have to see I guess... Although if you see games on XBLA using Natal, then it will probably mean it'll be available for XNA... which will probably mean it'll be available on the PC

Unless some genius crack the protocol. Is Brian Peek already on it? haha

Minh said:

I wonder if this will be possible...

I mean, MS didn't invent this technology, they are licensing it. And the licensing fees usually depend on how widely the gadget is used...

If MS paid $X to have it on the Xbox, then MS would have to pay $X + $Y to also have it be used on Windows...

This is different than the Xbox joypad controller that MS made available on the PC also because the joypad is as commoditized as it gets -- even though they did have to shell out some more dough for rumble...

We'll just have to see I guess... Although if you see games on XBLA using Natal, then it will probably mean it'll be available for XNA... which will probably mean it'll be available on the PC

Unless some genius crack the protocol. Is Brian Peek already on it? haha

as far as i understand ms bought the company [3DV] that built the 3d camera so that should be fine.. in any case id still have to buy a retail natal [producing royalties for whoever], its not like they know i dont have an xbox

but still i see how that could be a problem.. i hope it isnt though

i hear the software is the really tough bit in the natal, taking the point cloud and extracting a trackable skeleton from it seems like heavy stuff.. check out this blog post from the awsome johnny lee: http://procrastineering.blogspot.com/2009/06/project-natal.html

--edit--

the natal wikipedia page is coming along nicely
http://en.wikipedia.org/wiki/Natal_(Video_Game_controller) [the last parenthasies doesnt get included when you click the link for some reason]
those TOF cameras are pretty impressive i wonder if the tracking calculations are done on the natal it self or on the host.. [host seems likely though]

aL_ said:

Minh said:

*snip*

as far as i understand ms bought the company [3DV] that built the 3d camera so that should be fine.. in any case id still have to buy a retail natal [producing royalties for whoever], its not like they know i dont have an xbox

but still i see how that could be a problem.. i hope it isnt though

i hear the software is the really tough bit in the natal, taking the point cloud and extracting a trackable skeleton from it seems like heavy stuff.. check out this blog post from the awsome johnny lee: http://procrastineering.blogspot.com/2009/06/project-natal.html

--edit--

the natal wikipedia page is coming along nicely
http://en.wikipedia.org/wiki/Natal_(Video_Game_controller) [the last parenthasies doesnt get included when you click the link for some reason]
those TOF cameras are pretty impressive i wonder if the tracking calculations are done on the natal it self or on the host.. [host seems likely though]

 

according to this article:

http://seattletimes.nwsource.com/html/technologybrierdudleysblog/2009296568_e3_new_info_on_microsofts_nata.html

a PC version/compability of natal doesnt seem entierly out of the question!

Q: You've apparently drawn on the gesture recognition work that led to Microsoft's Surface computer.

A: I think it's very smart for people to look at Natal and think about natural user input and how it applies in front of your TV, how it might apply sitting in front of a laptop or sitting in front of a PC.

MS didn't invent this technology, they are licensing it.

ms bought the company [3DV] that built the 3d camera

As far as I know, 3DV didn't build Natal, and Microsoft didn't license it. It was built by Microsoft, and they bought 3DV to prevent others from licensing similar technology. See here. Even if the hardware is designed by 3DV, they didn't license the technology, they bought it. (3DV sold all their assets to Microsoft after all.) So using it on Windows shouldn't be a problem licensing-wise.

I'd love to see some drivers for it. It should be able to solve the old 'I don't want to sit with my arms outstretched' problem people apparently have with touchscreens. And using it for media center would be awesome. Flip through your albums, make a hush-hush gesture to turn down or mute the volume, throw up the horns to play your death-metal playlist...

Also, I'd want this to happen if only for the facial recognition. Sit down in front of your PC and hey, you're logged in. Sounds awesome.

Bas said:

*snip*

As far as I know, 3DV didn't build Natal, and Microsoft didn't license it. It was built by Microsoft, and they bought 3DV to prevent others from licensing similar technology. See here. Even if the hardware is designed by 3DV, they didn't license the technology, they bought it. (3DV sold all their assets to Microsoft after all.) So using it on Windows shouldn't be a problem licensing-wise.

I'd love to see some drivers for it. It should be able to solve the old 'I don't want to sit with my arms outstretched' problem people apparently have with touchscreens. And using it for media center would be awesome. Flip through your albums, make a hush-hush gesture to turn down or mute the volume, throw up the horns to play your death-metal playlist...

Also, I'd want this to happen if only for the facial recognition. Sit down in front of your PC and hey, you're logged in. Sounds awesome.

i think 3dv built the depth maera, they apparently had a some sort of 3d webcam lined up but then ms bought them (see http://en.wikipedia.org/wiki/ZCam) in any case, i hope licensing wont be a problem

the face recognition login would be really great, but also think of what you could do with the POV/head tracking.. you could make sure messages appear where the user is (or is not) looking [on a multimon system for example] you could "peek" outside the edge of your desktop by meaning to the opposite direction

also, because you sit much closer to your pc than your tv, the depth resolution (in x/y) should be higher so maybe you can have it track your hands and fingers.. if thats possible, imagine the a 3d modeling app where you actually use your hands to manipulate your model just beeing able to do minority report type manipulation on the pc would be great

maybe i shouldnt get to exited yet though the word is still out on accuracy and resoultion i suppose

aL_ said:

Bas said:

*snip*

i think 3dv built the depth maera, they apparently had a some sort of 3d webcam lined up but then ms bought them (see http://en.wikipedia.org/wiki/ZCam) in any case, i hope licensing wont be a problem

the face recognition login would be really great, but also think of what you could do with the POV/head tracking.. you could make sure messages appear where the user is (or is not) looking [on a multimon system for example] you could "peek" outside the edge of your desktop by meaning to the opposite direction

also, because you sit much closer to your pc than your tv, the depth resolution (in x/y) should be higher so maybe you can have it track your hands and fingers.. if thats possible, imagine the a 3d modeling app where you actually use your hands to manipulate your model just beeing able to do minority report type manipulation on the pc would be great

maybe i shouldnt get to exited yet though the word is still out on accuracy and resoultion i suppose

I can imagine a *LOT* of uses for this tech for a pc....

for example a disabled person who may have very limited use of arms and legs able to use one hand and or the face to control stuff at home... could be great for that kind of thing.

and yes i can see using it for media center and for pc games....

Dare I say this?   World Of Warcraft on a giant screen with this controller ???  I think that would be cool for WOW fans!

figuerres said:

aL_ said:

*snip*

I can imagine a *LOT* of uses for this tech for a pc....

for example a disabled person who may have very limited use of arms and legs able to use one hand and or the face to control stuff at home... could be great for that kind of thing.

and yes i can see using it for media center and for pc games....

Dare I say this?   World Of Warcraft on a giant screen with this controller ???  I think that would be cool for WOW fans!

WoW is repeptitve strain, people are always looking for ways to make stuff like grinding more efficent and automatic, not harder or more interactive.

Bass said:

I just discovered that Natal is like, not coming out for a long time. Seriously what is it with Microsoft and annoncing things like years in advance? When Apple announces a product, it often comes out the next day.

unless its not that far away consider win7, first it was going to come out some time in 2010 but now they're  thinking october.. also, they have the dev kits so it cant be that far off.. but maybe in just wishing

@disabilities

thats also a great use  i wonder if it can also track the point where the user is watching.. it seems sorta likely based on the demos they've shown. devices like that are usually really expensive but the natal is a mass market product so it will be available for everyone
please, mighty lords of channel9, talk to the natal team [and tape it ]

Bass said:

I just discovered that Natal is like, not coming out for a long time. Seriously what is it with Microsoft and annoncing things like years in advance? When Apple announces a product, it often comes out the next day.

Natal is not a product. Snow Leopard was announced, what, two years ago?

PaoloM said:

Bass said:

*snip*

Natal is not a product. Snow Leopard was announced, what, two years ago?

true but apple is the king of hype also the products are pretty diffrent.. having devkits must mean the at least got a somewhat finished api..

btw it seems natal is doing at least some processing internally [check out some of the info "dots" here], good for speed but possible bad programability.. i prefer speed in this case though. the speculation continues...

--edit--
the internal processor info is also on wikipedia.

Bass said:

I just discovered that Natal is like, not coming out for a long time. Seriously what is it with Microsoft and annoncing things like years in advance? When Apple announces a product, it often comes out the next day.

It's a platform though like, they're just saying "hey we made this cool stuff, sent out a bunch of dev kits, expect cool games in the future". I can't believe someone would complain about a company being open about their roadmap.  Apple being secretive isn't helping anything except Apple.

aL_ said:

PaoloM said:

*snip*

true but apple is the king of hype also the products are pretty diffrent.. having devkits must mean the at least got a somewhat finished api..

btw it seems natal is doing at least some processing internally [check out some of the info "dots" here], good for speed but possible bad programability.. i prefer speed in this case though. the speculation continues...

--edit--
the internal processor info is also on wikipedia.

I'm guessing it's like the wiimote, which has an IR camera, but you have no way of getting the raw IR image. There's an onboard processor that analyses the image from the IR camera, does blob tracking and stuff like that, and then gives out an image with up to four dots. I'm guessing Natal has a similar processor that gives you the individual joints it sees and perhaps an ID for the face it recognizes (as well as a simple RGB image, see the video chat or the reflection in the water in the Milo demo), but not the actual raw grayscale depth-image.

CreamFilling512 said:

Bass said:

*snip*

It's a platform though like, they're just saying "hey we made this cool stuff, sent out a bunch of dev kits, expect cool games in the future". I can't believe someone would complain about a company being open about their roadmap.  Apple being secretive isn't helping anything except Apple.

I'm sorry, but it's quite a rare occasion when I am actually excited about a Microsoft product. But wait, this isn't a product. Well at least it has a release date.

Minh said:

I wonder if this will be possible...

I mean, MS didn't invent this technology, they are licensing it. And the licensing fees usually depend on how widely the gadget is used...

If MS paid $X to have it on the Xbox, then MS would have to pay $X + $Y to also have it be used on Windows...

This is different than the Xbox joypad controller that MS made available on the PC also because the joypad is as commoditized as it gets -- even though they did have to shell out some more dough for rumble...

We'll just have to see I guess... Although if you see games on XBLA using Natal, then it will probably mean it'll be available for XNA... which will probably mean it'll be available on the PC

Unless some genius crack the protocol. Is Brian Peek already on it? haha

Have you guys seen Sony's answer to the motion controller question?

OK, first of all, augmented reality is cool...!!!

Secondly, I think Sony's solution has a better chance of creating good games than Natal,  at least in the near-term... It's really like a WiiMote... with much greater pointing range... since w/ the WiiMote, the camera inside it must see the IR lights in the sensorbar.

I think Natal is going to miss having a trigger... Unless all Natal games will be of the Wii Fit variety...

I'm hoping what we're seeing in Natal isn't the final version

Bas said:

aL_ said:

*snip*

I'm guessing it's like the wiimote, which has an IR camera, but you have no way of getting the raw IR image. There's an onboard processor that analyses the image from the IR camera, does blob tracking and stuff like that, and then gives out an image with up to four dots. I'm guessing Natal has a similar processor that gives you the individual joints it sees and perhaps an ID for the face it recognizes (as well as a simple RGB image, see the video chat or the reflection in the water in the Milo demo), but not the actual raw grayscale depth-image.

I'm guessing it's like the wiimote, which has an IR camera, but you have no way of getting the raw IR image

I think you do have access to the image buffer... You saw the facial recognition demo... I guess the camera can always hash your face and send it to the Xbox... but I think MS has bigger plans for the camera than just as a motion controller.

aL_ said:

Bas said:

*snip*

i think 3dv built the depth maera, they apparently had a some sort of 3d webcam lined up but then ms bought them (see http://en.wikipedia.org/wiki/ZCam) in any case, i hope licensing wont be a problem

the face recognition login would be really great, but also think of what you could do with the POV/head tracking.. you could make sure messages appear where the user is (or is not) looking [on a multimon system for example] you could "peek" outside the edge of your desktop by meaning to the opposite direction

also, because you sit much closer to your pc than your tv, the depth resolution (in x/y) should be higher so maybe you can have it track your hands and fingers.. if thats possible, imagine the a 3d modeling app where you actually use your hands to manipulate your model just beeing able to do minority report type manipulation on the pc would be great

maybe i shouldnt get to exited yet though the word is still out on accuracy and resoultion i suppose

i think 3dv built the depth maera, they apparently had a some sort of 3d webcam lined up but then ms bought them (see http://en.wikipedia.org/wiki/ZCam) in any case, i hope licensing wont be a problem

Hmm... It would seem that MS does own Natal entirely -- who knows... IANAL...

But that does bode well for having a cheap full-body motion controller for Windows...

Bass said:

I just discovered that Natal is like, not coming out for a long time. Seriously what is it with Microsoft and annoncing things like years in advance? When Apple announces a product, it often comes out the next day.

I actually prefer having this knowledge. Yes, it's like waiting for the candystore to open but if you want ignorance, just turn off the PC

Bas said:

aL_ said:

*snip*

I'm guessing it's like the wiimote, which has an IR camera, but you have no way of getting the raw IR image. There's an onboard processor that analyses the image from the IR camera, does blob tracking and stuff like that, and then gives out an image with up to four dots. I'm guessing Natal has a similar processor that gives you the individual joints it sees and perhaps an ID for the face it recognizes (as well as a simple RGB image, see the video chat or the reflection in the water in the Milo demo), but not the actual raw grayscale depth-image.

possebly allthough you can actually get the ir image from the wiimote, you can see it in the wii when you adjust the brightness of the wiimote camera.

it seems likely that they would make the blob identificationon the device using an fpga or something but i wonder if they do the joint calculations of there..

come on microsoft give us more info you cant drop a bomb like that on a nerd like me and then not provide more info i wonder if this was the secret thing beckman was working on or if that is something diffrent

Minh said:

Minh said:

*snip*

Have you guys seen Sony's answer to the motion controller question?

OK, first of all, augmented reality is cool...!!!

Secondly, I think Sony's solution has a better chance of creating good games than Natal,  at least in the near-term... It's really like a WiiMote... with much greater pointing range... since w/ the WiiMote, the camera inside it must see the IR lights in the sensorbar.

I think Natal is going to miss having a trigger... Unless all Natal games will be of the Wii Fit variety...

I'm hoping what we're seeing in Natal isn't the final version

I can't believe that there won't be something like the sony/wii controller being added to Natal.  People like props, and you'll need something extra for really fine precision, but I think the the Natal technology is a better base to build on.

Minh said:

I wonder if this will be possible...

I mean, MS didn't invent this technology, they are licensing it. And the licensing fees usually depend on how widely the gadget is used...

If MS paid $X to have it on the Xbox, then MS would have to pay $X + $Y to also have it be used on Windows...

This is different than the Xbox joypad controller that MS made available on the PC also because the joypad is as commoditized as it gets -- even though they did have to shell out some more dough for rumble...

We'll just have to see I guess... Although if you see games on XBLA using Natal, then it will probably mean it'll be available for XNA... which will probably mean it'll be available on the PC

Unless some genius crack the protocol. Is Brian Peek already on it? haha

Heheh...As soon as they give me one, I'll be on it.  So, likely about a year from now. 

PerfectPhase said:

Minh said:

*snip*

I can't believe that there won't be something like the sony/wii controller being added to Natal.  People like props, and you'll need something extra for really fine precision, but I think the the Natal technology is a better base to build on.

In case of stuff like racing games.. why not just hold something round? I mean, the Wii racing wheels are really just plastic wheels with a controller in them. With natal you wouldn't even need the controller, just the plastic wheel.

Bas said:

PerfectPhase said:

*snip*

In case of stuff like racing games.. why not just hold something round? I mean, the Wii racing wheels are really just plastic wheels with a controller in them. With natal you wouldn't even need the controller, just the plastic wheel.

I remember way, way back I used to play Need for Speed with the Microsoft Sidewinder Freestyle Pro (basically a gamepad with sensors in it so you could use it as an analog controller by moving it around). It didn't work so well, mainly because there's no feedback, no resistance, no indication when you are steering as much as possible. Steering wheels don't really work for me unless they're attached to something.

Not to mention that for arcade racing games, a regular d-pad controller or even the keyboard will almost always be easier and allow you to be faster. Realistic controls are only suited for realistic games.

Ray7 said:

I wonder if the tech could be scaled down to work on handheld devices.  Multi-touch is all very nice, but it would be great if you didn't have to touch the screen for it work. 

Unless they are using some really unconventional algorithm then I say no, because the issue with machine vision algorithms tend to be memory size and memory bandwidth (and also CPU to an extent), and handheld's current resources don't cut it for most things except the most trivial.

I think even from Project Natal they are using some of the Xbox 360's resources anyway, because it would seem very expensive to put all this processing on a ASIC. This isn't like a Wiimote which is doing something very trivial algorithmically. But if Natal is done in software it would make writing a driver quite hard.

Really the demo they did makes it look like what they are doing is easy, but it's really really hard. TBH I wouldn't be suprised if it didn't work nearly as well as does in the demo when it's finally released.

Bass said:

Ray7 said:

*snip*

Unless they are using some really unconventional algorithm then I say no, because the issue with machine vision algorithms tend to be memory size and memory bandwidth (and also CPU to an extent), and handheld's current resources don't cut it for most things except the most trivial.

I think even from Project Natal they are using some of the Xbox 360's resources anyway, because it would seem very expensive to put all this processing on a ASIC. This isn't like a Wiimote which is doing something very trivial algorithmically. But if Natal is done in software it would make writing a driver quite hard.

Really the demo they did makes it look like what they are doing is easy, but it's really really hard. TBH I wouldn't be suprised if it didn't work nearly as well as does in the demo when it's finally released.

if not something like a cell phone, maybe a netbook? a netbook with a somewhat powerful gpu [assuming gpu acceleration here] sould be able to handle the natal..

im also slightly worried about the push for have no controller at all by microsoft.. [as mentioned earlier in this thread] i think you do need some sort of artifact. in real life you're not waving your arms around in this air when driving, you got a wheel, when using a gun you have a trigger etc, its actually less realistic to just have nothing at all

sony in their presentation mention that there are just some thing you need a button for and i think thats more realistic in terms of how games actually work.. from looking at the videos of natal it looks like theres like 250-500ms lag [totaly unscientific and a pure guesstimate] and thats not good enough for shooting games..

then again there is absolutly nothing stopping people from using the natal in conjunction with a regular controller it also remains to be seen if the natal can track other things that humans, like wheels or swords or whatever, it a tad unsetteling that the joint tracking demos doesnt include a hand joint.. finger joints may be to much to ask for but not tracking the hand joint would make sword swinging a lot less cool :/ compare that with the sony setup where basically all they track is the hand joint. its far less advanced but looks more responsive and accurate.. i do hope im wrong about that though

aL_ said:

Bass said:

*snip*

if not something like a cell phone, maybe a netbook? a netbook with a somewhat powerful gpu [assuming gpu acceleration here] sould be able to handle the natal..

im also slightly worried about the push for have no controller at all by microsoft.. [as mentioned earlier in this thread] i think you do need some sort of artifact. in real life you're not waving your arms around in this air when driving, you got a wheel, when using a gun you have a trigger etc, its actually less realistic to just have nothing at all

sony in their presentation mention that there are just some thing you need a button for and i think thats more realistic in terms of how games actually work.. from looking at the videos of natal it looks like theres like 250-500ms lag [totaly unscientific and a pure guesstimate] and thats not good enough for shooting games..

then again there is absolutly nothing stopping people from using the natal in conjunction with a regular controller it also remains to be seen if the natal can track other things that humans, like wheels or swords or whatever, it a tad unsetteling that the joint tracking demos doesnt include a hand joint.. finger joints may be to much to ask for but not tracking the hand joint would make sword swinging a lot less cool :/ compare that with the sony setup where basically all they track is the hand joint. its far less advanced but looks more responsive and accurate.. i do hope im wrong about that though

I think a low-mid powered GPU, with about 256 MB of very high bandwidth memory (typical of what is in video cards anyway) should be sufficent if dedicated to this kind of work. It's not something you'd find in a cellphone today, but maybe in a few years. Actually with these kind of intelligent algorithms proves we still need Moore's law to continue working if we want to advance software.

aL_ said:

Bass said:

*snip*

if not something like a cell phone, maybe a netbook? a netbook with a somewhat powerful gpu [assuming gpu acceleration here] sould be able to handle the natal..

im also slightly worried about the push for have no controller at all by microsoft.. [as mentioned earlier in this thread] i think you do need some sort of artifact. in real life you're not waving your arms around in this air when driving, you got a wheel, when using a gun you have a trigger etc, its actually less realistic to just have nothing at all

sony in their presentation mention that there are just some thing you need a button for and i think thats more realistic in terms of how games actually work.. from looking at the videos of natal it looks like theres like 250-500ms lag [totaly unscientific and a pure guesstimate] and thats not good enough for shooting games..

then again there is absolutly nothing stopping people from using the natal in conjunction with a regular controller it also remains to be seen if the natal can track other things that humans, like wheels or swords or whatever, it a tad unsetteling that the joint tracking demos doesnt include a hand joint.. finger joints may be to much to ask for but not tracking the hand joint would make sword swinging a lot less cool :/ compare that with the sony setup where basically all they track is the hand joint. its far less advanced but looks more responsive and accurate.. i do hope im wrong about that though

I do not know why the demos showed the skeletal tracking with only those few points. Supposedly it tracks 48 points per body (just Bing "Project Natal 48 points"). So I was trying to figure out what those 48 points might be, and it seems like it should include the fingers, or at least the hand joint.  Unless this includes the face (for facial expression).  Then again if it can go down as fine as facial expression, I can't see why they can't also include fingers.  Maybe the prototypes at this point only track as many points.

I also don't see why they can't scan any object of your desire, like a tennis racquet, sword, whatever, and allow you to use that as part of the tracking.

But the lag will need to be fixed. I think with some smart algorithms they can actually reduce lag to the point where it becomes insignificant.  Here is one example I was thinking of: Since Natal contains both an RGB camera as well as the depth camera, they can be made to complement each other.  The RGB camera frames are 2D and require much less post-frame processing, whereas the depth camera requires much more post-frame processing (presumably).  Let's say it is a boxing game.  If the player has their fist close to their body, the skeletal mapping will clearly show the elbow sticking out far from the fist (if the player is facing the camera).  Internally, the 3D skeleton is mapped to points on the RGB 2D image. Now, let's say the player gives a fast stab with their fist. The elbow starts moving "towards" the fist and eventually they are in line, with the elbow now "behind" the fist. Let’s say it takes some time for the post 3D image to be processed to determine this.  But the RGB camera captured this change as well. Using much simpler (faster) 2D image processing algorithms, it can track the fact that the elbow is moving "towards" the fist.  Since Natal knows the dimensions of the player's skeletal parts, it knows that the only way this can happen is if the fist actually moved towards the camera.  So using that knowledge, it can make assumptions about how parts of the body are moving before the depth algorithms are completed. Once the 3D processing completes, it can correct for any errors in the 2D processing, but it should be quite accurate if the 3D to 2D mapping algorithm is doing a good job.

Then again, the final product's processing might be fast enough, and the algorithms might be refined enough that it is fast enough to begin with.

I don't necessarily see this as completely replacing the 360 controller, more like adding additional input.  For instance, let's say in COD6, there is a menu option that becomes enabled if it detects Project Natal.  In that menu, you can select to map certain body motions to certain functions.  Like lean left/right to strafe left/right, or a sudden fist jab to simulate a melee attack, etc. You still play with your controller, but now you can have some additional control.

I was thinking of complete FPS control like this:

• Walk forward/back: Lean slightly forward/back
• Run forward/back: Lean left/right more
• Strafe left/right: Lean slightly left/right
• Turn left/right: Rotate your shoulders left/right.
• Look left/right: Rotate your head slightly left/right
• Look up/down: Tilt your head slightly up/down
• Aim (zoom): Lift your hands slightly up
• Fire: Flick your fingers (not 100% sure about that one)
• Reload: Fist over fist (a rolling motion with two fists)
• Switch weapons: Right hand over right shoulder
• Switch grenades: Left hand over left shoulder
• Melee: Stab/swing with your fist. This could actually mirror what you are doing: Left/right stabs, uppercuts, smash from above, etc

Notice that you can do all of these while still sitting on your couch. Of course you could still use your controller and just replace some of the controls with the motion versions. And also notice that it is optional, so that those that don't have Natal can still play the game.

BitFlipper said:

aL_ said:

*snip*

I do not know why the demos showed the skeletal tracking with only those few points. Supposedly it tracks 48 points per body (just Bing "Project Natal 48 points"). So I was trying to figure out what those 48 points might be, and it seems like it should include the fingers, or at least the hand joint.  Unless this includes the face (for facial expression).  Then again if it can go down as fine as facial expression, I can't see why they can't also include fingers.  Maybe the prototypes at this point only track as many points.

I also don't see why they can't scan any object of your desire, like a tennis racquet, sword, whatever, and allow you to use that as part of the tracking.

But the lag will need to be fixed. I think with some smart algorithms they can actually reduce lag to the point where it becomes insignificant.  Here is one example I was thinking of: Since Natal contains both an RGB camera as well as the depth camera, they can be made to complement each other.  The RGB camera frames are 2D and require much less post-frame processing, whereas the depth camera requires much more post-frame processing (presumably).  Let's say it is a boxing game.  If the player has their fist close to their body, the skeletal mapping will clearly show the elbow sticking out far from the fist (if the player is facing the camera).  Internally, the 3D skeleton is mapped to points on the RGB 2D image. Now, let's say the player gives a fast stab with their fist. The elbow starts moving "towards" the fist and eventually they are in line, with the elbow now "behind" the fist. Let’s say it takes some time for the post 3D image to be processed to determine this.  But the RGB camera captured this change as well. Using much simpler (faster) 2D image processing algorithms, it can track the fact that the elbow is moving "towards" the fist.  Since Natal knows the dimensions of the player's skeletal parts, it knows that the only way this can happen is if the fist actually moved towards the camera.  So using that knowledge, it can make assumptions about how parts of the body are moving before the depth algorithms are completed. Once the 3D processing completes, it can correct for any errors in the 2D processing, but it should be quite accurate if the 3D to 2D mapping algorithm is doing a good job.

Then again, the final product's processing might be fast enough, and the algorithms might be refined enough that it is fast enough to begin with.

I don't necessarily see this as completely replacing the 360 controller, more like adding additional input.  For instance, let's say in COD6, there is a menu option that becomes enabled if it detects Project Natal.  In that menu, you can select to map certain body motions to certain functions.  Like lean left/right to strafe left/right, or a sudden fist jab to simulate a melee attack, etc. You still play with your controller, but now you can have some additional control.

I was thinking of complete FPS control like this:

• Walk forward/back: Lean slightly forward/back
• Run forward/back: Lean left/right more
• Strafe left/right: Lean slightly left/right
• Turn left/right: Rotate your shoulders left/right.
• Look left/right: Rotate your head slightly left/right
• Look up/down: Tilt your head slightly up/down
• Aim (zoom): Lift your hands slightly up
• Fire: Flick your fingers (not 100% sure about that one)
• Reload: Fist over fist (a rolling motion with two fists)
• Switch weapons: Right hand over right shoulder
• Switch grenades: Left hand over left shoulder
• Melee: Stab/swing with your fist. This could actually mirror what you are doing: Left/right stabs, uppercuts, smash from above, etc

Notice that you can do all of these while still sitting on your couch. Of course you could still use your controller and just replace some of the controls with the motion versions. And also notice that it is optional, so that those that don't have Natal can still play the game.

I think you are making an incorrect assumption saying 3D image processing algorithms are more computationally more difficult. Edge detection is can be done with 3D data without almost any effort because of the added Z dimension, but with 2D data requires convolutions of spatial frequencies. Convolutions are very, very slow.

With a real 3D camera, you have more information to make correct assumptions with less processing. This assumes the hardware actually includes a 3D camera, a piece of hardware which actually returns 3D vectors, because in reality such hardware is very expensive, out the realm of consumer electronics. 

Bass said:

BitFlipper said:

*snip*

I think you are making an incorrect assumption saying 3D image processing algorithms are more computationally more difficult. Edge detection is can be done with 3D data without almost any effort because of the added Z dimension, but with 2D data requires convolutions of spatial frequencies. Convolutions are very, very slow.

With a real 3D camera, you have more information to make correct assumptions with less processing. This assumes the hardware actually includes a 3D camera, a piece of hardware which actually returns 3D vectors, because in reality such hardware is very expensive, out the realm of consumer electronics. 

Project Natal includes both a 2D RGB camera as well as a 3D time-of-flight camera. This is what makes it so amazing for something that is expected to cost around $200 (not much more than a Guitar Hero set). From your post it sounds like you don't believe that this is the case (that it is a true 3D camera where each pixel contains the true depth at that point). This fact has been pretty much confirmed. This is not a bad place to start:

http://procrastineering.blogspot.com/2009/06/project-natal.html

Also, I realize that for the grayscale depth image returned by the 3D camera, it is computationally easy to find edges.  But this is not the only thing Natal does. It maps that info into multiple skeletal points, and that is where the heavy processing is.  What I was proposing was that if that part of the processing is what is really causing the lag, that using additional, simpler 2D/3D overlay tracking could help to reduce lag since the software can make assumptions about how the skeleton is moving (from the 2D image) before the heavy 3D processing is able to even calculate the true position of the skeleton.

BitFlipper said:

Bass said:

*snip*

Project Natal includes both a 2D RGB camera as well as a 3D time-of-flight camera. This is what makes it so amazing for something that is expected to cost around $200 (not much more than a Guitar Hero set). From your post it sounds like you don't believe that this is the case (that it is a true 3D camera where each pixel contains the true depth at that point). This fact has been pretty much confirmed. This is not a bad place to start:

http://procrastineering.blogspot.com/2009/06/project-natal.html

Also, I realize that for the grayscale depth image returned by the 3D camera, it is computationally easy to find edges.  But this is not the only thing Natal does. It maps that info into multiple skeletal points, and that is where the heavy processing is.  What I was proposing was that if that part of the processing is what is really causing the lag, that using additional, simpler 2D/3D overlay tracking could help to reduce lag since the software can make assumptions about how the skeleton is moving (from the 2D image) before the heavy 3D processing is able to even calculate the true position of the skeleton.

At the bottom it says : "Product vision. Actual features may vary." or something similar to that.

If Microsoft has a 2D camera that estimates 3D data, that's not the same thing. In practice if you want accurate 3d data you need a 3d range finder, a piece of hardware that returns 3d vectors directly, based on lasers and what not.

Trust me I would like nothing more then to be able to purchase a REAL 3d range finder for $200. I have a lot of ideas personally that could benefit from a $200 range finder. If Microsoft could somehow make it happen with some crazy economics of scale or whatever the hell that would be amazing, but the fact of the matter is right now they cost thousands of dollars. That would be a massive drop in price. Seems almost unrealistic.

Bass said:

BitFlipper said:

*snip*

At the bottom it says : "Product vision. Actual features may vary." or something similar to that.

If Microsoft has a 2D camera that estimates 3D data, that's not the same thing. In practice if you want accurate 3d data you need a 3d range finder, a piece of hardware that returns 3d vectors directly, based on lasers and what not.

Trust me I would like nothing more then to be able to purchase a REAL 3d range finder for $200. I have a lot of ideas personally that could benefit from a $200 range finder. If Microsoft could somehow make it happen with some crazy economics of scale or whatever the hell that would be amazing, but the fact of the matter is right now they cost thousands of dollars. That would be a massive drop in price. Seems almost unrealistic.

It seems like you think know everything, except you choose to ignore the technology in Project Natal. Why would you do that? Instead of aguing with me about it and stating as fact that it does not contain a 3D camera, why don't you do the basic research? You bothered to post a reply here, why not also do a quick search on the subject to get more up to speed?

I realize you are very anti-MS, but that is no reason to be in denial about the technology in Project Natal.  It doesn't take much searching to find all sorts of information about the camera technology in Natal, which proves you 100% wrong.  Since you refuse to do that little bit of reaserch, I'll do it for you, and hopefully you will stop being in denial.

"The 3D sensor itself is a pretty incredible piece of equipment providing detailed 3D information about the environment similar to very expensive laser range finding systems but at a tiny fraction of the cost. Depth cameras provide you with a point cloud of the surface of objects that is fairly insensitive to various lighting conditions allowing you to do things that are simply impossible with a normal camera.

But once you have the 3D information, you then have to interpret that cloud of points as "people". This is where the researcher jaws stay dropped. The human tracking algorithms that the teams have developed are well ahead of the state of the art in computer vision in this domain. The sophistication and performance of the algorithms rival or exceed anything that I've seen in academic research, never mind a consumer product."
http://procrastineering.blogspot.com/2009/06/project-natal.html

"The device features an "RGB camera, depth sensor, multi-array microphone, and custom processor running proprietary software", which provides full-body 3D motion capture, facial recognition, and voice recognition capabilities. The Project Natal sensor's microphone array enables the Xbox 360 to conduct acoustic source localization and ambient noise suppression, allowing for things such as headset-free party chat over Xbox Live.

The depth sensor consists of an infrared projector combined with a monochrome CMOS sensor, and allows the Project Natal sensor to see in 3D under any ambient light conditions. The active depth-sensing range of the depth sensor is adjustable, with the Project Natal software capable of automatically calibrating the sensor based on the gameplay and environment conditions, such as the presence of couches."
http://en.wikipedia.org/wiki/Project_Natal

This link shows a video demo of the depth camera that was created by 3DV Systems, which is the company MS bought to get the hardware for Project Natal. BTW, research into Project Natal was going on years before they bought 3DV Systems. It is much more than just the 3D camera.
http://www.3dvsystems.com/gallery/movies/Face.mpg

An interview with Shane Kim, corporate vice president for strategy and business development at Microsoft’s game division:
"It uses an RGB camera (image sensor), a 3-D depth camera (which determines how far away an object is from the camera), and a multi-array microphone."
http://venturebeat.com/2009/06/02/microsoft-games-executive-describes-origins-of-project-natal-game-controls/

BitFlipper said:

Bass said:

*snip*

It seems like you think know everything, except you choose to ignore the technology in Project Natal. Why would you do that? Instead of aguing with me about it and stating as fact that it does not contain a 3D camera, why don't you do the basic research? You bothered to post a reply here, why not also do a quick search on the subject to get more up to speed?

I realize you are very anti-MS, but that is no reason to be in denial about the technology in Project Natal.  It doesn't take much searching to find all sorts of information about the camera technology in Natal, which proves you 100% wrong.  Since you refuse to do that little bit of reaserch, I'll do it for you, and hopefully you will stop being in denial.

"The 3D sensor itself is a pretty incredible piece of equipment providing detailed 3D information about the environment similar to very expensive laser range finding systems but at a tiny fraction of the cost. Depth cameras provide you with a point cloud of the surface of objects that is fairly insensitive to various lighting conditions allowing you to do things that are simply impossible with a normal camera.

But once you have the 3D information, you then have to interpret that cloud of points as "people". This is where the researcher jaws stay dropped. The human tracking algorithms that the teams have developed are well ahead of the state of the art in computer vision in this domain. The sophistication and performance of the algorithms rival or exceed anything that I've seen in academic research, never mind a consumer product."
http://procrastineering.blogspot.com/2009/06/project-natal.html

"The device features an "RGB camera, depth sensor, multi-array microphone, and custom processor running proprietary software", which provides full-body 3D motion capture, facial recognition, and voice recognition capabilities. The Project Natal sensor's microphone array enables the Xbox 360 to conduct acoustic source localization and ambient noise suppression, allowing for things such as headset-free party chat over Xbox Live.

The depth sensor consists of an infrared projector combined with a monochrome CMOS sensor, and allows the Project Natal sensor to see in 3D under any ambient light conditions. The active depth-sensing range of the depth sensor is adjustable, with the Project Natal software capable of automatically calibrating the sensor based on the gameplay and environment conditions, such as the presence of couches."
http://en.wikipedia.org/wiki/Project_Natal

This link shows a video demo of the depth camera that was created by 3DV Systems, which is the company MS bought to get the hardware for Project Natal. BTW, research into Project Natal was going on years before they bought 3DV Systems. It is much more than just the 3D camera.
http://www.3dvsystems.com/gallery/movies/Face.mpg

An interview with Shane Kim, corporate vice president for strategy and business development at Microsoft’s game division:
"It uses an RGB camera (image sensor), a 3-D depth camera (which determines how far away an object is from the camera), and a multi-array microphone."
http://venturebeat.com/2009/06/02/microsoft-games-executive-describes-origins-of-project-natal-game-controls/

 

Sorry, I know it includes a "range finder". I am still skepetical this will be accurate without any information regarding how this range finder actually works. If Microsoft managed to reduce the cost of a product by an order of mangitude overnight, and doesn't explain HOW, I don't think I am being unreasonable by questioning it. If that makes me "very anti-MS" then so be it.

Bass said:

BitFlipper said:

*snip*

Sorry, I know it includes a "range finder". I am still skepetical this will be accurate without any information regarding how this range finder actually works. If Microsoft managed to reduce the cost of a product by an order of mangitude overnight, and doesn't explain HOW, I don't think I am being unreasonable by questioning it. If that makes me "very anti-MS" then so be it.

I give up.  You obviously will be in denial no matter what anybody says, even those that worked closely on the project.

At least have enough of a spine to admit that you were wrong about the fact that it actually includes a 3D camera. Too much to ask for?

Let me gues: If this was an Apple project, you'd be gushing all over it, right...?

Figures...

EDIT: Funny that you will call it a "range finder", when pretty much everyone else, including those that worked on the project, do not call it that (3D camera, time-of-flight camera, depth sensor). A "range finder" is what you buy in Home Depot as a tape measure replacement that tells you the range to a solid object using a laser. Once again, I realize you need to downplay any kind of innovation from MS, but you are starting to look a little bit ridiculous now.

BitFlipper said:

Bass said:

*snip*

Project Natal includes both a 2D RGB camera as well as a 3D time-of-flight camera. This is what makes it so amazing for something that is expected to cost around $200 (not much more than a Guitar Hero set). From your post it sounds like you don't believe that this is the case (that it is a true 3D camera where each pixel contains the true depth at that point). This fact has been pretty much confirmed. This is not a bad place to start:

http://procrastineering.blogspot.com/2009/06/project-natal.html

Also, I realize that for the grayscale depth image returned by the 3D camera, it is computationally easy to find edges.  But this is not the only thing Natal does. It maps that info into multiple skeletal points, and that is where the heavy processing is.  What I was proposing was that if that part of the processing is what is really causing the lag, that using additional, simpler 2D/3D overlay tracking could help to reduce lag since the software can make assumptions about how the skeleton is moving (from the 2D image) before the heavy 3D processing is able to even calculate the true position of the skeleton.

$200 seems like an awful lot though, the whole console itself starts at $200, my guess is it'll be somewhere between $100 and 200.  It could even be sold at a loss initially like a new video game console.

BitFlipper said:

Bass said:

*snip*

I give up.  You obviously will be in denial no matter what anybody says, even those that worked closely on the project.

At least have enough of a spine to admit that you were wrong about the fact that it actually includes a 3D camera. Too much to ask for?

Let me gues: If this was an Apple project, you'd be gushing all over it, right...?

Figures...

EDIT: Funny that you will call it a "range finder", when pretty much everyone else, including those that worked on the project, do not call it that (3D camera, time-of-flight camera, depth sensor). A "range finder" is what you buy in Home Depot as a tape measure replacement that tells you the range to a solid object using a laser. Once again, I realize you need to downplay any kind of innovation from MS, but you are starting to look a little bit ridiculous now.

No, I just want more information. Personally I'd like nothing more then to have all the specifications, the source code, every little bit of info about this thing. That's what _I want_. Yes, I know I won't get it. But don't construct me as some kind of anti-MS nut because I am interested in more then just a bit of marketing speak about something.

PS: I don't own a single Apple product.

Bass said:

BitFlipper said:

*snip*

No, I just want more information. Personally I'd like nothing more then to have all the specifications, the source code, every little bit of info about this thing. That's what _I want_. Yes, I know I won't get it. But don't construct me as some kind of anti-MS nut because I am interested in more then just a bit of marketing speak about something.

PS: I don't own a single Apple product.

Well, it is interesting that I had to post multiple replies with links just to get you to realize that it actually had a true 3D camera. If you didn't even want to do that yourself, I imagine you probably also didn't bother looking at the videos of the demos they had, including the posts from the press where they got hands-on with Natal, stating that it works pretty much as well as the demos showed.

If you really want to have the specs, source (Why? The API and docs should be enough), you could always become a games developer. MS stated that they started shipping dev kits to game developers last week.

And if you are concerned about the depth resolution, look at the video I posted above:
http://www.3dvsystems.com/gallery/movies/Face.mpg

It clearly shows some a pretty well-defined depth outline of the person, while they are talking. Once again this is the same camera technology that MS bought to use with Natal.

It is one thing to be sceptical, but another entirely to completely ignore the information that is easy to find about this technology. You claim you want "more information", but I doubt that.  You didn't even know the basics.

BitFlipper said:

Bass said:

*snip*

Well, it is interesting that I had to post multiple replies with links just to get you to realize that it actually had a true 3D camera. If you didn't even want to do that yourself, I imagine you probably also didn't bother looking at the videos of the demos they had, including the posts from the press where they got hands-on with Natal, stating that it works pretty much as well as the demos showed.

If you really want to have the specs, source (Why? The API and docs should be enough), you could always become a games developer. MS stated that they started shipping dev kits to game developers last week.

And if you are concerned about the depth resolution, look at the video I posted above:
http://www.3dvsystems.com/gallery/movies/Face.mpg

It clearly shows some a pretty well-defined depth outline of the person, while they are talking. Once again this is the same camera technology that MS bought to use with Natal.

It is one thing to be sceptical, but another entirely to completely ignore the information that is easy to find about this technology. You claim you want "more information", but I doubt that.  You didn't even know the basics.

http://venturebeat.com/2009/06/02/microsoft-games-executive-describes-origins-of-project-natal-game-controls/

They are saying the 3DV aquisition has nothing to do with Natal.  It was very recent anyway, it's probably just to avoid IP issues.

BitFlipper said:

Bass said:

*snip*

Well, it is interesting that I had to post multiple replies with links just to get you to realize that it actually had a true 3D camera. If you didn't even want to do that yourself, I imagine you probably also didn't bother looking at the videos of the demos they had, including the posts from the press where they got hands-on with Natal, stating that it works pretty much as well as the demos showed.

If you really want to have the specs, source (Why? The API and docs should be enough), you could always become a games developer. MS stated that they started shipping dev kits to game developers last week.

And if you are concerned about the depth resolution, look at the video I posted above:
http://www.3dvsystems.com/gallery/movies/Face.mpg

It clearly shows some a pretty well-defined depth outline of the person, while they are talking. Once again this is the same camera technology that MS bought to use with Natal.

It is one thing to be sceptical, but another entirely to completely ignore the information that is easy to find about this technology. You claim you want "more information", but I doubt that.  You didn't even know the basics.

They might say it's a true "3D camera", but my expirence (yes professional expirence) is a "true" 3d camera currently costs in the range of five figures USD. So if it's a true 3D camera, and it's in the three figure range, whats that typical expression?

"How the f**k?"

It's like Toyota just annonced a $300 Camry or something. Is that suppose to be taken at face value?

Sadly my question of "how the f**k?" is so far unanswered, even by you. I actually have real use for a cheap * sensor that can return accurate 3D vector data of a scene, in real time. This is more then just "lawl video games", this is "f**king awesome, now I can build a robot that doesn't break itself by banging into the f**king wall".

Bass said:

BitFlipper said:

*snip*

They might say it's a true "3D camera", but my expirence (yes professional expirence) is a "true" 3d camera currently costs in the range of five figures USD. So if it's a true 3D camera, and it's in the three figure range, whats that typical expression?

"How the f**k?"

It's like Toyota just annonced a $300 Camry or something. Is that suppose to be taken at face value?

Sadly my question of "how the f**k?" is so far unanswered, even by you. I actually have real use for a cheap * sensor that can return accurate 3D vector data of a scene, in real time. This is more then just "lawl video games", this is "f**king awesome, now I can build a robot that doesn't break itself by banging into the f**king wall".

Take it easy there, no need to get so angry...

Well, judging from the demos, descriptions of what it is comprised of, and hands-on from the press, all I can say is that apparently they are doing it.  MS has the money to do the R&D, and ability to mass produce/market this so that it does become affordable to the end users.  I doubt they will be selling the cameras seperately for $200 just so that people can experiment with it outside of the Xbox (or at least they are not going to give you the SDK for free as a non-Xbox developer). The value of this to them lies in the fact that it adds to the Xbox universe, not hobyists that want a cheap 3D camera. I am sure it will get hacked and people will connect it to their PCs and do all sorts of wonderful things with it, but that is not their goal.

Anyway you slice it, no matter how many thimes you say it is not possible, they are doing it. Sorry to dissapoint you.

i thing its pretty clear that  natal does include a z camera from 3dv or some derivative there of. as for the ms exec statement, i belive they are talking about the software and its origins beeing before the 3dv buy out. in an interview that i think ive already posted on this thread, the algorithms where in development since way back and they where looking for something to "plug them into" and thats probobly where 3dv came in..

as for pricing, yes TOF cameras are typpically very expensive.. but so was accelerometers before the wiimote its all about the volume.. consider cpus and gpus, they are really hard things to make [40nm processes, pushing on 32nm] and yet they are available in consumer products.. i do belive microsoft has the real deal here

that face video was cool hadnt seen that one.. i did some more poking around their site and i found this:
http://www.3dvsystems.com/technology/tech.html
im gonna have a look at the pdf:s now, looks interesting indeed.

i also agree that this tech seems a little to good to be true.. it does seem to be pretty much real though, but i think microsoft should release more info

--edit--

whitepapers are nice and scientific.. me like..
also the gallery:
http://www.3dvsystems.com/gallery/gallery.html
has some cool stuff to check out (including full body tracking, looks pretty old though)

--edit2--

actually the papers could have been more scientific.. but the videos are cool also, it seems like the 3dv zcam had a 1.3 mega pixel resolution.. it would seem likely that natal will have something similar

aL_ said:

now, now folks, lets all try and keep cool

i thing its pretty clear that  natal does include a z camera from 3dv or some derivative there of. as for the ms exec statement, i belive they are talking about the software and its origins beeing before the 3dv buy out. in an interview that i think ive already posted on this thread, the algorithms where in development since way back and they where looking for something to "plug them into" and thats probobly where 3dv came in..

as for pricing, yes TOF cameras are typpically very expensive.. but so was accelerometers before the wiimote its all about the volume.. consider cpus and gpus, they are really hard things to make [40nm processes, pushing on 32nm] and yet they are available in consumer products.. i do belive microsoft has the real deal here

that face video was cool hadnt seen that one.. i did some more poking around their site and i found this:
http://www.3dvsystems.com/technology/tech.html
im gonna have a look at the pdf:s now, looks interesting indeed.

i also agree that this tech seems a little to good to be true.. it does seem to be pretty much real though, but i think microsoft should release more info

--edit--

whitepapers are nice and scientific.. me like..
also the gallery:
http://www.3dvsystems.com/gallery/gallery.html
has some cool stuff to check out (including full body tracking, looks pretty old though)

--edit2--

actually the papers could have been more scientific.. but the videos are cool also, it seems like the 3dv zcam had a 1.3 mega pixel resolution.. it would seem likely that natal will have something similar

Yea, this sounds very promising:

"The technology performs superior depth imaging (depth resolution of millimeters) in real-time (60 fps or more), using little or no CPU"
http://www.3dvsystems.com/technology/tech.html

One thing I think people should realize:

The actual 3D camera concept is pretty simple, if you look at what each pixel does. It is only slightly more complex than what a typical, cheap CCD does in a consumer camera. The biggest reason why it has been so expensive up to now is that our technology just wasn't fast enough to perform the shutter speeds required to make this work (the technical description I saw showed the actual "shutter" as a multiplexing electronic switch, one for each pixel).  Once we have the required speed, it becomes much more viable to mass produce this, resulting in much lower cost.

That is why I don't believe it is so far fetched that something like this can end up costing $200. Once again, not much more than a set of Guiter Hero controls, and we all know what a "failure" that turmed out to be

EDIT: This is how I understand it to work:

Each pixel in the 3D camera CCD consists of one light detector, and two capacitors (as opposed to one detector and one capacitor for a typical CCD). Initially, both capacitors start out discharged. An IR light pulse is sent out. The first capacitor it connected to the light detector.  Exactly halfway through the cycle, the light detector is switch over to the second capacitor. At the end of the cycle, the difference between the voltage of the 1st and 2nd capacitor describes the distance at that pixel.  Due to the fact that the difference between the two capacitor is used as the actual value to process, the level of ambient light is cancelled out since it will cause each of the two capacitors to have the same value.  Only the dynamic light is having an effect on the final output. If the object is close, both capacitors will have almost the same voltage. The further the object, the longer the first capacitor will be without the light from the pulse, resulting in it having a lower voltage at the end of the cycle.

Yes this is an oversimplification, but this is how I understand the basic functionalty.  Not that much more complex than the current mass-produced CCDs we find in cheap cameras. The biggest challange seems to be to get the high speed of switching required to make it work. Since they have now reached that speed, it now becomes possible to mass-produce it much more cheaply.

BitFlipper said:

aL_ said:

*snip*

Yea, this sounds very promising:

"The technology performs superior depth imaging (depth resolution of millimeters) in real-time (60 fps or more), using little or no CPU"
http://www.3dvsystems.com/technology/tech.html

One thing I think people should realize:

The actual 3D camera concept is pretty simple, if you look at what each pixel does. It is only slightly more complex than what a typical, cheap CCD does in a consumer camera. The biggest reason why it has been so expensive up to now is that our technology just wasn't fast enough to perform the shutter speeds required to make this work (the technical description I saw showed the actual "shutter" as a multiplexing electronic switch, one for each pixel).  Once we have the required speed, it becomes much more viable to mass produce this, resulting in much lower cost.

That is why I don't believe it is so far fetched that something like this can end up costing $200. Once again, not much more than a set of Guiter Hero controls, and we all know what a "failure" that turmed out to be

 

EDIT: This is how I understand it to work:

Each pixel in the 3D camera CCD consists of one light detector, and two capacitors (as opposed to one detector and one capacitor for a typical CCD). Initially, both capacitors start out discharged. An IR light pulse is sent out. The first capacitor it connected to the light detector.  Exactly halfway through the cycle, the light detector is switch over to the second capacitor. At the end of the cycle, the difference between the voltage of the 1st and 2nd capacitor describes the distance at that pixel.  Due to the fact that the difference between the two capacitor is used as the actual value to process, the level of ambient light is cancelled out since it will cause each of the two capacitors to have the same value.  Only the dynamic light is having an effect on the final output. If the object is close, both capacitors will have almost the same voltage. The further the object, the longer the first capacitor will be without the light from the pulse, resulting in it having a lower voltage at the end of the cycle.

Yes this is an oversimplification, but this is how I understand the basic functionalty.  Not that much more complex than the current mass-produced CCDs we find in cheap cameras. The biggest challange seems to be to get the high speed of switching required to make it work. Since they have now reached that speed, it now becomes possible to mass-produce it much more cheaply.

right, the 3dv design is solid state so there is far less mucking about with mechanics.. thats probobly why they think they can get down to consumer prices.. cool stuff.

BitFlipper said:

aL_ said:

*snip*

Yea, this sounds very promising:

"The technology performs superior depth imaging (depth resolution of millimeters) in real-time (60 fps or more), using little or no CPU"
http://www.3dvsystems.com/technology/tech.html

One thing I think people should realize:

The actual 3D camera concept is pretty simple, if you look at what each pixel does. It is only slightly more complex than what a typical, cheap CCD does in a consumer camera. The biggest reason why it has been so expensive up to now is that our technology just wasn't fast enough to perform the shutter speeds required to make this work (the technical description I saw showed the actual "shutter" as a multiplexing electronic switch, one for each pixel).  Once we have the required speed, it becomes much more viable to mass produce this, resulting in much lower cost.

That is why I don't believe it is so far fetched that something like this can end up costing $200. Once again, not much more than a set of Guiter Hero controls, and we all know what a "failure" that turmed out to be

 

EDIT: This is how I understand it to work:

Each pixel in the 3D camera CCD consists of one light detector, and two capacitors (as opposed to one detector and one capacitor for a typical CCD). Initially, both capacitors start out discharged. An IR light pulse is sent out. The first capacitor it connected to the light detector.  Exactly halfway through the cycle, the light detector is switch over to the second capacitor. At the end of the cycle, the difference between the voltage of the 1st and 2nd capacitor describes the distance at that pixel.  Due to the fact that the difference between the two capacitor is used as the actual value to process, the level of ambient light is cancelled out since it will cause each of the two capacitors to have the same value.  Only the dynamic light is having an effect on the final output. If the object is close, both capacitors will have almost the same voltage. The further the object, the longer the first capacitor will be without the light from the pulse, resulting in it having a lower voltage at the end of the cycle.

Yes this is an oversimplification, but this is how I understand the basic functionalty.  Not that much more complex than the current mass-produced CCDs we find in cheap cameras. The biggest challange seems to be to get the high speed of switching required to make it work. Since they have now reached that speed, it now becomes possible to mass-produce it much more cheaply.

If it's true, this is bigger then a video game controller. There are many many many many many

many many many

applications that have been waiting for an inexpensive 3d range finder

Bass said:

BitFlipper said:

*snip*

If it's true, this is bigger then a video game controller. There are many many many many many

many many many

applications that have been waiting for an inexpensive 3d range finder

Yeah no joke, imagine if digital cameras stored depth components for each pixel like this does.  You edit images in Photoshop easily, remove the background just by deleting pixels greater than a certain depth, etc. Or you could use depth data to extrude a 3d model.

Bass said:

BitFlipper said:

*snip*

If it's true, this is bigger then a video game controller. There are many many many many many

many many many

applications that have been waiting for an inexpensive 3d range finder

True, but the video game market is an obvious mass market to target , where people are more likely to pay the premium because of obvious benefits. Give it 5 years and this kind of tech will probably be in every webcam going.

aL_ said:

yeah i mean look at soemthing like zbrush (http://www.pixologic.com/) they've got a concept of 2.5d pixels that are batically pixels with depth. photoshop already got some mechanisms for describing depth with gradients and stuff.. if they play their cards right natal could be huuuuuge in the pc space.. [thats why you should release a windows driver microsoft ]

checking out

http://www.youtube.com/user/xboxprojectnatal

some good gameplay videos actually looks like input lag is pretty much on par or even slightly better than the wiimote, looks like 200-300 ms to me.. but it also seems to vary, perhaps if you're naked it'll be 100 ms

other impressions on the latency?

Minh said:

Minh said:

*snip*

Have you guys seen Sony's answer to the motion controller question?

OK, first of all, augmented reality is cool...!!!

Secondly, I think Sony's solution has a better chance of creating good games than Natal,  at least in the near-term... It's really like a WiiMote... with much greater pointing range... since w/ the WiiMote, the camera inside it must see the IR lights in the sensorbar.

I think Natal is going to miss having a trigger... Unless all Natal games will be of the Wii Fit variety...

I'm hoping what we're seeing in Natal isn't the final version

Yup, for certain games they are still going to need buttons for stuff like shooting and picking things up.

Since Natal seems to do quite well already with just tracking a players body it can only be more accurate when it's use with a prop with either glowing or reflective dots on it.

dentaku said:

Minh said:

*snip*

Yup, for certain games they are still going to need buttons for stuff like shooting and picking things up.

Since Natal seems to do quite well already with just tracking a players body it can only be more accurate when it's use with a prop with either glowing or reflective dots on it.

I think part of the problem is that we are so used to controllers that it is hard for us to imagine playing any other way.  Using a controller seems "natural" when in fact it is anything but natural.

Personally I find it difficult to use a console controller to play FPS games. It feels like all my intended moves are funelled through a straw. A melee attack is reduced to a single button push. There are so many ways you can perform a melee attack, but all I get is to push a single button.  I used to play FPS games since before the original Doom even came out using a mouse and keyboard on a PC. To me that allows for much finer control. Mathematically, an analog stick results in the 1st derivative of where you are trying to aim, while a mouse is a 1:1 mapping of where you are trying to aim.  Even so, I have gotton somewhat used to using analog sticks to play FPS games.

I think if Natal proves to be accurate enough, and they can reduce the lag to such an extent that it becomes a non-issue (which I believe they can as this is still far from final shipping hardware), that there is nothing preventing game developers from creating a whole new FPS control mechanism based on body motion that could end up being much less restrictive that a controller. And unlike some people's idea that you will need to "walk/run in place" to simulate walking/running, it can be something much simpler like leaning in the direction you want to move. I already gave an idea of a motion-only, full control scheme for an FPS while comfortably sitting down on your couch.

The good news is that the developers are now getting the development kits. Let's hope they come up with some cool things that we just can't imagine right now since we are so brainwashed into thinking that these controllers are the "natural" or only way to do things.

BitFlipper said:

dentaku said:

*snip*

I think part of the problem is that we are so used to controllers that it is hard for us to imagine playing any other way.  Using a controller seems "natural" when in fact it is anything but natural.

Personally I find it difficult to use a console controller to play FPS games. It feels like all my intended moves are funelled through a straw. A melee attack is reduced to a single button push. There are so many ways you can perform a melee attack, but all I get is to push a single button.  I used to play FPS games since before the original Doom even came out using a mouse and keyboard on a PC. To me that allows for much finer control. Mathematically, an analog stick results in the 1st derivative of where you are trying to aim, while a mouse is a 1:1 mapping of where you are trying to aim.  Even so, I have gotton somewhat used to using analog sticks to play FPS games.

I think if Natal proves to be accurate enough, and they can reduce the lag to such an extent that it becomes a non-issue (which I believe they can as this is still far from final shipping hardware), that there is nothing preventing game developers from creating a whole new FPS control mechanism based on body motion that could end up being much less restrictive that a controller. And unlike some people's idea that you will need to "walk/run in place" to simulate walking/running, it can be something much simpler like leaning in the direction you want to move. I already gave an idea of a motion-only, full control scheme for an FPS while comfortably sitting down on your couch.

The good news is that the developers are now getting the development kits. Let's hope they come up with some cool things that we just can't imagine right now since we are so brainwashed into thinking that these controllers are the "natural" or only way to do things.

but then again alot of games include things that you just cant do in real life i cant jump 20 ft in the air for example, but in games i can, thats where the abstraction for the natal sort of breaks.. also when youre running around or turning in say a fps, the TV doesnt follow you around.. natal works best when youre relativly stationary or doing some fine grain maniuplations like aiming or something. but for course grained stuff like ducking or runngin around, i think a controller is needed.

the human body is great but its also limited, our fingers are to fat to select a  single zergling out of a whole bunch, but a mouse is excelent for that Also in a fast paces fps you might want to switch weapons instantly, even if its more realisitc to reach for them.. the over shoulder reach is still cool though

i see natal as more or a replacement/compilient for the mouse. consider old school fps:s where you used only the keyboard, then the mouse came around and revolutionized the quake scene but only in conjunction with the keyboard i think something similar will happen with natal, it will compliement another input device, but not replace it completly  

Minh said:

I wonder if this will be possible...

I mean, MS didn't invent this technology, they are licensing it. And the licensing fees usually depend on how widely the gadget is used...

If MS paid $X to have it on the Xbox, then MS would have to pay $X + $Y to also have it be used on Windows...

This is different than the Xbox joypad controller that MS made available on the PC also because the joypad is as commoditized as it gets -- even though they did have to shell out some more dough for rumble...

We'll just have to see I guess... Although if you see games on XBLA using Natal, then it will probably mean it'll be available for XNA... which will probably mean it'll be available on the PC

Unless some genius crack the protocol. Is Brian Peek already on it? haha

License it?   Microsoft OWNS it!! They bought the company that made the original natal.  They can do whatever they want with it.

INCLUDING ADDING SUPPORT FOR IT TO WINDOWS 7!!!!!   If they do that, despite any of windows usual bugs I will gladly upgrade to windows 7.  To hell with snow leopard!!

StalkeZERO said:

Minh said:

*snip*

License it?   Microsoft OWNS it!! They bought the company that made the original natal.  They can do whatever they want with it.

INCLUDING ADDING SUPPORT FOR IT TO WINDOWS 7!!!!!   If they do that, despite any of windows usual bugs I will gladly upgrade to windows 7.  To hell with snow leopard!!

INCLUDING ADDING SUPPORT FOR IT TO WINDOWS 7!!!!!

Support for what? A product that "technically" does not exist?

Windows 7 is pretty much done. If you are to ask for supporting some new technology, you should look at the next version.

Minh said:

I wonder if this will be possible...

I mean, MS didn't invent this technology, they are licensing it. And the licensing fees usually depend on how widely the gadget is used...

If MS paid $X to have it on the Xbox, then MS would have to pay $X + $Y to also have it be used on Windows...

This is different than the Xbox joypad controller that MS made available on the PC also because the joypad is as commoditized as it gets -- even though they did have to shell out some more dough for rumble...

We'll just have to see I guess... Although if you see games on XBLA using Natal, then it will probably mean it'll be available for XNA... which will probably mean it'll be available on the PC

Unless some genius crack the protocol. Is Brian Peek already on it? haha

BTW, they didn't "license", nor "buy" Natal.  They were working in-house on the software part of it for a very long time.  When it came close to having an actual product to demo and send out to developers, they needed to own the hardware as well. So they bought a company that developed a 3D camera and which also owned a lot of patents on it. Two birds with one stone.

As was said many times by those that know what they are talking about, both the hardware and software used in Natal is pretty cutting edge stuff.  To say that they didn't invent anything because they bought the hardware is pretty close-minded.

You may or may not have heard of Johnny Chung Lee:

"Speaking as someone who has been working in interface and sensing technology for nearly 10 years, this is an astonishing combination of hardware and software. The few times I’ve been able to show researchers the underlying components, their jaws drop with amazement... and with good reason.

But once you have the 3D information, you then have to interpret that cloud of points as "people". This is where the researcher jaws stay dropped. The human tracking algorithms that the teams have developed are well ahead of the state of the art in computer vision in this domain. The sophistication and performance of the algorithms rival or exceed anything that I've seen in academic research, never mind a consumer product."
http://procrastineering.blogspot.com/2009/06/project-natal.html

BitFlipper said:

Minh said:

*snip*

BTW, they didn't "license", nor "buy" Natal.  They were working in-house on the software part of it for a very long time.  When it came close to having an actual product to demo and send out to developers, they needed to own the hardware as well. So they bought a company that developed a 3D camera and which also owned a lot of patents on it. Two birds with one stone.

As was said many times by those that know what they are talking about, both the hardware and software used in Natal is pretty cutting edge stuff.  To say that they didn't invent anything because they bought the hardware is pretty close-minded.

You may or may not have heard of Johnny Chung Lee:

"Speaking as someone who has been working in interface and sensing technology for nearly 10 years, this is an astonishing combination of hardware and software. The few times I’ve been able to show researchers the underlying components, their jaws drop with amazement... and with good reason.

But once you have the 3D information, you then have to interpret that cloud of points as "people". This is where the researcher jaws stay dropped. The human tracking algorithms that the teams have developed are well ahead of the state of the art in computer vision in this domain. The sophistication and performance of the algorithms rival or exceed anything that I've seen in academic research, never mind a consumer product."
http://procrastineering.blogspot.com/2009/06/project-natal.html

Do you mean this Johnny Chung Lee?

Johnny Chung Lee joins Project Natal

I would still like to see some limited form of the this tech going into handheld devices in the future. Something that can identify your face and track simple hand movements in front of the screen without having to touch it.

PaoloM said:

StalkeZERO said:

*snip*

Support for what? A product that "technically" does not exist?

Windows 7 is pretty much done. If you are to ask for supporting some new technology, you should look at the next version.

Doesn't MS often add support for new hardware in service packs?

PaoloM said:

StalkeZERO said:

*snip*

Support for what? A product that "technically" does not exist?

Windows 7 is pretty much done. If you are to ask for supporting some new technology, you should look at the next version.

why you're probobly correct in that windows wont radically change to accomodate natal, having support for natal in windows doesnt require a new windows version or even any changee to windows it self. they just have to write a driver for the natal for wondows, like all hardware manufacturers do win7 already supports multitouch soo thats something they could get almost free.. also remember that the driver can map gestures to keystrokes, so makeing the natal play the next song in wmp (or vlc for that matter) should be to hard..

even if there is no windows ui support at all just beeing able to program against the natal would be huuuuge.. natal could potentially change the scene as radically as the mouse did when it showed..

aL_ said:

PaoloM said:

*snip*

why you're probobly correct in that windows wont radically change to accomodate natal, having support for natal in windows doesnt require a new windows version or even any changee to windows it self. they just have to write a driver for the natal for wondows, like all hardware manufacturers do win7 already supports multitouch soo thats something they could get almost free.. also remember that the driver can map gestures to keystrokes, so makeing the natal play the next song in wmp (or vlc for that matter) should be to hard..

even if there is no windows ui support at all just beeing able to program against the natal would be huuuuge.. natal could potentially change the scene as radically as the mouse did when it showed..

in case you havent heard, billy G himself reckons natal is heading to windows

http://news.cnet.com/8301-13860_3-10286309-56.html