Time-of-flight in the new Kinect
Today we continue our new Kinect mini-series, this time with a great post on the research behind the time-of-flight feature...
A Silicon Valley-based hardware team collaborated with Microsoft Research to overcome technological hurdles with the new time-of-flight sensing camera in Xbox One.
Cyrus Bamji had encountered a challenge. Luckily for him, Microsoft Research had just the solution.
Bamji, Microsoft partner hardware architect for Microsoft’s Silicon Valley-based Architecture and Silicon Management group, and members of his team were trying to incorporate a time-of-flight camera into Xbox One, the successor to the wildly popular Xbox 360.
A time-of-flight camera emits light signals and then measures how long it takes them to return. That needs to be accurate to 1/10,000,000,000 of a second—remember, we’re talking the speed of light here. With such measurements, the camera is able to differentiate light reflecting from objects in a room and the surrounding environment. That provides an accurate depth estimation that enables the shape of those objects to be computed.
That speed-of-light capability would be a major advancement for the Kinect sensor portion of Xbox One, being released to 13 launch markets next month. The new Kinect, a key differentiator for Xbox One against its competition, needed to capture a larger field of view with greater accuracy and higher resolution. An infrared sensor will enable object identification requiring little to no light, and improved hand-pose recognition, giving gamers and more casual users the ability to control the console with their hands.
But Cyrus Bamji had a challenge. The sensor was great, but it also left those working on it eager to do even more with it.
Krupka—with a few Microsoft Research colleagues making contributions of their own—worked well with their Xbox partners. Their combined domain knowledge meshed well.
“The reason Eyal and I were successful,” Perry says, “was because of his extensive knowledge of computer vision, signal processing, and machine learning, along with my knowledge of time-of-flight technology and the system tradeoffs, allowing us to make the right decisions in a short amount of time and keep to the tight schedule.”
“The research aspects of what we delivered for Xbox One did not start on the day we start working with the product team,” he says. “It starts years before we learn about any specific project or problem. It is based on accumulating a wide range of research expertise through exploration on multiple research projects, accumulating engineering and research tools and practices—including rapid research methods.
“This is achieved by rotating cycles of working on long-term research problems, then switching to short-term research tasks. This is critical to the success. If we did only short-term, on-demand research, we couldn’t have the critical assets when we work on the product’s problems. If we worked only on long-term research, we would have had a harder time switching gears to deliver solutions on a product group’s timeline.”
The analog nature of the time-of-flight data posed challenges to delivering such a solution.
Other challenges presented themselves. For one thing, processing time became an issue. In the academic literature about time-of-flight systems, processing time wasn’t an issue. In the laboratory environment, the technology worked fine. But Xbox One needs to process a whopping 6.5 million pixels per second. And only a small part of Xbox One’s computing power could be harnessed for this task. The lion’s share is reserved, understandably, for essentials such as gaming, skeleton tracking, face recognition, and audio.
“You need to do very, very light computation for each pixel,” Krupka says, “and this is one of the things that made the problem challenging and different from the typical approach in the academic literature in this field.”
Other Microsoft Research contributions to the Xbox One include Fang Wen and her team at Microsoft Research Asia, who contributed their state-of-the-art technologies for high-accuracy face recognition. Ivan Tashev of Microsoft Research Redmond brought his audio expertise, demonstrated during the development of the Xbox 360, to optimize the sound experience enabled by the Xbox One. Ranveer Chandra contributed a wireless-controller protocol. And, from Krupka’s team, Ido Leichter wrote valuable simulations of the camera and environment, and Daniel Freedman provided innovative solutions to improve the robustness of depth images.
“We found people eager to help us solve our problems,” says Bamji of his collaboration with Microsoft Research. “The researchers were very interested in making an impact by working on products that were going to market.”
Krupka found the collaboration challenging—and, ultimately, rewarding.
“For me,” he says, “it was the most intensive three months of work in my whole career. It seemed like I was working 24 hours a day, and the response from the Silicon Valley Xbox team was similar.
“We had a clear deadline, and we understood what was at stake. We understood that this was very critical for the project. We were aware of it from the beginning. Everyone was on board, and now, we’re excited to see how Xbox One customers respond later this fall. It was a great example of teamwork that, hopefully, translates into a ‘just works’ experience that delights Xbox One customers and developers.”