Windows 10: How Microsoft is making its most sensitive HoloLens depth sensor yet

Brink · Oct 9, 2018

The next-generation HoloLens sensor is a feat of research and design that crams advanced signal processing and silicon engineering into a tiny, reliable module that Microsoft intends to sell to customers and other manufacturers as Project Kinect for Azure. The company will also use the new sensor in its own products, including the next HoloLens.

At Build this year, Satya Nadella promised it would be "the most powerful sensor with the least amount of noise", and with an ultra-wide field of view. It's small enough and low power enough to use in mobile devices. But what does the sensor do, and how will a HoloLens (or anything else using it) 'see' the world into which it mixes 3D holograms?

There are different ways of measuring depth. Some researchers have tried to use ultrasound, while the traditional geometric methods use precisely structured patterns of light, in stereo -- with two beams of light hitting the same object, you can calculate the distance from the angle between the beams. Microsoft has used stereoscopic sensors in the past, but this time took a different approach called 'phased time-of-flight'.

Cyrus Bamji, the hardware architect who led the team to build Microsoft's first time-of-flight (ToF) camera into the Kinect sensor that came with Xbox One, explained to ZDNet how the new sensor works...
Click to expand...

Read more: How Microsoft is making its most sensitive HoloLens depth sensor yet | ZDNet

:)

Brink · Oct 9, 2018

HoloLens facilitates computer vision research with Research Mode

Microsoft HoloLens is the world’s first self-contained holographic computer. Remarkably, in Research Mode, available in the newest release of Windows 10 for HoloLens, it’s also a potent computer vision research device. Application code can not only access video and audio streams but can also at the same time leverage the results of built-in computer vision algorithms such as SLAM (simultaneous localization and mapping) to obtain the motion of the device as well as the spatial-mapping algorithms to obtain 3D meshes of the environment. These capabilities are made possible by several built-in image sensors that complement the color video camera normally accessible to applications.

Specifically, HoloLens has four gray-scale environment tracking cameras and a depth camera to sense its environment and capture gestures of the user. As shown in Figure 1, two of the gray-scale cameras are configured as a stereo rig capturing the area in front of the device so that the absolute depth of tracked visual features can be determined through triangulation. Meanwhile the two additional gray-scale cameras help provide a wider field of view to keep track of features. These synchronized global-shutter cameras are significantly more light-sensitive than the color camera and can be used to capture images at a rate of up to 30 frames per second (FPS).

The depth camera uses active infrared (IR) illumination to determine depth through time-of-flight. The camera can operate in two modes. The first mode enables high-frequency (30 FPS) near-depth sensing, commonly used for hand tracking, while the other is used for lower-frequency (1-5 FPS) far-depth sensing, currently used by spatial mapping. In addition to depth, this camera also delivers actively illuminated IR images that can be valuable in their own right because they are illuminated from the HoloLens and reasonably unaffected by ambient light.

Figure 1 – The HoloLens additional built-in gray scale image sensors complement the color camera normally available to applications.

With the newest release of Windows 10 for HoloLens, researchers now have the option to enable Research Mode on their HoloLens devices to gain access to all these raw image sensors streams, shown in Figure 2. Researchers can still use the results of the built-in computer vision algorithms but can now also choose to use the raw sensor data for their own algorithms. The sensors’ streams can either be processed on device or transferred wirelessly to another PC or to the cloud for more computationally demanding tasks. This opens a wide range of new computer vision applications for HoloLens. In egocentric vision, HoloLens can be used to analyze the world from the perspective of a user wearing the device. For these applications, HoloLens abilities to visualize results of the algorithms in the 3D world in front of the user can be a key advantage. HoloLens sensing capabilities can also be very valuable for robotics where these can, for example, enable a robot to navigate its environment.

Figure 2 – A sample HoloLens application that displays any of the Research Mode streams in real time.

These new HoloLens capabilities will be demonstrated at a tutorial on June 19th, 2018, at the IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR) in Salt Lake City. The next generation HoloLens depth sensing capabilities, which will also be made available through Project Kinect for Azure, will also be demonstrated at this tutorial.

We hope to see you there!
Click to expand...

Source: Microsoft HoloLens facilitates computer vision research by providing access to raw image sensor streams with Research Mode - Microsoft Research

DouglasSinger · Oct 9, 2018

Windows10 camera depth sensor

I have Hipstreet W10 tablet

Windows 10 installed, camera depth perception sensors not working. what do I do

CGThakData · Oct 9, 2018

Microsoft Hololens

Will Cortana from Microsoft Windows 10 be as helpful on the Microsoft Hololens? Are they going to to include Cortana on the Microsoft Hololens?

Windows 10: How Microsoft is making its most sensitive HoloLens depth sensor yet

How Microsoft is making its most sensitive HoloLens depth sensor yet

How Microsoft is making its most sensitive HoloLens depth sensor yet

How Microsoft is making its most sensitive HoloLens depth sensor yet

How Microsoft is making its most sensitive HoloLens depth sensor yet - Similar Threads - Microsoft making sensitive

Is there anyway to live stream sensor data on HoloLens 2?

Is there anyway to live stream sensor data on HoloLens 2?

Can't find a depth sensor in 3D scan using an Xbox One Kinect

Hololens sensors driver Bug

Asus touchpad speed -- 'Most Sensitive' not sensitive enough.

Microsoft stops manufacturing its Kinect sensors

This is Microsoft HoloLens

Microsoft HoloLens in space: Making science fiction (mixed) reality

Designed to be the most secure Windows yet