For virtual reality to work, the position of the headset in 3D space is crucial information. Inside-out VR tracking uses onboard cameras and machine learning to perform this measurement, and it’s quickly becoming the gold standard for VR head tracking.
Conventional Positional Tracking
There have been many different approaches to positional tracking for VR, but when we refer to “conventional” positional tracking, it’s in reference to the modern VR revolution, starting around the mid-2010s, rather than the early virtual reality days of the late 80s or early 90s.
The original Oculus Rift was at the vanguard of this new generation of VR headsets. Built on technologies developed for smartphones, the Rift contained sensors that allow for finely grained detection of rotation and lateral movement. In other words, using internal sensors, the headset can report rotation and when it’s moving forward, backward or side-to-side. What it can’t do is report depth. That is, where within 3D space it is relative to other points.
The original Rift’s solution was to use an external tracking camera. The camera sees in the infrared spectrum and on the surface of the headset is an array of infrared lights. The camera tracks these lights and, since it knows their exact scale and pattern, can figure out exactly where the headset is relative to the camera itself.
Combined, all of this sensor data offers accurate positioning anywhere within 3D space, at least insofar as the external tracker can see the headset. Later, products such as the HTC Vive would introduce laser-based room-scale trackers. These were mounted on the walls of a room and allowed headset positions to be tracked anywhere within their range.
Inside-Out Tracking
Inside-out tracking flips the script on the traditional method we’ve just described. The headset still uses camera technology to figure out where it is in 3D space, but the cameras are all on the headset itself.
There are no external tracking markers, instead, sophisticated machine vision software looks for points of reference in the room around you. The software watches the movement of those reference points and then infers the headset’s movement.
The onboard cameras can do even more than that. Thanks to this spatial awareness, it’s possible to create a software-based virtual barrier, so that you don’t accidentally walk into a wall. This is also possible with room-scale tracking systems, but not with traditional single-camera external tracking.
The onboard cameras can also precisely track the position of your VR controllers relative to the headset. In a brilliant further development, it’s now also possible to use these onboard cameras to directly track your bare hands for interaction within VR. No controllers needed!
Inside-out tracking can also offer a form of “mixed reality.” That is, where real-world and virtual objects are blended together as one experience. The onboard headset cameras can do things like let you see your real-world keyboard when using an app like Virtual Desktop or bring your real-world chair into the virtual space to make it easy to sit down or get up.
The Advantages of Inside-Out Tracking
Inside-out tracking is quickly becoming the norm for VR headsets and it’s not hard to see why. This approach has numerous benefits, but the most prominent one is elegance and simplicity.
On a headset like the Oculus Quest 2, there is only the headset and the two controllers. When using the Quest 2 as a tethered headset, you only need a single USB cable. Getting into VR is quick and easy. You don’t have to prepare equipment, just put on the headset and play. Moving from one location to the next is equally simplified. There’s no clutter and no messing around with cables, cameras, or trackers that have to be mounted. (Even if you want to use the Oculus Quest 2 as a PC VR headset, you can do it with a single USB cable—or wirelessly.)
As we mentioned above, inside-out tracking also offers new opportunities to enhance VR that go beyond positional tracking. What started out as a solution to enable true standalone VR has become what seems to be the likely future of all VR headsets.
Inside-out tracking also solved the problem of occlusion, where the external tracking camera can’t see your headset or controllers because they’re obscured. One of the reasons there are at least two wall-mounted trackers with room-scale setups is to avoid anything being occluded from a sensor.
Oculus Quest 2
The Oculus Quest 2 is our favorite VR headset given its ease of setup and inexpensive price. It features inside-out tracking so you don’t have to set up any separate cameras—you don’t even need a PC. (However, the Quest 2 can function as a PC VR headset, too.)
$299.00 
The Downside of Inside-Out Tracking
Inside-out tracking is an impressive technology, but some of the most premium VR products, such as the Valve Index, still use externally-mounted trackers. If inside-out tracking is so great, why isn’t it universal? The main reason comes down to precision and accuracy. The software that drives inside-out tracking has to do a lot of “guessing” since no algorithm is 100% accurate. This is limited by how much computational power the headset has on-board since more accurate and sophisticated machine vision technology needs more processing power. Over time, this problem is addressed simply by the march of computer performance.
As a result, inside-out tracking on virtual reality headsets is not quite as accurate or fast as outside-in tracking, but it’s catching up rapidly and is already good enough that you probably won’t notice any problems while swinging a lightsaber around or trying to win a virtual boxing match.
