Unraveling the Magic of Lidar on iPhones and iPads: What Can It Really Do?

A few years back, Apple made waves in the mobile industry by incorporating lidar sensors into its latest iPhones and iPads. To this day, they remain the sole players in this arena, offering this feature on their devices. However, lidar technology has a rich history, and many modern smartphones employ principles akin to it. So, what exactly is lidar, and what are its practical applications on your Apple gadgets?

What Is Lidar?

Lidar stands for “LIght Detection And Ranging” and functions on principles similar to radar. In both systems, an electromagnetic wave—laser light in the case of lidar and radio waves for radar—is emitted from a transmitter. A sensor then listens for any reflections of the wave upon striking an object. The distance (or range) between the reflecting object and the source of the wave is calculated based on the time it takes for the transmission and reception.

The origins of lidar trace back to the early 1960s, closely following the invention of the laser. By the 1970s, NASA was utilizing lidar to map the topography of the moon. Today, lidar finds application in diverse fields, from archaeology—uncovering lost cities in dense jungles—to the autonomous vehicle industry, enabling vehicles to sense objects in their vicinity.

And yes, it’s a feature found in the latest iPhone Pro models.

Phones with Laser Precision

The integration of lasers into phones began with LG’s G3 in 2014, where they served as time-of-flight sensors to enhance autofocus capabilities. This marks the most fundamental use of lidar, where an infrared laser is emitted to gauge the distance to the subject of a photo. This distance data is then employed to adjust the camera’s focus automatically.

Fast forward three years, and Apple introduced the iPhone X, featuring the groundbreaking TrueDepth camera system for its revolutionary Face ID technology. Although facial recognition for unlocking phones wasn’t pioneered by Apple (Android 4.0 laid the groundwork back in 2011), Apple’s TrueDepth system brought a significant shift. While Android’s initial foray into facial recognition was notably insecure—allowing unlocking with a mere photo—Apple’s TrueDepth system was remarkably robust. It could distinguish between identical twins while accommodating accessories like sunglasses and hats.

How did Apple achieve this technological feat? Lasers played a pivotal role. The TrueDepth system bombards the user’s face with an array of 30,000 infrared lasers, which are captured by an infrared camera and analyzed by AI software. This process yields a precise 3D scan of your face within seconds. Unlike time-of-flight sensors, which measure the time taken for laser light to return to the camera, TrueDepth relies on analyzing how the pattern of lasers (structured light) is altered by objects in the scene to infer 3D data.

In 2020, three years after introducing the TrueDepth system, Apple unveiled its lidar system. Similar to TrueDepth, it emits an array of lasers (in a known pattern), but instead of measuring light deformation, a specialized sensor gauges the time each laser beam takes to return to the sensor. These time-of-flight measurements are combined with data from the device’s motion sensor and camera to generate a comprehensive 3D scan.

Unlocking the Potential of Lidar

Apple’s lidar technology is undeniably impressive, but what practical uses does it offer? The range of applications is extensive, albeit somewhat niche, and these applications aren’t necessarily game-changers for most users. More crucially, it’s unlikely to significantly influence purchasing decisions when shopping for a new phone.

One of the most straightforward uses of Apple’s lidar system is as a time-of-flight sensor, effectively speeding up autofocus on rear-facing cameras. While most phone cameras from recent years rely on contrast-based or phase-detection autofocus—requiring sufficient light to function—time-of-flight sensors sidestep this limitation, granting phones equipped with them an edge in low-light photography.

Apple also leverages its lidar technology in a couple of in-house apps. The Measure app, utilizing lidar data and augmented reality, employs simple trigonometry to provide precise measurements of people and objects.

For 3D printing enthusiasts and game developers in need of 3D models, apps like Scaniverse and Polycam offer solutions. By scanning an object from multiple angles, users can construct a 3D model that can be exported to standard design software. The choice between the two boils down to personal preference and budget, as Scaniverse is free, while Polycam offers a free tier with some features locked behind a subscription.

If you’re more inclined toward gaming, there’s good news. One of the standout lidar-enabled games on the App Store is RC Club, which allows you to drive a virtual RC car within your living space. The lidar scanner ensures you’ll bump into walls and navigate obstacles realistically. Moreover, your virtual RC car responds to the type of surface it’s traversing, slowing down when encountering carpeted areas.

Where to Find Lidar

Apple’s lidar system is exclusive to iPhone Pro models and iPad devices released since 2020, which limits its availability. No equivalent technology is present in Android phones, and standalone lidar systems come with significant price tags. While this exclusivity represents a triumph for Apple, it could potentially stifle software-based innovation and confine lidar to its current niche position. On the flip side, Apple’s efforts to miniaturize and enhance the robustness of these systems for pocket-sized devices suggest that it may not be long before lidar becomes a standard feature in every new smartphone.