We could be controlling Apple's AR/VR headsets with… an iPhone?

We could be controlling Apple's AR/VR headsets with… an iPhone?

We've already reported on a recently published patent from Apple suggesting Apple's AR and VR headsets will require an iPhone for at least some of its processing power. And if you already need to have Cupertino's latest smartphone on hand, then maybe it makes sense for that to also act as a controller. At least, that's the prospect included in another patent from Apple, also published last week.

Okay, it doesn't specifically mention the iPhone by name, but the description seems to point pretty heavily in that direction to us. The patent centres around a user interface for virtual environments, where the user wears a head-mounted device (HMD) with a camera to track a marker on the display of external hardware, 'Such as the touch screen of a separate mobile device'. Touch screen? Mobile? iPhone?

FIG. 2 illustrates the example physical environment… including a display of the first device in accordance with some implementations

The second device to be used as a 3D controller, pointer and user interface input device.

From real to virtual

According to the patent, this mobile is how the user could provide, 'input via a separate real-world device' - but it could prove difficult for the user to operate unless the headset itself can accurately track the handheld unit. That marker is an image containing texture/information that allows the image to be detected and makes it possible for the headset to determine the image's position. The net result is that it enables, 'The second device to be used as a 3D controller, pointer and user interface input device'.

In fact, the patent goes so far as to suggest that moving the mobile could enable the user to select different buttons in the virtual world; we're thinking control surfaces in a flight sim. The patent says, 'When the user… touches a virtual button on the… remote control, the user actually touches a corresponding portion of the touch screen of the [mobile], which is recognized as input that controls or otherwise initiates an interaction with the virtual environment'.

Light of your life

The documentation also specifies that as well as a marker on the mobile display, it could emit a light source such as LEDs, pixels, a visible light source or infrared to help synchronise with motion data such as an accelerometer. That could be the selfie-flash on modern iPhones, or even new specific lighting solutions in future iterations of the smartphone.

The premise is that motion detection of the external hardware, in combination with computer vision is, 'Intended to move an associated curser [sic]'. In other words, movement in the real world can act as the user interface with finer granularity than a simple button press. Back to our flight sim, and could the system enable say, joystick controls? Maybe so.

FIG. 6 is a block diagram of an example head-mounted device (HMD) in accordance with some implementations

Why wouldn't Apple want to sell you a $1,000 iPhone to go with your brand new VR/AR headset?


Like we always say, patents can be great fun to take apart looking for a glimpse of the future, but we never expect them to release in full. And this one is no different.

Sure, it makes a lot of sense for Apple to power its headsets from external hardware such as the iPhone. And like we said up top, if that's the case and you need an iPhone to power XR experiences, then it also makes sense for it to serve double duty as a controller too. I mean, why wouldn't Apple want to sell you a $1,000 iPhone to go with your brand new VR/AR headset? It'll be fine, right?

Maybe. But that kind of corporate compromise reminds us of Sony's decision to implement its already aging Move controller technology for PS VR. At the time, that too was, well… fine. But compared with either HTC's Vive wands or even more-so, Oculus' Touch controllers, the Move was very much the poor cousin of the VR family. And it did not age well. At all.

We wouldn't be surprised for a moment if Apple played to its strengths built over years of experience with the now ubiquitous iPhone and went all-in on selling its ecosystem for XR. After all, that's exactly the play we've come to expect, from iPods to iPads and iWatches. And while we can see an iPhone control interface working for AR, we're going to conclude that we'd prefer a 'proper' control interface for Apple's VR hardware. Only time will tell...

Managing Editor

Steve is an award-winning editor and copywriter with nearly 25 years’ experience specialising in consumer technology and video games. He was part of a BAFTA nominated developer studio. In addition to editing, Steve contributes to,, and, as well as creating marketing content for a range of SMEs and agencies.