A detailed report from The Information lays out the decisions behind the apparent years-long delay of Apple’s upcoming AR/VR headset.
In 2021, Bloomberg, The Information and supply chain analyst Ming-Chi Kuo published reports claiming that Apple was preparing to release a high-end headset for virtual reality and augmented reality with color transmission high resolution.
Apple’s VR/AR team is called Technology Development Group (TDG) and is led by former Dolby executive Mike Rockwell. As previously reported by Bloomberg in 2020, TDG initially developed a powerful wireless “hub” that would stream high-fidelity content to lightweight headphones. But former Apple design director Jony Ive apparently “bailed” on the idea, instead insisting that the headset be self-contained. The standoff between Rockwell and Ive reportedly lasted for months, when CEO Tim Cook sided with Ive in 2019.
The decision to ship a standalone headset apparently left some TDG members in “disbelief” as they had built high-fidelity software designed to run on a powerful external processing unit, including photorealistic avatars.
The report details the challenges of delivering an ultra-high-end headset with all the processing built-in without causing overheating or compromising battery life. Apple management apparently expects an AR experience “far beyond” what Mark Zuckerberg’s Meta can deliver, and Rockwell is apparently struggling to deliver.
The headset apparently has 14 cameras in total, including those used for passthrough, position tracking, eye tracking, face tracking, hand tracking, and body tracking, causing ” head” to hardware and algorithm engineers. To handle this, Apple has built a dedicated image signal processor (ISP) chip called Bora, but engineers are apparently struggling to eliminate latency between this and the headset’s main processor – the upcoming M2. To mitigate this, Apple apparently had to include a third chip to act as a “fast conduit”, but that didn’t completely fix the problem.
A unique visible feature of the helmet is a screen in the front to allow others in the room to see the top of your face and your eyes. But adding this screen apparently meant passthrough cameras had to be placed in “awkward” positions away from where the user’s eyes would be, making it more difficult to develop passthrough reprojection algorithms. .
The Information quotes “several people familiar with the helmet” as describing these makeshift solutions as a reflection of “overengineering, overly complicated solutions that often result from poor planning”.
While Jony Ive left Apple in 2019, the report claims he’s still involved with the project as a consultant, with employees expected to travel to his San Francisco home to get his approval on the changes. Previous prototypes apparently integrated the battery into the headband, but Ive “prefers” a design with an attached battery that the user wears. The information indicates that it is unclear which approach will be used in the final design.
The information states that “four people who worked on the project” criticized the lack of focus on the games. TDG apparently “almost never” mentions games in internal presentations and doesn’t develop a tracked controller, instead the input methods being hand tracking and a “clothespin-shaped clothespin” that has appeared in the repositories Apple patents.
The News report earlier this week cited five sources revealing that Tim Cook rarely visits the team working on the headset – a stark difference to Meta where some employees are said to be frustrated by Zuckerberg’s obsession with virtual reality and augmented reality. The lack of a senior Apple executive to champion the project apparently made it more difficult to allocate engineering personnel and resources compared to the iPhone and Mac. To garner support for the project, team members apparently warned that companies like Facebook and Magic Leap could end up owning the industry.
Previously reported information that Apple is considering pricing the product at $3,000, but says Apple’s top executive, Dan Riccio, was assigned to the project last year to focus on lowering the cost materials to make it more affordable. It should end up competing with Meta’s Project Cambria, which is slated to launch later this year for “significantly” above $800. If the reports so far are to be believed, however, Apple’s product will have a higher resolution, a more powerful processor and a slimmer design – although achieving all of this is clearly not without its challenges.