Choosing a Technology Stack for Apple Vision Pro Applications

When building an Apple Vision Pro application, developers have some options to choose from to build an immersive spatial computing experience. The decision which setup to pick depends on the application type and its later intended use. Let’s go through some of the technology options that can be used to develop Apple Vision Pro applications and analyze their adequacy per use case.

Types of Apps for Apple Vision Pro

WIth Apple Vision Pro slated for commercial release on February 2, 2024, Apple has already released lots of information about hardware, specifications, and technologies of the Apple Vision Pro headset to support product development. That said, with the technology stack and software development kits available now, you can start developing your application for Apple Vision Pro to be one of the first movers once the device hits the market.

The key information to understand in terms of Apple Vision Pro app development is the application types it can run. The biggest difference, aside from the viewing experience of each app type, is the app’s capability to understand and then react to user input.

Input (spatial data) options:

• Head pose — Head pose describes the orientation and position of the headset, allowing the system to adjust visuals based on where and how the user is looking.
• Hand pose — Hand pose refers to the recognition and interpretation of the positions and movements of a user's fingers and hand. It allows the user to interact with on-screen content using natural hand gestures.
• Spatial data — Spatial data pertains to information about the physical location and shape of objects in space, which the user can manipulate using eyes, hands, and voice.
• Eye tracking — Eye tracking technology observes and analyzes eye movements and gaze, offering a unique way to interact with the device. This advancement also marks a significant step towards improved accessibility for individuals with disabilities.

With that terminology checked off, here are the app types for Apple Vision Pro:

Volumes

These are dedicated to 3D content. The Volumes apps lack head pose, hand pose, and spatial data. Imagine a floating hologram which you can view and maneuver but have limited interaction options (i.e., the app doesn’t receive spatial data and can’t adjust to the user's input, it views and uses only that app's content) – that’s Volumes.

Spaces

Here’s where things become truly immersive. Spaces incorporate head pose, giving users an experience that spans the entire room. They also support hand and spatial data, allowing users to place digital objects in physical space and interact with them freely.

Windows

These are akin to iPad apps, but Apple Vision Pro offers a more immersive, distraction-free data-viewing experience. Ideal for displaying news, charts, or sheets, they can be resized to optimize the data-viewing experience.

An Apple Vision Pro application can comprise a mix of these types. For example, a Spaces app can house multiple Windows and Volumes apps. These are called shared-space apps (different apps running in a user’s space).

Apple's Official Technology Stack

When developing applications for Apple Vision Pro, utilizing Apple's official technology stack helps deliver that familiar Apple feeling. Here's an overview of the key components of the official technology stack as provided by Apple:

Core development environment

Xcode

The backbone of Apple’s development ecosystem, Xcode is an integrated development environment offering a complete set of tools. It includes project management support, code editors, visual editors for UI, debugging tools, simulators for different devices, and a suite of frameworks for software development.

Programming languages and frameworks

SwiftUI

A modern, declarative programming model designed for creating an app’s core functionality, making it a primary choice for new projects in visionOS.

UIKit: Essential for building an interactive user interface, UIKit has been adapted for use in visionOS, ensuring familiarity for developers experienced with iOS and iPadOS app development.

3D and spatial computing

RealityKit

The tool is pivotal in visionOS apps for managing and animating 3D objects, offering a programmable way to create detailed and interactive 3D content.

ARKit

Facilitates immersive interactions with the user’s environment and supports the creation of custom gestures, a key feature in spatial computing.

Content creation and scene building

Unity

Known for its robust, familiar authoring tools, Unity is compatible with visionOS, allowing developers to leverage both Unity and visionOS features.

Reality Composer Pro

A powerful tool for building scenes with comprehensive elements like objects, animations, sounds, and visual effects.

Advanced graphics and file formats

MaterialX

An open standard for dynamic effects, MaterialX shaders can be integrated into RealityKit scenes for visually striking results.

USDZ Files

Used for building complex 3D objects, allowing for nondestructive edits and integration into larger scenes.

Audio and video integration

AVKit and AVFoundation

These frameworks support the integration and management of audio-visual content, including 3D movies and Spatial Audio.

PHASE

Enhances audio experiences by supporting the creation of dynamic Spatial Audio in games and apps.

Data management and internationalization

Data Frameworks

The Swift Standard Library and Foundation Framework provide essential data types and support for managing content, while Core Data and CloudKit offer models for handling structured data.

Localization and Internationalization

Foundation Framework helps make multiple apps globally accessible by formatting content for different languages and regions.

Different Immersion Modes in Apple Vision Pro

Apple Vision Pro by design is an AR-first device. What it means is that the device blends digital content into the user’s environment (by way of pass-through technology: a feature that uses the headset's camera to view the user's real-world surroundings).

• Mixed — Blends digital and real-world content.
• Full — A total digital takeover.
• Progressive — Gradually shifts from real-world to digital content using the Digital Crown button.

Including different immersion modes into an application gives developers an option to add layered spatial experiences, where the user can choose to, for example, work in a content simulator without seeing their surroundings or keep the simulator as part of the user space in AR mode. Apple puts great emphasis on the customization of the experience, giving users freedom in how they want to create their space.

Focus on Collaboration in Apple Vision Pro

Augmented reality delivered on headsets can be an extended immersive workspace where the user switches between different applications (for example Sheets, design application, and conferencing app) for a highly collaborative experience. A key feature of Apple Vision Pro is SharePlay. It allows multiple users, either in proximity or connected via Facetime, to view the same digital content.

Tech Stack Apple Vision Pro: Overview

Apple Vision Pro supports many technology options from which developers can create their digital products. And while this list is by no means exhaustive and there are other technologies that you can use to build an Apple Vision Pro app, we have tested the below collection extensively. If you want to learn more about specific features you can implement using different Apple technologies for the headset, go to our blog post on features list for AVP. You'll find which tools to use to integrate spatial audio and high-quality 3D animations and other types of incredibly advanced technology to build apps for Apple Vision Pro.

Note: Developers can seamlessly blend SwiftUI, RealityKit, and Metal to craft apps with all views: Windows, Spaces, and Volumes.

SwiftUI

SwiftUI is Apple’s modern UI toolkit. When combined with RealityKit, it facilitates the creation of 3D elements within apps.

Benefits

• out-of-the-box native look & feel familiar to Apple

Drawbacks

• can be insufficient for applications heavier in 3D content

Best for

• Windows-type applications

RealityKit

RealityKit is Apple’s library for rendering 3D content.

Benefits

• adds a 3D layer to applications built in SwiftUI

Drawbacks

• the particle system is limited (there’s only a basic particle emitter)
• less code reusability for apps that run multiple devices
• the current version (August 2023) has less features compared to Unity

Best for

• for 3D-heavy spatial apps
• Volumes and Spaces apps

Metal

Think of Metal as Apple's powerhouse for graphic-intensive tasks (Apple’s version of OpenGL and DirectX). It's a low-level programming language optimized for performance-heavy applications.

Benefits

• as it operates very close to the graphic card it adds a necessary performance stimulus for 3D rendering

Drawbacks

• steep learning curve

Best for

• detailed 3D models
• photo-editing software

Unity

Unity is one of the biggest platforms for creating real-time 3D content. PolySpatial is Unity’s SDK created specifically for building applications for Apple Vision Pro.

Benefits

• cross-platform optimized (code reusability across different devices and headsets)
• no royalties for using in application (you pay only for development platform)
• mature platform with large support
• editor in Unity is platform agnostic
• Unity has a plugin for that supports Apple’s RealityKit default format, .usd, but Unity supports a larger number of formats, which gives developers more flexibility
• the learning curve is easy because of the abundance of materials and community, tutorials, and plugins. Windows apps can be ported from iPad to AVP, however, Spaces and Volumes require 3D knowledge

Drawbacks

• it can be challenging to replicate Apple's distinct look & feel

Best for

• Spaces and Volumes AVP apps
• heavy 3D content and spatial experiences
• games and visualizers due to a large library of special effects

Flutter

Mainly known for mobile and web app development, Flutter can also be utilized to port iPad apps to Apple Vision Pro.

Benefits

• established UI framework for developing multi-platform applications

Drawbacks

• it can be challenging to replicate Apple's distinct look & feel

Best for

• Windows-type applications without 3D content

The Decision Tree for Choosing a Tech Stack for Apple Vision Pro

Depending on project status and the availability of development resources, you can choose the most suitable tech stack by following the below advice.

Deeper Considerations When Choosing a Tech Stack

Hardware limitations and capabilities

The Apple Vision Pro, given its advanced AR functionalities, is powered by high-end processors and graphics units. While it offers great computational power, it's essential to select a stack that efficiently utilizes the device's hardware without causing overheating or lags.

Battery life implications

Augmented reality apps, especially those with a high degree of 3D content, can be power-hungry. Certain frameworks or coding practices might deplete the battery faster. SwiftUI and RealityKit, being Apple's native tools (they render directly to the device), are optimized for power efficiency, whereas third-party tools like Unity might consume more power due to the added abstraction layers.

Security

Apple's ecosystem is renowned for its security. Using Apple's native tools such as SwiftUI will integrate seamlessly with the device's in-built security protocols. However, if opting for third-party solutions, always ensure that user data protection complies with Apple's security guidelines.

Scalability

As your app's features and user base grow, you'll need a tech stack that scales efficiently. While Apple's native tools are robust, Unity's vast ecosystem might offer more plugins and solutions for scalability challenges.

Developer community and support

Unity boasts a massive developer community, providing countless resources, plugins, and troubleshooting guides. Apple's community revolving around the technology stack for Apple Vision pro is less developed given the device is new.

Maintenance and updates

Native tools, like SwiftUI, will receive regular updates aligned with Apple's OS updates, which translates into faster bug patches. Third-party solutions like Unity or Flutter won’t be able to fix bugs without official patches released by Apple.

Cost implications

While Apple's native tools come bundled with the developer account, third-party platforms might have licensing costs, especially for when targeting multiple platforms (e.g., Switch, PlayStation, or Xbox). For example, Unity requires developers to purchase Unity Pro when the app reaches $100 K revenue.

Also, from January 1, 2024, Unity will introduce a Unity Runtime Fee tied to game installs. This applies only to games surpassing certain revenue and install benchmarks: $200,000 with 200,000 installs for Unity Personal/Plus and $1,000,000 with 1 million installs for Unity Pro/Enterprise. Additionally, starting November, cloud-based storage, Unity DevOps, and AI at runtime will be integrated into Unity subscription plans without extra charges.

Integration with other systems

If you're looking to integrate cloud services, databases, or third-party APIs, Apple's ecosystem offers seamless integrations with its services like iCloud. For broader integrations, a platform like Unity might offer more flexibility as it’s API agnostic.

Future trends and forecasts

Augmented reality is rapidly evolving. While Apple's tools will consistently adapt to the latest trends, platforms like Unity can potentially pivot faster due to their broader focus on VR/AR across devices.

The Principles Behind Choosing Your Development Pathway for an Apple Vision Pro App

1. Anticipating future trends

In the tech world, innovation often lies at the intersection of what's current and what's envisioned. Brands like Apple exemplify this by being pioneers who also transform markets when they step in. As AR and VR technologies hover on the brink of mainstream adoption, it's crucial to look beyond immediate capabilities. Instead, focus on the long-term trajectory and potential scalability and compatibility when choosing a tech stack.

2. The depth of expertise

Broad technical skills are important, but niche proficiencies can often make the difference between a good application and an exceptional one. When choosing a development partner for a device as specialized as Apple Vision Pro, it's not just about knowing AR and VR. It's about understanding the intricacies of the platform it's built on — in this case, the Apple ecosystem. Thus, depth, as much as breadth, should be a priority when seeking tech solutions or partners. A partner with a combination of AR and VR expertise with experience in iOS development can help navigate the development swiftly.

3. Launching at the right moment

Timing can often be as crucial as the product itself. In technology, being a first-mover might provide a distinct advantage. The early bird doesn't just get the worm — it sets the trend and positions itself as a market leader. When choosing a tech stack or a development partner, consider how they can enable you to hit the market at optimal moments, ensuring your application rides the wave of early adoption.

4. Cross-platform preparedness

While it's essential to be specialized, cross-platform compatibility can be a huge benefit in the long run. Whether it's for ensuring wider reach or future-proofing against market shifts, your chosen tech stack and development partner should be ready for adaptability. This is not just about moving between AR and VR but also about ensuring that applications can work on different hardware ecosystems.

5. Development for end-user experience

While back-end proficiency is vital, the success of any application ultimately hinges on its user experience. The chosen tech stack and development strategy should prioritize the end-user. This means intuitive interfaces, seamless transitions between AR and VR, and ensuring optimal performance even as the application scales. For this to happen, an experienced team of developers that knows the intricacies of building augmented reality products is key.

6. Continuous learning and adaptability

AR and VR are highly dynamic. Today's cutting-edge solution might become tomorrow's standard practice. Hence, it's important that your development partner comes with a commitment to continuous learning, research, and adaptability. They should be in sync with evolving best practices, emerging tools, and the shifting preferences of the end-user.

How Other Technologies Work When Building Applications for the Vision Pro Headset

Let’s take Unity as an example.

The resulting application will utilize Apple's RealityKit for augmented reality. However, any limitations of RealityKit might affect Unity's capabilities.

Getting Your Choices Right for Apple Vision Pro Development

The Apple Vision Pro, bridging the realms of AR and VR, represents a new frontier for immersive applications. As developers venture into this expansive domain, it's essential to not only comprehend the technological nuances but also anticipate long-term shifts in the AR and VR landscapes. Whether adapting existing apps or innovating new ones, informed decisions, strategic foresight, and a deep understanding of the device's capabilities are pivotal. Your tech stack choices will not only mold the present application's success but also set the trajectory for its future evolution in this dynamic and ever-evolving ecosystem of Apple platforms. Ensuring apps launch seamlessly from the start, will help you solidify your position on the market. Apple Vision Pro helps companies innovate during Apple's journey toward AR glasses.