Undervannsoperasjoner i mixed reality med Apple Vision Pro

fredag 10. mai 2024 / Blueye Shore Lab

Since day one, the Blueye underwater drones have been operated using iOS and Android devices. This has allowed us to leverage hardware and software innovations in the iOS and Android ecosystem. Blueye customers can choose from a large selection of compatible devices, from mobile phones to rugged tablets and large-screen setups.

The introduction of the Apple Vision Pro headset opens exciting new possibilities for underwater operations in mixed reality. We have just scratched the surface, and in this article, we will share our first impressions from using the Apple Vision Pro for ROV operations and demonstrate what the future might hold.

Operating the Blueye X3 using Apple Vision Pro

Our first digital dive masks

Since the first prototypes of the Blueye underwater drone, our team has experimented with various digital headset alternatives.

The original idea was that the headset would be your digital diving mask, offering an immersive view of the underwater world.

For the Blueye Pioneer's initial release in 2017, we partnered with MovieMask (now known as DroneMask) and bundled their headset with our underwater drone. The MovieMask is a simple device that allows you to insert your mobile device into a headset with lenses that give the impression of looking at a much larger screen. The MovieMask also blocks out light, removing the problem of hard-to-read screens in bright sunlight.

The MovieMask was included with the Blueye Pioneer when launched in 2017.

However, the MovieMask headset has some significant drawbacks, perhaps the biggest being its lack of spatial awareness when worn. With the headset on, you cannot see your surroundings. Improvements in device hardware, with brighter and better screens, have reduced the problem of using mobile devices in direct sunlight. Hence, the use of MovieMask to mitigate this is no longer that relevant.

2016: Test of an HDMI based headset connected to a laptop.
2016: Mechanical engineer Rune Hansen using MovieMask.
2017: The DJI Googles connected to iPhone using HDMI adapter.

Apple Vision Pro

Blueye X3 with the Apple Vision Pro

The Apple Vision Pro is a mixed-reality headset with incredible spatial mapping and passthrough technology, allowing you to interact with the physical world even when wearing the headset. The Vision Pro uses cameras to record your surroundings before it overlays content in 3D and renders the scene onto the two 4K displays in front of each eye—all in less than 15 milliseconds. You interact with the device using eye tracking, simply looking at elements and selecting them using gestures such as pinching your fingers.

If you are unfamiliar with the Apple Vision Pro, the official video from Apple gives a great introduction to the platform:

Introducing Apple Vision Pro: The era of spatial computing is here, where digital content blends seamlessly with your physical space. So you can do the things you love in ways never before possible. This is Apple Vision Pro.

Operating the Blueye X3 using Apple Vision Pro

The Apple visionOS is based on iPadOS and can run most iPad apps out of the box. This means that both the Blueye App and the Blueye Observer App are already available for the Apple Pro Vision. Running the Blueye App on Pro Vision gives you the impression of controlling the drone on a crisp 4K screen as big as you’d like.

You can configure a virtual control center with multiple screens showing sonar imagery, maps, mission planning, reference documents, or your dive plan. The screens can be anchored in 3D space and you can move around with the headset and interact with the physical world.

Virtual Control Center: Operating the Blueye X3 using Apple Pro Vision, with multibeam to the left, video stream in the center, and positioning system to the right.

The Pro Vision is not a peripheral to the iPhone or iPad but a self-contained unit with a high-performance M3 chip onboard. The Blueye App runs directly on the headset, and you connect an Xbox Controller or other compatible game controller to operate the drone.

Our vision for the Vision Pro

The Vision Pro is the first version in a new product category from Apple. Over the years, we can expect product improvements in battery life, weight, ruggedness, price point, and other aspects of the device. However, already in its initial version, the Vision Pro offers exciting new possibilities for better, more immersive underwater operations in a compact form factor.

We believe the Vision Pro headsets can be used as compact, portable virtual control center for our underwater drones. Large, dedicated screens for sonar, video, maps, and telemetry that can easily fit inside a backpack.

All the functionality in the scenario demonstrated in this article, and video is done using hardware and software that is commercially available today. No prototypes or beta versions were used.

Jonas Follesø operating the Blueye X3 with the Apple Vision Pro

The Vision Pro will help spatial computing and mixed reality become more mainstream and increase innovation in the field. The other big player in the mixed reality space, Meta, has recently announced that they are opening their mixed reality headset operating system to third-party hardware makers. The Meta Horizon OS is based on Android, and even though you cannot run all Android apps out of the box today, there is a possibility that the Blueye Android app might also become available on mixed- and virtual reality headsets in the future.

If you are curious to learn more about the Blueye underwater drones or perhaps want to try out the Vision Pro in an underwater context, please contact us to book a demo.

Learn more about using Vision Pro for professional use