Creating and Viewing Immersive Video Locally on Vision Pro

Question

Mitsuhiro_Yamamoto OP

Created Sep ’24

Replies 4

Boosts 0

Participants 4

We would like to create an Immersive video and store the video file locally in Vision Pro for viewing.

By Immersive video, I mean the video that is played at the end of the Vision Pro experience at the Apple Store (LeBron's dunk, Curry's 3-point shot, tightrope walk, etc.). It is unclear if a way is currently provided to view Immersive video locally.

I can find some information about Spatial video on the Dev site, but I can't find any information about Immersive video. My understanding is:

Spatial video: A video window appears in space and plays video with depth. Up to 4K side-by-side video can be converted to MV-HEVC format using Xcode and played back in the Photos app.

Immersive video: 180VR video, but I’m not sure how it was created. Similar to Spatial video, I converted a side-by-side 180VR video to MV-HEVC format using Xcode, but it could not be played back in the Photos app as expected.

Vision Pro's Photos app features an Immersive button during video playback, but this appears to be for zooming in on Spatial video to the full field of view, which seems different from Immersive video.

The demo video provided by Apple is streamed from Apple TV, and there are no local files available.

We are currently considering creating an app that displays different videos to each eye, but we prefer not to go this route due to licensing and distribution issues.

Answered by Vision Pro Engineer in 805499022

Hello @Mitsuhiro_Yamamoto ,

To clarify, you can create a spatial video from a MV-HEVC video by the addition of spatial metadata to your file. A spatial video is a QuickTime movie with a stereo MV-HEVC video track with that additional spatial metadata. This metadata allows visionOS to apply specific treatments to your media when played back in the Photos app, for example.

Spatial media is always rectilinear, typically with a field of view less than 90 degrees. If your content uses an equirectangular, fisheye, or other non-rectilinear projection, don’t encode it as spatial media. Can you verify that you have added the required spatial metadata and that the projection of your video file meets these requirements?

With regard to the videos you mentioned (LeBron's dunk, etc.): these are also stereo MV-HEVC video tracks. You mention you'd prefer not to create an app that displays different videos to each eye, but essentially this is all stereoscopic MV-HEVC video is. Additionally, 180 degree video typically needs to be played back on a 180 degree curved surface (such as an interior section of a spherical dome), and you can use VideoMaterial to animate the texture of this surface with your stereoscopic content.

I'd be happy to go into more detail about any of these topics, so let me know if you have any questions.

Boost

Answer 1

Vision Pro Engineer OP

Apple

Sep ’24

Accepted Answer

Hello @Mitsuhiro_Yamamoto ,

To clarify, you can create a spatial video from a MV-HEVC video by the addition of spatial metadata to your file. A spatial video is a QuickTime movie with a stereo MV-HEVC video track with that additional spatial metadata. This metadata allows visionOS to apply specific treatments to your media when played back in the Photos app, for example.

Spatial media is always rectilinear, typically with a field of view less than 90 degrees. If your content uses an equirectangular, fisheye, or other non-rectilinear projection, don’t encode it as spatial media. Can you verify that you have added the required spatial metadata and that the projection of your video file meets these requirements?

With regard to the videos you mentioned (LeBron's dunk, etc.): these are also stereo MV-HEVC video tracks. You mention you'd prefer not to create an app that displays different videos to each eye, but essentially this is all stereoscopic MV-HEVC video is. Additionally, 180 degree video typically needs to be played back on a 180 degree curved surface (such as an interior section of a spherical dome), and you can use VideoMaterial to animate the texture of this surface with your stereoscopic content.

I'd be happy to go into more detail about any of these topics, so let me know if you have any questions.

2

Answer 2

Mitsuhiro_Yamamoto OP

Sep ’24

Thanks for the reply. Please let me know for my understanding.

there is no such thing as Immersive Video, it is Spatial Video transformed like an Equirectangular, there is no metadata for immersive, 2. there is no such thing as Immersive Video, it is Spatial Video transformed like an Equirectangular, there is no metadata for immersive,
Spatial Video is MV-HEVC with Spatial metadata added, and VIsion Pro recognizes Spatial Video with this meta tag.
The default application does not have the ability to playback locally placed 180-degree videos.
The only way to playback locally placed 180-degree video in MV-HEVC format is to develop a new application.

0

Answer 3

Vision Pro Engineer OP

Apple

Sep ’24

Apple Immersive Video is 180 degree 8K 3D video with Spatial Audio. It is distinct from Spatial Video.
Yes. The Photos app can display Spatial Video and Photos, and you can review this sample project to learn how to create Spatial Videos from existing MV-HEVC content.
Yes.
Yes.

1

Answer 4

MTei OP

1w

Could you please confirm my understanding? To create an immersive 180 VR movie (similar to Apple TV), we would utilize our 180 VR SBS videos and convert them to MV-HEVC format, as demonstrated in the Apple sample code. However, we would adjust the kVTCompressionPropertyKey_HorizontalFieldOfView to 180 degrees. This video would appear distorted when played in Finder or Photos. However, with a dedicated MV-HEVC immersive player app, the video would display as expected. If this is correct, I am curious why the Apple Photos app or Finder Quicklook does not offer the ability to play such media with proper playback. Additionally, Apple TV immersive videos can be played without the immersive mode, which aligns with my expectations for Finder Quicklook as well.

0