VideoMaterial to display SBS Stereoscopic 3D video? [VisionOS]

Hi, I love VideoMaterial API that gives so much power to play video on any mesh. But I am trying to play a side-by-side 3D video usingVideoMaterial:

RealityView { content in

    let mesh = MeshResource.generatePlane(width: 300.0, height: 300.0, cornerRadius: 0) //generate mesh

    let vidMaterial = VideoMaterial(avPlayer: AVPlayer(url: URL(string: "https://someurl/test/master.m3u8")!)) //VideoMaterial
                
    vidMaterial.controller.preferredViewingMode = .stereo //<-- no idea why it doesn't work for SBS video in simulator
    vidMaterial.avPlayer?.play()

    let planeEntity = Entity() //new entity
     planeEntity.components.set(ModelComponent(mesh: mesh, materials: [vidMaterial])) //set a new ModelComponent to the entity

    content.add(planeEntity)
}

this code works well for plain 2D video playback but how do I display a Side-by-Side or Top-Bottom 3D video?

I found GeometrySwitchCameraIndex in custom ShaderGraphMaterial but if I use input node as a image texture then how do I pass the video frame as texture into my custom shader to achieve the 3D effect or maybe there is an even better way to deal with this?

There seems to be additional API .preferredViewingMode on the VideoMaterial's controller that can be set to .stereo but it doesn't give any stereo effect. Perhaps it's only for MV-HEVC media playback?

Answered by DTS Engineer in 797708022

Hello @AbhiStack,

Check out this thread which covers something similar: https://forums.developer.apple.com/forums/thread/746484?answerId=781707022#781707022

There seems to be additional API .preferredViewingMode on the VideoMaterial's controller that can be set to .stereo but it doesn't give any stereo effect. Perhaps it's only for MV-HEVC media playback?

Yes, that is only for MV-HEVC media playback. In general, unless you have a good reason not to, I recommend converting your side-by-side or top-bottom media to MV-HEVC: https://developer.apple.com/documentation/avfoundation/media_reading_and_writing/converting_side-by-side_3d_video_to_multiview_hevc_and_spatial_video

Best regards,

Greg

Hello @AbhiStack,

Check out this thread which covers something similar: https://forums.developer.apple.com/forums/thread/746484?answerId=781707022#781707022

There seems to be additional API .preferredViewingMode on the VideoMaterial's controller that can be set to .stereo but it doesn't give any stereo effect. Perhaps it's only for MV-HEVC media playback?

Yes, that is only for MV-HEVC media playback. In general, unless you have a good reason not to, I recommend converting your side-by-side or top-bottom media to MV-HEVC: https://developer.apple.com/documentation/avfoundation/media_reading_and_writing/converting_side-by-side_3d_video_to_multiview_hevc_and_spatial_video

Best regards,

Greg

VideoMaterial to display SBS Stereoscopic 3D video? [VisionOS]
 
 
Q