VisionOS 2 - Passthrough in screen capture

Aside from a new enterprise api for internal apps, that isn't allowed due to privacy concerns,

Never mind, I missed you got the approval.

Are you seeing a black background instead of passthrough? If so, it's likely an entitlement configuration issue. Can you confirm you've added the com.apple.developer.screen-capture.include-passthrough entitlement to your project? If you haven't adding it should cause passthrough to show up. Here are setup instructions for entitlements; the instructions cover an unrelated entitlement, but the steps are the same.

I've tried with ReplayKit and the following function.

let recorder = RPScreenRecorder.shared()
recorder.startCapture(handler: { ... })

But I think that this is not the right way, because I can see only my View in the buffer.

I don't know what is the right method to do the Passthrough in screen capture because I can't find any documentation except com.apple.developer.screen-capture.include-passthrough entitlement.

You can live broadcast using ReplayKit (with a Broadcast Extension). ReplayKit is an existing multi platform framework. It's available without the EnterpriseKit entitlement, but without the entitlement the frames captured contain a black background instead of passthrough. This is covered around 4:30 of video, but only briefly. Here's some more specific information to help you.

At a high level, you need to 2 things to broadcast: An app for users to start/stop the broadcast and a broadcast upload extension to receive the video frames and send them to your broadcast endpoint.

In the application

Use RPSystemBroadcastPickerView to allow users to pick a broadcast destination (aka broadcast upload extension). Here's code for that.

// Define a wrapper for RPSystemBroadcastPickerView that you use to integrate into your SwiftUI view hierarchy.
struct BroadcastButtonView: UIViewRepresentable {
    func makeUIView(context: Context) -> RPSystemBroadcastPickerView {
        let broadcastPickerView = RPSystemBroadcastPickerView(frame: CGRect(x: 0, y: 0, width: 44, height: 44))
        broadcastPickerView.preferredExtension = "com.yourdomain.broadcast-services-upload"
        broadcastPickerView.showsMicrophoneButton = true
        return broadcastPickerView
    }

    func updateUIView(_ uiView: RPSystemBroadcastPickerView, context: Context) {
        // Update the view if needed
    }
}

// Add the button to your view.
BroadcastButtonView()
    .frame(width: 44, height: 44)

Creating the extension

Use Xcode to create a broadcast upload extension; be sure to add the aforementioned entitlement. "Broadcast upload extension" has been around for a while so there's lot's of examples online (in the context of iOS, but broadcast upload extensions are multi platform), but here's the TLDR: Creating the extension generates SampleHandler which extends RPBroadcastSampleHandler. Implement the methods in that class to upload content to your endpoint.

Please let me know how it goes!

Thank you very much for the explanation! I have tried ReplayKit and Broadcast Extension and actually I can stream.

The problem now, but maybe because it's still in beta, is that when I enter "Passtrought in Screen Capture" on the Broadcast Extension capabilities, after about 8 seconds, it stops the stream and I get the following message on the Vision Pro "Live Broadcast to Broadcast Upload Extension has stopped due to: Attempted to start an invalid broadcast session."

While removing the capability (so with black background), it does not freeze and everything works.

Do you have any advice?

Happy to help.

Our engineering teams need to investigate this issue. I'd greatly appreciate it if you could open a bug report, include a focused code sample that reproduces the issue, and post the FB number here once you do.

Another thing to keep in mind, your broadcast extension runs in a separate thread. If it throws an exception you will not see any exception in Xcode; your app will appear to fail silently. What's odd in this case is it works without the entitlement, but errors with the entitlement. Nonetheless, it's worth taking a closer look at the code in your broadcast extension.

Hi, I apologize for the delay in responding, but I wanted to investigate the issue more thoroughly.

The "invalid broadcast session" problem was caused by excessive RAM consumption of our data upload library. Once replaced, I no longer encountered the problem and everything is working properly!

So, I did not open any bug reports.

Thanks for helping me solve this!