Synchronized depth and video data not being received with builtInLiDARDepthCamera

Hello,

Faced with a really perplexing issue. Primary problem is that sometimes I get depth and video data as expected, but at other times I don't. And sometimes I'll get both data outputs for a 4-5 frames and then it'll just stop. The source code I implemented is a modified version of the sample code provided by Apple, and interestingly enough I can't re-create this issue with the Apple sample app. So wondering what I could be doing wrong?

Here's the code for setting up the capture input. preferredDepthResolution is 1280 in my case. I'm running this on an iPad Pro (6th gen). iOS version 17.0.3 (21A360). Encounter this issue on iPhone 13 Pro as well. iOS version is 17.0 (21A329)

private func setupLiDARCaptureInput() throws {
        // Look up the LiDAR camera.
        guard let device = AVCaptureDevice.default(.builtInLiDARDepthCamera, for: .video, position: .back) else {
            throw ConfigurationError.lidarDeviceUnavailable
        }
        
        guard let format = (device.formats.last { format in
            format.formatDescription.dimensions.width == preferredWidthResolution &&
            format.formatDescription.mediaSubType.rawValue == kCVPixelFormatType_420YpCbCr8BiPlanarFullRange &&
            format.videoSupportedFrameRateRanges.first(where: {$0.maxFrameRate >= 60}) != nil &&
            !format.isVideoBinned &&
            !format.supportedDepthDataFormats.isEmpty
        }) else {
            throw ConfigurationError.requiredFormatUnavailable
        }
        
        guard let depthFormat = (format.supportedDepthDataFormats.last { depthFormat in
            depthFormat.formatDescription.mediaSubType.rawValue == kCVPixelFormatType_DepthFloat16
        }) else {
            throw ConfigurationError.requiredFormatUnavailable
        }
        
        // Begin the device configuration.
        try device.lockForConfiguration()

        // Configure the device and depth formats.
        device.activeFormat = format
        device.activeDepthDataFormat = depthFormat
        
        let desc = format.formatDescription
        dimensions = CMVideoFormatDescriptionGetDimensions(desc)
        
        let duration = CMTime(value:1, timescale:CMTimeScale(60))
        device.activeVideoMinFrameDuration = duration
        device.activeVideoMaxFrameDuration = duration
        
        // Finish the device configuration.
        device.unlockForConfiguration()
        
        self.device = device
        
        print("Selected video format: \(device.activeFormat)")
        print("Selected depth format: \(String(describing: device.activeDepthDataFormat))")
        
        // Add a device input to the capture session.
        let deviceInput = try AVCaptureDeviceInput(device: device)
        captureSession.addInput(deviceInput)
                
        guard let audioDevice = AVCaptureDevice.default(for: .audio) else {
            return
        }
        
        // Configure audio input - always configure audio even if isAudioEnabled is false
        audioDeviceInput = try! AVCaptureDeviceInput(device: audioDevice)
        captureSession.addInput(audioDeviceInput)

        deviceSystemPressureStateObservation = device.observe(
            \.systemPressureState,
             options: .new
        ) { _, change in
            guard let systemPressureState = change.newValue else { return }
            print("system pressure \(systemPressureState.levelAsString()) due to \(systemPressureState.factors)")
        }
    }

Here's how I'm setting up the output:

private func setupLiDARCaptureOutputs() {
        // Create an object to output video sample buffers.
        videoDataOutput = AVCaptureVideoDataOutput()
        captureSession.addOutput(videoDataOutput)
        
        // Create an object to output depth data.
        depthDataOutput = AVCaptureDepthDataOutput()
        depthDataOutput.isFilteringEnabled = false
        captureSession.addOutput(depthDataOutput)
        
        audioDeviceOutput = AVCaptureAudioDataOutput()
        audioDeviceOutput.setSampleBufferDelegate(self, queue: videoQueue)
        captureSession.addOutput(audioDeviceOutput)
        
        // Create an object to synchronize the delivery of depth and video data.
        outputVideoSync = AVCaptureDataOutputSynchronizer(dataOutputs: [depthDataOutput, videoDataOutput])
       
        outputVideoSync.setDelegate(self, queue: videoQueue)
        
        // Enable camera intrinsics matrix delivery.
        guard let outputConnection = videoDataOutput.connection(with: .video) else { return }
        if outputConnection.isCameraIntrinsicMatrixDeliverySupported {
            outputConnection.isCameraIntrinsicMatrixDeliveryEnabled = true
        }
    }

The top part of my delegate implementation is as follows:

func dataOutputSynchronizer(
        _ synchronizer: AVCaptureDataOutputSynchronizer,
        didOutput synchronizedDataCollection: AVCaptureSynchronizedDataCollection
    ) {
        // Retrieve the synchronized depth and sample buffer container objects.
        guard let syncedDepthData = synchronizedDataCollection.synchronizedData(for: depthDataOutput) as? AVCaptureSynchronizedDepthData,
              let syncedVideoData = synchronizedDataCollection.synchronizedData(for: videoDataOutput) as? AVCaptureSynchronizedSampleBufferData else {
            if synchronizedDataCollection.synchronizedData(for: depthDataOutput) == nil {
                print("no depth data at time \(mach_absolute_time())")
            }
            if synchronizedDataCollection.synchronizedData(for: videoDataOutput) == nil {
                print("no video data at time \(mach_absolute_time())")
            }
            return
        }
        
        print("received depth data \(mach_absolute_time())")
}

As you can see, I'm console logging whenever depth data is not received. Note because I'm driving the video frames at 60 fps, its expected that I'll only receive depth data for every alternate video frame.

Console output is posted as a follow up comment (because of the character limit). I edited some lines out for brevity. You'll see it started streaming correctly but after a while it stopped received both video and depth outputs (in some other runs, it works perfectly and in some other runs I receive no depth data whatsoever). One thing to note, I sometimes run quicktime mirroring to see the device screen to see what the app is displaying (so not sure if that's causing any interference - that said I don't see any system pressure changes either).

Any help is most appreciated! Thanks.