MultiThreaded rendering with actor

Hi, I'm trying to modify the ScreenCaptureKit Sample code by implementing an actor for Metal rendering, but I'm experiencing issues with frame rendering sequence. My app workflow is:

ScreenCapture -> createFrame -> setRenderData Metal draw callback -> renderAsync (getData from renderData)

I've added timestamps to verify frame ordering, I also using binarySearch to insert the frame with timestamp, and while the timestamps appear to be in sequence, the actual rendering output seems out of order.


// ScreenCaptureKit sample
func createFrame(for sampleBuffer: CMSampleBuffer) async {
    if let surface: IOSurface = getIOSurface(for: sampleBuffer) {
        await renderer.setRenderData(surface, timeStamp: sampleBuffer.presentationTimeStamp.seconds)
    }
}


class Renderer {
    ...

    func setRenderData(surface: IOSurface, timeStamp: Double) async {
        _ = await renderSemaphore.getSetBuffers(
            isGet: false,
            surface: surface,
            timeStamp: timeStamp
        )
    }

    func draw(in view: MTKView) {
        Task {
            await renderAsync(view)
        }
    }

    func renderAsync(_ view: MTKView) async {
        guard await renderSemaphore.beginRender() else { return }
        guard let frame = await renderSemaphore.getSetBuffers(
            isGet: true, surface: nil, timeStamp: nil
        ) else {
            await renderSemaphore.endRender()
            return }
        
        guard let texture = await renderSemaphore.getRenderData(
            device: self.device,
            surface: frame.surface) else {
            await renderSemaphore.endRender()
            return
        }
        
        guard let commandBuffer = _commandQueue.makeCommandBuffer(),
                let renderPassDescriptor = await view.currentRenderPassDescriptor,
                let renderEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPassDescriptor) else {
            await renderSemaphore.endRender()
            return
        }

        // Shaders ..
        renderEncoder.endEncoding()
        
        commandBuffer.addCompletedHandler() { @Sendable (_ commandBuffer)-> Swift.Void in
            updateFPS()
        }
        
        // commit frame in actor
        let success = await renderSemaphore.commitFrame(
            timeStamp: frame.timeStamp,
            commandBuffer: commandBuffer,
            drawable: view.currentDrawable!
        )
        
        if !success {
            print("Frame dropped due to out-of-order timestamp")
        }
        
        await renderSemaphore.endRender()
    }
}

actor RenderSemaphore {
    private var frameBuffers: [FrameData] = []
    private var lastReadTimeStamp: Double = 0.0
    private var lastCommittedTimeStamp: Double = 0
    
    private var activeTaskCount = 0
    private var activeRenderCount = 0
    private let maxTasks = 3
    
    private var textureCache: CVMetalTextureCache?
    
    init() {
    }
    
    func initTextureCache(device: MTLDevice) {
        CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, device, nil, &self.textureCache)
    }
    
    func beginRender() -> Bool {
        guard activeRenderCount < maxTasks else { return false }
        activeRenderCount += 1
        return true
    }
    
    func endRender() {
        if activeRenderCount > 0 {
            activeRenderCount -= 1
        }
    }
    
    func setTextureLoaded(_ loaded: Bool) {
        isTextureLoaded = loaded
    }
    
    func getSetBuffers(isGet: Bool, surface: IOSurface?, timeStamp: Double?) -> FrameData? {
        if isGet {
            if !frameBuffers.isEmpty {
                let frame = frameBuffers.removeFirst()
                if frame.timeStamp > lastReadTimeStamp {
                    lastReadTimeStamp = frame.timeStamp
                    print(frame.timeStamp)
                    return frame
                }
            }
            
            return nil
        } else {
            // Set
            let frameData = FrameData(
                surface: surface!,
                timeStamp: timeStamp!
            )
            
            // insert to the right position
            let insertIndex = binarySearch(for: timeStamp!)
            frameBuffers.insert(frameData, at: insertIndex)
            return frameData
        }
    }
    
    private func binarySearch(for timeStamp: Double) -> Int {
        var left = 0
        var right = frameBuffers.count
        
        while left < right {
            let mid = (left + right) / 2
            if frameBuffers[mid].timeStamp > timeStamp {
                right = mid
            } else {
                left = mid + 1
            }
        }
        
        return left
    }
    
    // for setRenderDataNormalized
    func tryEnterTask() -> Bool {
        guard activeTaskCount < maxTasks else { return false }
        activeTaskCount += 1
        return true
    }
    
    func exitTask() {
        activeTaskCount -= 1
    }
    
    func commitFrame(timeStamp: Double,
                     commandBuffer: MTLCommandBuffer,
                     drawable: MTLDrawable) async -> Bool {
        
        guard timeStamp > lastCommittedTimeStamp else {
            print("Drop frame at commit: \(timeStamp) <= \(lastCommittedTimeStamp)")
            return false
        }
        
        commandBuffer.present(drawable)
        commandBuffer.commit()
        lastCommittedTimeStamp = timeStamp
        
        return true
    }
    
    func getRenderData(
        device: MTLDevice,
        surface: IOSurface,
        depthData: [Float]
    ) -> (MTLTexture, MTLBuffer)? {
        let _textureName = "RenderData"
        
        var px: Unmanaged<CVPixelBuffer>?
        let status = CVPixelBufferCreateWithIOSurface(kCFAllocatorDefault, surface, nil, &px)
        guard status == kCVReturnSuccess, let screenImage = px?.takeRetainedValue() else {
            return nil
        }
        
        CVMetalTextureCacheFlush(textureCache!, 0)
        
        var texture: CVMetalTexture? = nil
        let width = CVPixelBufferGetWidthOfPlane(screenImage, 0)
        let height = CVPixelBufferGetHeightOfPlane(screenImage, 0)
        
        let result2 = CVMetalTextureCacheCreateTextureFromImage(
            kCFAllocatorDefault,
            self.textureCache!,
            screenImage,
            nil,
            MTLPixelFormat.bgra8Unorm,
            width,
            height,
            0, &texture)
        
        guard result2 == kCVReturnSuccess,
              let cvTexture = texture,
              let mtlTexture = CVMetalTextureGetTexture(cvTexture) else {
            return nil
        }
        mtlTexture.label = _textureName
        
        let depthBuffer = device.makeBuffer(bytes: depthData, length: depthData.count * MemoryLayout<Float>.stride)!
        return (mtlTexture, depthBuffer)
    }
}

Above's my code - could someone point out what might be wrong?

Your example is kinda complex, and it’s built in terms of things that I only vaguely understand, so let me know if I’ve misunderstood the issue here.

I’m trying to understand your specific complaint. My best guess is that:

  1. Frames show up at the top of renderAsync(…) in sequence. I’m specifically referring to the frame value returned by getSetBuffers(…).

  2. But they’re out of sequence by the time they get to the commitFrame(…) call at the end of that method.

Is that an accurate summary of the problem?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

  1. Yes
  2. Yes. And I print the timestamp after commitFrame and it shows sequential. So I think that is so strange and I cannot get where is the problem

OK, then I can explain that behaviour. Within renderAsync(…) there are multiple suspension points, each marked with an await. At each suspension point the concurrency runtime is free to suspend the task running this (async) function and use that core to run other tasks. So, if multiple tasks are running renderAsync(…) simultaneously, there’s no guarantee that they’ll hit the commitFrame(…) call in the same order as they hit the getSetBuffers(…) call.

As to how you fix this… ah… um… that’s where my ignorance of these graphics APIs start to show )-: My main question centres around your draw(in:) routine. Do you have to draw every frame? Or would it be better to just draw the latest frame?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Hi, thanks for your explanation!

Ideally, I would like to draw as many captured frames as possible. However, I understand that if processing speed isn’t sufficient, I may need to drop some frames to keep up with real-time rendering. That said, my goal is definitely not to draw only the latest frame, as I want to preserve as much of the original capture data as possible.

Let me know if this aligns with what you’re asking!

MultiThreaded rendering with actor
 
 
Q