How to Convert a MTLTexture into a TextureResource?

Hi, I would suggest using the DrawableQueue API. You can get a MTLTexture per frame, and then render into that texture each frame.

https://developer.apple.com/documentation/realitykit/textureresource/drawable/texture

I need a solution that uses textures I’ve created using a regular Metal renderer, not textures that are drawables. i.e. I need to arbitrary size textures (could be many of them) that could be applied in a realitykit scene. If DrawableQueue is usable somehow for this case (arbitrary resolution textures, many of them, per-frame updating), would someone have an example? The docs do not show anything specific. Thanks!

Following as i'm curious about similar efficiencies. Its not clear in your use case what you know when.

The the sequence

DoFrequently {
      KnownDrawsize -> MetalCreateBuffer -> MetalRender -> Set TexttureResource 
}

I dont see specifics about the DrawableQueue implementation, but given the lack of specifics about load or update rate, i assume is scales the queue dynamically. (or may adjust according to the timeout property) The behavior seems implicit.

You might be able to do something like

DoFrequently {
      KnownDrawsize -> TextureResourceCreate -> DrawableQueueCreate -> getNextDrawable(*blocking) ->  MetalRender -> drawablePresent -> Set TexttureResource 
}

I guess in both scenarios, the issue is that to create the texture resource, create the drawable queue, replace the DQ onto the TR, all of these are main actor concurrency.
IF the size/descriptor were known in advance, I have thought of creating a pool of textureresource/drawablequeue pairs that were preallocated and ready to use. ShaderGraphMaterial.SetParameter is not @MainActor so can likely be called anytime.

Allocate TR/DQ pool of TextureResource/DrawableQueue

DoFrequently {
     if TR/DQ not needed: remove from SGM.Material -> return to TR/DQ pool
     if TR/DQ needed get from pool -> set on SGM.Material

     foreach TextureUpdate ready to draw
         get TR/DQ from pool -> getNextDrawable -> do Metal Render -> drawablePresent   -> set on SGM.Material
}

I'd have to double check, i think this avoids blocking mainactor calls on the DoFrquently loop.

This is assuming a number of things with having read the DrawableQueue code.
Food for thought anyway.

Did you ever find a solution to this? I'm facing the same difficulty and there isn't much documentation on the DrawableQueue API.

Hi Joe,

It's involved and I have not verified i'm using all the best APIs. I made an effort to ensure that Idid not make extra buffer copies. Your implementation may have a different optimal route depending on your texture source

But this shows the essence of working with the drawable queue.

code-block
func drawNextTexture(pixelBuffer: CVPixelBuffer) {
    
    guard let textureResource = textureResource else { return }
    guard let drawableQueue = drawableQueue else { return }
    guard let scalePipelineState = scalePipelineState else { return }
    guard let scalePipelineDescriptor = scalePipelineDescriptor else { return }
    guard let commandQueue = commandQueue else { return }
    guard let textureCache = textureCache else { return }
    let srcWidth = CVPixelBufferGetWidth(pixelBuffer)
    let srcHeight = CVPixelBufferGetHeight(pixelBuffer)
    
    autoreleasepool {
        
        var drawableTry: TextureResource.Drawable?
        do {
            drawableTry = try drawableQueue.nextDrawable() // may stall for up to 1 second. 
            guard drawableTry != nil else {
                return // no frame needed
            }
        } catch {
            print("Exception obtaining drawable: \(error)")
            return
        }
        guard let drawable = drawableTry else { return }

        guard let commandBuffer = commandQueue.makeCommandBuffer() else {
            return
        }
        
        var cvMetalTextureTry: CVMetalTexture?
        CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                                  textureCache,
                                                  pixelBuffer,
                                                  nil,
                                                  .bgra8Unorm_srgb, // linear color; todo try srgb
                                                  srcWidth,
                                                  srcHeight,
                                                  0,
                                                  &cvMetalTextureTry)
        
        guard let cvMetalTexture = cvMetalTextureTry,
              let sourceTexture = CVMetalTextureGetTexture(cvMetalTexture) else {
            return
        }
        
        // Check if the sizes match
        if srcWidth == textureResource.width && srcHeight == textureResource.height {
            // Sizes match, use a blit command encoder to copy the data to the drawable's texture
            if let blitEncoder = commandBuffer.makeBlitCommandEncoder() {
                blitEncoder.copy(from: sourceTexture,
                                 sourceSlice: 0,
                                 sourceLevel: 0,
                                 sourceOrigin: MTLOrigin(x: 0, y: 0, z: 0),
                                 sourceSize: MTLSize(width: srcWidth, height: srcHeight, depth: 1),
                                 to: drawable.texture,
                                 destinationSlice: 0,
                                 destinationLevel: 0,
                                 destinationOrigin: MTLOrigin(x: 0, y: 0, z: 0))
                blitEncoder.endEncoding()
            }
        } else {
            // Sizes do not match, need to scale the source texture to fit the destination texture
            let renderPassDescriptor = MTLRenderPassDescriptor()
            renderPassDescriptor.colorAttachments[0].texture = drawable.texture
            renderPassDescriptor.colorAttachments[0].loadAction = .clear
            renderPassDescriptor.colorAttachments[0].clearColor = MTLClearColorMake(0, 0, 0, 1) // Clear to opaque black
            renderPassDescriptor.colorAttachments[0].storeAction = .store
            
            if let renderEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPassDescriptor) {
                renderEncoder.setRenderPipelineState(scalePipelineState)
                renderEncoder.setVertexBuffer(scaleVertexBuffer, offset: 0, index: 0)
                renderEncoder.setVertexBuffer(scaleTexCoordBuffer, offset: 0, index: 1)
                
                renderEncoder.setFragmentTexture(sourceTexture, index: 0)
                
                renderEncoder.drawPrimitives(type: .triangleStrip, vertexStart: 0, vertexCount: 4)
                renderEncoder.endEncoding()
            }
        }
        
        commandBuffer.present(drawable)
        commandBuffer.commit()
    }
}

Good luck.