Building a custom render pipeline with RealityKit

Hello experts, and question seekers,

I have been trying to get Gaussian splats working with RealityKit, however it seems not to work out for me.

The library I use for Gaussian splatting: https://github.com/scier/MetalSplatter

My idea was to use the renderers provided by RealityKit (aka RealityRenderer) https://developer.apple.com/documentation/realitykit/realityrenderer and the renderer provided by MetalSplatter (aka. SplatRenderer) https://github.com/scier/MetalSplatter/blob/main/MetalSplatter/Sources/SplatRenderer.swift

Then with a custom render pipeline, I would be able to compose the outputs of the renderers, enabling the possibility, for example to build immersive scenery with realistic environment scans, as Gaussian splats, and RealityKit to provide the necessary features to build extra scenery around Gaussian splats, eg. dynamic 3D models inside Gaussian splats.

However the problem is, as of now I am not able to do that with the current implementation of RealityRenderer.

It seems to be, that first RealityRenderer is supposed to be an API, just to render colour information onto a texture, which in first glance might be useful, but misses important information, such as for example depth, and stencil information. Second issue is, even with that in mind, currently I am not able to execute RealityRenderer.updateAndRender, due to the following error messages:

Could not resolve material name 'engine:BuiltinRenderGraphResources/Common/realityRendererBackground.rematerial' in bundle at '/Users/<user>/Library/Developer/CoreSimulator/Devices/<device UUID>/data/Containers/Bundle/Application/<app UUID>/<app name>.app'. Loading via asset path.

exiting spatial tracking service update thread because wait returned 37”

I was able to build a custom Metal view with UIViewRepresentable, MTKView, and MTKViewDelegate, enabling me to build a custom rendering pipeline, by utilising some of the Metal developer workflows.

Reference: https://developer.apple.com/documentation/xcode/metal-developer-workflows/

Inside draw(in view: MTKView), in a class derived by MTKViewDelegate:

guard let currentDrawable = view.currentDrawable else {
    return
}
let realityRenderer = try! RealityRenderer()
try! realityRenderer.updateAndRender(deltaTime: 0.0, cameraOutput: .init(.singleProjection(colorTexture: currentDrawable.texture)), whenScheduled: { realityRenderer in
    print("Rendering scheduled")
}, onComplete: { RealityRenderer in
    print("Rendering completed")
})

Can you please tell me, what I am doing wrong? Is there any solution, that enables me to use RealityKit with for example Gaussian splats?

Any help is greatly appreciated.

All the best,

Ethem Kurt