I am working on a project for a university which wants to alter the passthrough camera feed more so than the standard filters (saturation/contract/etc) that some of the headsets provide.
I don't have access to the headset or enterprise SDK yet, as I'd like to nail down whether or not this is feasible before we purchase the hardware. In the API I see I can use CameraFrameProvider to access a CameraFrame and then grab a sample. The sample has a CVPixelBuffer. I have 2 questions regarding the pixelBuffer:
- I see that the buffer itself is read only but can I alter the bytes within this pixel buffer? Lets say change all green pixels to red (not my actual use case but just an example)
- Will the updated pixel buffer then be used in the passthrough screen?
If not, then is there any way to have control over the video feed that is being displayed as passthrough? Our ideal setup would be to have access to a frame, alter it however we want, and then have the frame displayed in passthrough. I realize I could take the feed and copy it into a floating window and alter that, but that breaks the immersion we are shooting to create here.
Thanks in advance!
Hello @ickydime,
Our ideal setup would be to have access to a frame, alter it however we want, and then have the frame displayed in passthrough.
The answer here depends on the extent of the alterations. If you only want to add things (with no dependency on what the pixel value behind that thing is), then you can use CompositorServices.
If you want to alter things with a dependency on the pixel values, you should file an enhancement request using Feedback Assistant.
Best regards,
Greg
