I was trying to migrate Core Image based code that's rotating an image in a CVPixelBuffer to the newer VTPixelRotationSession from Video Toolbox. Hoping to increase performance.
The original code does:
let rotatedImage = CIImage(cvPixelBuffer: origPixelBuffer).oriented(.left)
context.render(rotatedImage, to: newPixelBuffer)
The new code uses a session:
_ = VTPixelRotationSessionRotateImage(rotationSession, origPixelBuffer, newPixelBuffer)
However I immediately ran into memory limitations, since my code has to be able to run in an iOS extension. It seems VTPixelRotationSessionRotateImage easily lets memory usage spike over the 50MB of allowed memory. While the CIImage based implementation has no such high memory usage at all.
Is this expected? Does the VTPixelRotationSession implementation gain more performance by sacrificing memory? Or is there something I'm overlooking?
I was expecting the VTPixelRotationSession at worst to be on par in terms of memory usage and processing speed compared to CIImage. At this moment it seems VTPixelRotationSession is unusable in extensions.
See also Feedback: FB14977240