Media Extension API - How to properly vend GOP samples from a MediaFormat Extension

Hello

I am testing the new Media Extension API in macOS 15 Beta 4.

Firstly, THANK YOU FOR THIS API!!!!!! This is going to be huge for the video ecosystem on the platform. Seriously!

My understanding is that to support custom container formats you make a MEFormatReader extension, and to support a specific custom codec, you create a MEVideoDecoder for that codec.

Ok - I have followed the docs - esp the inline header info and have gotten quite far

• A Host App which hosts my Media Extenion (MKV files)
• A Extension Bundle which exposes the UTTYpes it supports to the system and plugin class ID as per the docs
• Entitlements as per docs
• I'm building debug - but I have a valid Developer ID / Account associated in Teams in Xcode
• My Plugin is visible to the Media Extension System preference
• My Plugin is properly initialized, I get the MEByteReader and can read container level metadata in callbacks
• I can instantiate my tracks readers, and validate the tracks level information and provide the callbacks
• I can instantiate my sample cursors, and respond to seek requests for samples for the track in question

Now, here is where I get hit some issues.

My format reader is leveraging FFMPEGs libavformat library, and I am testing with MKV files which host AVC1 h264 samples, which should be decodable as I understand it out of the box from VideoToolbox (ie, I do not need a separate MEVideoDecoder plugin to handle this format).

Here is my CMFormatDescription which I vend from my MKV parser to AVFoundation via the track reader

Made Format Description: <CMVideoFormatDescription 0x11f005680 [0x1f7d62220]> {
	mediaType:'vide' 
	mediaSubType:'avc1' 
	mediaSpecific: {
		codecType: 'avc1'		dimensions: 1920 x 1080 
	} 
	extensions: {(null)}
}

My MESampleCursor implementation implements all of the callbacks - and some of the 'optional' sample cursor location methods: (im only sharing the optional ones here)

- (MESampleLocation * _Nullable) sampleLocationReturningError:(NSError *__autoreleasing _Nullable * _Nullable) error

- (MESampleCursorChunk * _Nullable) chunkDetailsReturningError:(NSError *__autoreleasing _Nullable * _Nullable) error

I also populate the AVSampleCursorSyncInfo and AVSampleCursorDependencyInfo structs per each AVPacket* I decode from libavformat

Now my issue:

I get these log files in my host app:

<<<< VRP >>>> figVideoRenderPipelineSetProperty signalled err=-12852 (kFigRenderPipelineError_InvalidParameter) (sample attachment collector not enabled) at FigStandardVideoRenderPipeline.c:2231
<<<< VideoMentor >>>> videoMentorDependencyStateCopyCursorForDecodeWalk signalled err=-12836 (kVideoMentorUnexpectedSituationErr) (Node not found for target cursor -- it should have been created during videoMentorDependencyStateAddSamplesToGraph) at VideoMentor.c:4982
<<<< VideoMentor >>>> videoMentorThreadCreateSampleBuffer signalled err=-12841 (err) (FigSampleGeneratorCreateSampleBufferAtCursor failed) at VideoMentor.c:3960
<<<< VideoMentor >>>> videoMentorThreadCreateSampleBuffer signalled err=-12841 (err) (FigSampleGeneratorCreateSampleBufferAtCursor failed) at VideoMentor.c:3960

Which I presume is telling me I am not providing the GOP or dependency metadata correctly to the plugin.

I've included console logs from my extension and host app:

And my SampleCursor implementation is here

https://github.com/vade/FFMPEGMediaExtension/blob/main/LibAVExtension/LibAVSampleCursor.m

Any guidance is very helpful.

Thank you!