Selecting an appropriate AVCaptureDeviceFormat

My app currently captures video using an AVCaptureSession set with the AVCaptureSessionPreset1920x1080 preset. However, I'd like to update this behavior, such that video can be recorded at a range of different resolutions.

There isn't a preset aligning to each desired resolution, so I thought I'd instead directly set the AVCaptureDeviceFormat. For any desired resolution, I would find the format that is closest without going under the desired resolution, and then crop it down as a post-processing step.

However, what I've observed is that there can be a range of available formats for a device at each resolution, with various differing settings. Presumably there is logic within AVCaptureSession that selects a reasonable default based on all these different settings, but since I am applying the format directly, I think I don't have a way to make use of that default logic? And it is undocumented?

Does this mean that the only way to select a format is to implement a comparison function that considers all different values of all different properties on AVCaptureDeviceFormat, and then sort the formats according to this comparator?

If so, what if some new property is added to AVCaptureDeviceFormat in the future? The sort would not take this new property into account, and the function might select a format with some new undesired property.

Are there any guarantees about what types for formats will be supported on a device? For example, can I take for granted that a '420v' format will exist at each resolution? If so I could filter the formats down only to those with this setting without risking filtering out all of the supported formats.

I suspect I may be missing something obvious. Any help would be greatly appreciated!

Answered by DTS Engineer in 815449022

Hello @johndetloff,

I don't think you are missing anything obvious here.

Are there any guarantees about what types for formats will be supported on a device? For example, can I take for granted that a '420v' format will exist at each resolution? If so I could filter the formats down only to those with this setting without risking filtering out all of the supported formats.

There aren't any such guarantees, I wouldn't recommend doing this. At a minimum, you should fallback the non-filtered formats array if your filtered array is empty.

Does this mean that the only way to select a format is to implement a comparison function that considers all different values of all different properties on AVCaptureDeviceFormat, and then sort the formats according to this comparator?

You don't have to consider all of the different properties, only the ones that matter to your app. For example, you may not care what the videoMaxZoomFactor is if your app doesn't support zooming, so it shouldn't matter to you if the format you selected happens to have any particular value for that property.

If so, what if some new property is added to AVCaptureDeviceFormat in the future? The sort would not take this new property into account, and the function might select a format with some new undesired property.

This is one of many reasons that it is highly recommended to test your app during each beta cycle, to check for potential changes in behavior that negatively impact your app.

Best regards,

Greg

Hello @johndetloff,

I don't think you are missing anything obvious here.

Are there any guarantees about what types for formats will be supported on a device? For example, can I take for granted that a '420v' format will exist at each resolution? If so I could filter the formats down only to those with this setting without risking filtering out all of the supported formats.

There aren't any such guarantees, I wouldn't recommend doing this. At a minimum, you should fallback the non-filtered formats array if your filtered array is empty.

Does this mean that the only way to select a format is to implement a comparison function that considers all different values of all different properties on AVCaptureDeviceFormat, and then sort the formats according to this comparator?

You don't have to consider all of the different properties, only the ones that matter to your app. For example, you may not care what the videoMaxZoomFactor is if your app doesn't support zooming, so it shouldn't matter to you if the format you selected happens to have any particular value for that property.

If so, what if some new property is added to AVCaptureDeviceFormat in the future? The sort would not take this new property into account, and the function might select a format with some new undesired property.

This is one of many reasons that it is highly recommended to test your app during each beta cycle, to check for potential changes in behavior that negatively impact your app.

Best regards,

Greg

Thanks for the reply and this info!

I see what you mean about only having to consider properties that are important to my use case. However, I'm having trouble establishing confidence in which properties/values this might be.

For example, I know that I'll be needing to use an AVCaptureVideoDataOutput with a pixel format of kCVPixelFormatType_32BGRA, because I need to pass my video frames into a model that expects data in this format.

And I'm aware from this technote that an AVCaptureDeviceFormat with a different native format may result in my video being converted to BGRA when I use this output.

This technote states that the conversion will work for 420v, however, I don't know how I can determine whether this conversion will be successful with all possible AVCaptureDeviceFormat pixel formats. I see that AVCaptureVideoDataOutput has a property which can be checked after setting the activeFormat, and which returns a list of pixel formats that are supported given the activeFormat. This implies to me that it is not the case that this conversion will always succeed, at least not for all possible combinations of AVCaptureDeviceFormat and AVCaptureVideoDataOutput pixel formats.

However, this doesn't give me an appealing way to know which AVCaptureDeviceFormat instances are compatible with a kCVPixelFormatType_32BGRA output. Setting each format to be the active format, and then checking the updated value of availableVideoCVPixelFormatTypes, does not seem like a feasible option. On my device, there are 55 different format options, and setting each one as the active format in turn seems wasteful. I suppose I could test 1 AVCaptureDeviceFormat for each pixel format, but then I'm assuming that availableVideoCVPixelFormatTypes depends only on the pixel format of the activeFormat, as opposed to depending on any other property, and the documentation doesn't support that assumption (as far as I know).

Is there a better way to determine which AVCaptureDeviceFormat options will be compatible with my desired AVCaptureVideoDataOutput?

Selecting an appropriate AVCaptureDeviceFormat
 
 
Q