Decode video frames in lower resolution before processing

We are processing videos with Core Image filters in our apps, using an AVMutableVideoComposition (for playback/preview and export).

For older devices, we want to limit the resolution at which the video frames are processed for performance and memory reasons. Ideally, we would tell AVFoundation to give us video frames with a defined maximum size into our composition. We thought setting the renderSize property of the composition to the desired size would do that.

However, this only changes the size of output frames, not the size of the source frames that come into the composition's handler block. For example:

let composition = AVMutableVideoComposition(asset: asset, applyingCIFiltersWithHandler: { request in
    let input = request.sourceImage // <- this still has the video's original size
    // ...
})
composition.renderSize = CGSize(width: 1280, heigth: 720) // for example

So if the user selects a 4K video, our filter chain gets 4K input frames. Sure, we can scale them down inside our pipeline, but this costs resources and especially a lot of memory. It would be way better if AVFoundation could decode the video frames in the desired size already before passing it into the composition handler.

Is there a way to tell AVFoundation to load smaller video frames?