Capture HDR content with ScreenCaptureKit

Hello and welcome! My name is Ben, I am a software engineer on the ScreenCaptureKit team. Every day, people use ScreenCaptureKit to share their screens in video conference calls sharing high-quality streams with others around the world, taking screenshots and supporting remote desktop applications. When you use ScreenCaptureKit, your app will be integrated with the system UI experience. Your app can present the built-in picker to provide a familiar way to choose stream content.

Presenter overlay is available to enhance your app’s screen sharing experience. And, people can see a preview of what your app is sharing in the system UI. I’ll be showing you some new ways ScreenCaptureKit helps you build these same kinds of experiences in your app. Let’s say your app has a streaming feature. You’ll now be able to capture streams and screenshots with HDR. Your app may currently capture screen and audio content. I'll show you how to get microphone output from the same stream that provides you screen and audio samples.

If your app uses recorded screen content, you’ll be interested in the convenient recording API. ScreenCaptureKit handles all of the asset writing details for you.

In order to understand how to capture HDR content, let’s first get familiar with the basics for capturing screen and audio.

Here’s the SCStream, it is the object that you use to receive screen and audio samples in your app.

Use it to capture an entire screen, or just specific windows and apps.

Tell the stream what to capture by providing it an SCContentFilter. Say you’d like to customize the stream’s output, like specify a different resolution or pixel format. For this, use the SCStreamConfiguration to customize stream output.

Once you’ve set up a stream, you'll be able to receive screen and audio samples by using the SCStreamOutput.

These are the basic building blocks for capturing content, if you’d like more detail check out the previous ScreenCaptureKit videos.

New this year, you can capture HDR content. High Dynamic Range, or HDR, provides you with a wider range of brightness levels and colors when compared to the Standard Dynamic Range, or SDR. HDR offers high-contrast, resulting in more details in both shadows and highlights.

It also provides a wider range of colors.

I’m happy to tell you that the SCStream now outputs High Dynamic Range of captured content.

People receiving streamed HDR content will now experience the same rich details as the source.

To enable HDR capture in your app, you’ll configure your stream by setting some new properties on the SCStreamConfiguration. And here are the properties: captureDynamicRange tells ScreenCaptureKit to output screen content in either SDR or HDR. For HDR, set it to either hdrLocalDisplay or hdrCanonicalDisplay. But wait, what do I mean by local vs canonical display? When you use local display, you are telling ScreenCaptureKit that you are both capturing and rendering HDR content on the same screen.

For canonical display, it means you are capturing HDR content that is optimized for sharing with other HDR devices.

The next property to configure is pixelFormat. It describes how color information is stored for each pixel in an image.

For HDR you will use a format that has at least 10-bits per component. For most situations 10-bit YCbCr will be the best choice.

The third property is the colorSpaceName, which describes the range of colors and the transfer function for how they map to digital values. HDR requires either HLG or PQ transfer functions, for example, the Display P3 PQ color space.

And finally, there is the colorMatrix that is used to transform colors between the BGRA and YCbCr color spaces. If this sounds like a lot of information, don’t worry, because the SCStreamConfiguration is here to help you.

It provides two convenient presets for HDR. captureHDRStreamLocalDisplay and captureHDRStreamCanonicalDisplay If the destination for captured HDR content is the local display, use the Local Display preset. This configuration will be pre-populated with the values included here.

If the destination for HDR content will be other displays, use the Canonical Display preset. The values for this preset are listed here.

If your app requires different output settings, you are free to change any of these properties to suit the needs of your app. Let’s now go over a code example that uses a preset to capture HDR.

Begin by using SCShareableContent to get a list of available content.

The entire display will be captured here, so create an SCContentFilter for capturing the entire display.

Use the new preset API to get a configuration for the HDR canonical display.

And use the filter and configuration to create the SCStream.

To receive HDR screen samples in your app, add an output with the screen type.

And finally, start the stream to begin receiving HDR samples! Now, let’s say your app only needs a single screenshot instead of a stream. For this, the SCScreenshotManager offers a screenshot API. And I’m happy to tell you that you can also use this API to take screenshots in HDR! Just like SCStream, the SCScreenshotManager uses a content filter and a stream configuration.

And just like SCStream, presets are available for taking HDR screenshots. Here are the recommended preset values for taking HDR screenshots for either the local or canonical display.

I will now show an example of some code that uses the local display preset to take an HDR screenshot.

The first thing you will do is create a configuration for HDR local display.

If your app needs CMSampleBuffers for the screenshot, use the captureSampleBuffer function.

If CGImage is preferred, use the captureImage function.

So that’s how you can use the new APIs to get stunning HDR content from both streams and screenshots. If you would like to learn more about how to utilize HDR in your app, I invite you to check out these videos for supporting HDR images and media.

And with that, you can get started capturing HDR images and video with ScreenCaptureKit. To accompany your beautiful HDR streams, this year, you can also record the microphone along with your system audio.

By providing a new microphone output on a stream, your app can now capture three types of media: screen, system audio, and microphone.

For your app to capture microphone audio on a stream, the SCStreamConfiguration offers two new properties. captureMicrophone lets your app turn on microphone capture. microphoneCaptureDeviceID lets you choose which microphone device to capture.

To receive microphone samples in your app, you’ll use the SCStreamOutput. So let’s now step through some code for capturing microphone audio. You’ll first create a stream configuration to tell SCStream to capture the microphone. Set the captureMicrophone and microphoneCaptureDeviceID properties to capture the default microphone.

Create the SCStream with the filter and the configuration. And here’s where you add an output to the stream with the new microphone type. Now, start capture and use the SCStreamOutput’s didOutputSampleBuffer function to receive samples. So that’s all there is to it. Your app now gets all screen, audio and microphone content from an SCStream. One of the most common uses for receiving stream output is to write the samples to a file. You can now accomplish this by using a new API that has been added to the SCStream. Use this API to easily record and save screen, audio and microphone content in your app. Let’s go over how this works. Just like before, you will configure your SCStream to capture content. You will also use this same stream for recording.

To record, you just add this new output. The SCRecordingOutput can be configured using the SCRecordingOutputConfiguration.

This configuration lets you specify where to save the file, the file type and the video codec.

There is also the new SCRecordingOutputDelegate protocol that you may use to respond to the recording events on the stream. Let’s now check out some code for recording with this new API. You will first create an instance of the SCRecordingOutputConfiguration.

Set an outputURL to specify where to save the file.

You may optionally customize the file type and video codec for the saved file. Next, create an SCRecordingOutput instance with the recording configuration and set the delegate for receiving events.

You will then add the configured recordingOutput to the stream.

Call startCapture to start the stream, which will also start recording.

At some time in the future, you will call stopCapture to stop both the stream and recording. In some cases, you may want to keep streaming after you stop recording. To continue streaming content, call removeRecordingOutput instead to only stop recording.

During recording, your app will be notified about events via SCRecordingOutputDelegte. You will be notified when a recording starts, if an error occurs, and when the recording has successfully finished. At this moment the recorded file is now ready for use in your app.

And that’s it. This new recording API provides you a simple and convenient way to record screen, audio and microphone content in your app.

To recap, you can now use ScreenCaptureKit to capture HDR content in your app. A new microphone output is available to receive microphone samples from a stream. And finally, there is a new recording API, giving you an easy way to record screen, audio, and microphone content. To learn more about all of the amazing features ScreenCaptureKit offers, be sure to check out these previous videos. Thank you so much for watching.