Explore media metadata publishing and playback interactions

♪ ♪ Hi, my name is Nik, and I'm an engineer on the Video team. Today I'm excited to talk to you about media metadata publishing and playback interactions. So what exactly does that mean? There are a number of places on Apple devices where playback information is displayed, and where playback can be controlled. For example, the Now Playing section of Control Center displays the artwork, title, and progress for media that is currently playing on the device. It also lets you play, pause, or even skip forward or backward. Expanding the Now Playing tile shows more details, like the artwork and progress. it also allows you to scrub and increase or decrease the volume. Lock Screen also displays the same information and controls, giving users a convenient place to check in on progress, pause, or even AirPlay to another device without needing to unlock.

No matter what device is playing, the Now Playing app on Apple Watch provides the same experience. It even has an Apple TV remote built in.

On tvOS when using AVKit, the info overlay when controls are presented will show title and chapter information. When you swipe down to the info pane, more details like the artwork and description are shown.

Holding the TV button on your Apple TV remote shows Control Center, which like iOS has a Now Playing tile that also can be expanded. When audio content starts playing from the background on tvOS, be it from pressing the play button on the remote, or from selecting a track in the Music app from another device, a notification with the Now Playing information is presented. Additionally, after a brief period of inactivity on tvOS when playing audio, a full screen overlay showing what's currently playing is presented.

Lastly, on iOS, the Control Other Speakers and TVs button lets you view the Now Playing information on all of your devices, as well as control playback.

With the growing number of devices and UIs where Now Playing information is presented and where playback can be controlled from, properly publishing Now Playing information and responding to remote commands is more important than ever. Over the course of the rest of this session, we will cover responding to playback interactions in the form of remote commands, automatic metadata publishing, publishing with AVKit, and manual publishing. When using AVFoundation for media playback, the best way to publish Now Playing metadata and respond to playback interactions is using the MPNowPlayingSession class.

Historically, this class has only been available on tvOS, but is now available on iOS 16.

It is used to represent a distinct playback session, and offers control over Now Playing status if your app contains multiple active sessions. It supports both manual metadata publishing, as well as the new automatic publishing available in iOS and tvOS 16.

MPNowPlayingSession shouldn't be used on tvOS when using AVKit, which has its own automatic publishing mechanisms we'll cover later in the session.

Being the "Now Playing" app means that your app is what will populate Control Center, Lock Screen, etcetera, and receive the playback controls when the user, for example, presses pause from one of those interfaces. With MPNowPlayingSession, you can represent multiple concurrent playback sessions within a single app. However, when using multiple sessions, your app must promote one as the active session that will appear throughout the system when remote controlling your app. For example, with Picture in Picture you may have two concurrent playback sessions, where the full screen playback should be considered the active Now Playing session. The system also has a few heuristics to qualify apps as Now Playing eligible. First, you must register a handler for at least one remote command. As you can imagine, an app that won't respond to any playback interactions is most likely not an ideal candidate to show up as the Now Playing app. Second, your apps AVAudioSession must be configured with a non-mixable category and category option. Mixable playback categories and options are generally used when playing back notifications, and therefore this is a good indication to the system that whatever is playing is also not a good candidate for Now Playing. Here's a few examples to help understand playback sessions. In this example there is a single piece of content playing, so this would be represented using a single MPNowPlayingSession. If your app supports PiP, you would have two MPNowPlayingSessions: one for the main player, and one for the PiP playback. A more complex scenario would be a single MPNowPlayingSession that has several players. In this example, we have four players, one in each quadrant, showing different points of view for the same race. Players added to the same MPNowPlayingSession should always be part of the same content. And here's how each of these example sessions would be instantiated. The first, we're just playing a single piece of content, so we have the single session with the single player. The second example is using Picture-in-Picture, so we have two sessions, each with a single player. The first being the full screen content, and the second being the content in PiP. The last example, the multi-view race, is represented with a single session with four players. When an app does have multiple sessions, it's the apps responsibility for promoting a given session as active when applicable. For example, if media is playing in Picture-in-Picture, if the user expands it to be full screen, the previously full screen session should no longer be active, or Now Playing, and the PiP session that is now full screen should become active. This transition can be done by calling becomeActiveIfPossible on MPNowPlayingSession. Now that we've covered the basics of setting up instances of MPNowPlayingSession and controlling the Now Playing session, let's talk about receiving and responding to remote commands, be it from Lock Screen, or from a HomePod in another room. Let's start off with a basic example of registering for the play and pause command. Doing so will enable your app to receive callbacks when the user presses play or pause from another device, or issues that command using Siri. First, we instantiate our MPNowPlayingSession. Since we only have one session, we don't need to invoke the 'becomeActiveIfPossible' method. When you only have one session, it will be the default session when your app is the Now Playing app. Each MPNowPlayingSession instance has its own MPRemoteCommandCenter instance, which is used to declare which remote commands your playback session can respond to. Next we add a handler for the playCommand where we invoke the play method on our player, and return success. Then we do the same for the pauseCommand. You should add handlers for every command that your app supports and that is applicable for the currently playing content. Another example is the skip forward and skip backward command. This command should be used for most content, and wouldn't be applicable, for example, live streams where jumping forward isn't possible. First we have to indicate what our preferred intervals are, or the number of seconds we prefer to jump in either direction. In this case, we use 15 seconds. Then similar to what we did for the play and pause commands, we add a handler that will be invoked when the user presses the skip forward button or asks Siri to skip forward. In our handler, we will be receiving an MPSkipIntervalCommandEvent, so first we will cast the event to that type. We then calculate the new elapsed time by taking the current time, and the interval provided to us in the MPSkipIntervalCommandEvent, seek to it, and return success to indicate that we jumped to the new position. It's also possible that your app has situations where a command is temporarily not allowed, for example skipping forward while in an advertisement. In that case, the skipForwardCommand can be disabled. Now that we're responding to remote commands, we will cover automatic metadata publishing. Automatic publishing takes the hard work out of keeping metadata accurate by automatically maintaining metadata properties it can observe directly from the player such as duration, the current elapsed time, the playback state, and playback progress. If the content has ads baked into it that shouldn't contribute to the total duration and elapsed time, it can also take care of calculating the net time and report that instead. Other metadata such as the title, description, and artwork can be added to the AVPlayerItems directly using the nowPlayingInfo property. In this example, we will use automatic publishing to do the bulk of the work and set the title and artwork ourselves. First, we create a new MPMediaItemArtwork instance, passing in the artwork image. Most apps will perform a network request to fetch this. Then we set the string title of the content. Then we take our artwork and title and set them as the nowPlayingInfo dictionary on the current player item using MPMediaItemPropertyTitle and MPMediaItemPropertyArtwork. Now Playing metadata can consist of both MPMediaItemProperty's and MPNowPlayingInfoProperty's. Lastly, we create our MPNowPlayingSession instance passing in our player, and set automaticallyPublishNowPlayingInfo to true. Once automaticallyPublishNowPlayingInfo is set to true, the MPNowPlayingSession instance will begin observing the player for state changes such as scrubbing, play/pause events, or the current player item changing. Here's another example where we will show how to use automatic publishing for instances where ads are baked into the asset and you don't want the total duration or current elapsed time to include ad time. To do this, we'll create instances of MPAdTimeRange for every ad that we have baked in. In this example, we have a single 30-second ad that starts at the very beginning. So we create it with a starting point of zero, and a duration of 30 seconds. Similar to how we did the title and artwork earlier, we simply add an array of MPAdTimeRange's to the nowPlayingInfo dictionary on the player item using the MPNowPlayingInfoPropertyAdTimeRanges. Then just as we did before, create the MPNowPlayingSession and enable automatic publishing. Next is metadata publishing with AVKit. Publishing Now Playing metadata with AVKit on tvOS works very similar to MPNowPlayingSession: metadata is added directly to the AVPlayerItem, and values like elapsed time, duration, and playback state are published and kept up to date for you. The metadata gathered from the player and asset directly, combined with the metadata provided by your app on the AVPlayerItem are also used to populate the info pane in the player UI. AVKit also takes care of registering for and responding to remote commands. Using AVKit is the best and easiest way to integrate with the platform features we've discussed so far, as well as others such as AirPlay and Picture-in-Picture. Setting the metadata when using AVKit is done using the externalMetadata array on the AVPlayerItem, which consists of the AVMetadataItem instances to describe your content. You usually up setting three values on each AVMetadataItem. First, the identifier, which is the key to indicate what metadata the AVMetadataItem represents. For example, AVMetadataCommonIdentifierTitle for the content title, or AVMetadataCommonIdentifierArtwork for the artwork. Second is the value. For title, this would be a string containing the title. For artwork, this would be an NSData instance containing image data. The dataType is used to indicate the format of the artwork provided. If it contained JPEG data, kCMMetadatabaseDataType_JPEG would be used. Lastly, the extendedLanguageTag is used to indicate the language used for strings such as the title and description. Most of the time, the value "und" should be used here to ensure all audiences see the same values. You may be tempted to use "en-us" if the values are in English, but doing so would cause devices with the language set to any other language such as Spanish to not show the metadata.

Here we have an example where we are setting the artwork and title. First, we grab the artwork image data from our bundle. Most apps will fetch this from a network resource. Then we instantiate a new mutable AVMetadataItem. We set the identifier to .commonIdentifierArtwork. Then we set the value as the raw artwork image data as NSData. Since the image data is JPEG, we set the dataType to kCMMetadataBaseDataType_JPEG. If your artwork was instead a PNG, you would use kCMMetadataBaseDataType_PNG. Because we want this metadata to be visible to users with devices set to any language, we set the extendedLanguageTag to "und," or "undefined." We then repeat the same steps for the title, using .commonIdentifierTitle, and the string title for the value, and "und" once again for the extendedLanguageTag. Once we've set up all of our metadata items, we add them to an array and set it to the AVPlayerItem's externalMetadata property. Now that we have the artwork and title added to the player item, you can see how this maps to what is shown in Control Center and Lock Screen on iOS. Like artwork, there are other metadata types that can be set such as the description, subtitle information, and content rating. Your app should set as many of these as possible to provide the user with as rich of an experience as possible. So far we've covered automatic publishing with MPNowPlayingSession and publishing with AVKit. But MPNowPlayingSession and its automatic publishing feature require passing an AVPlayer instance to it. That may not be an option for all apps, and manual publishing is still possible. Publishing manually requires that you provide values for all metadata. Unlike automatic publishing, information such as elapsed time and playback rate can't be determined by the system for you. This means that you have manual fine grain control over low level playback state, and your app is responsible for keeping it accurate over time as playback changes. Note that registering for and responding to remote commands is still required as well, and because we are not using MPNowPlayingSession, the shared instance of MPRemoteCommandCenter must be used. Here's a basic example showing how to update the Now Playing Info dictionary. First, we create an MPMediaItemArtwork instance containing the image, similar to what we did for automatic publishing. Then, we create a dictionary containing the metadata that we have available. In this case, we set the title, artwork, and the player values duration, elapsed time, and playback rate. We then set it on the MPNowPlayingInfoCenter default instance. Updates to this metadata should be made any time significant changes happen during playback, such as a play or pause, the user scrubs forwards or backwards, or a new piece of content begins playing. You do not need to update elapsed time periodically. The system will always infer the correct elapsed time based on how much time has passed since the last update. Now that you are familiar with all of the different ways to publish Now Playing metadata and respond to remote commands from other devices and interfaces, you should integrate to maximize the user experience. It's easier than ever. Existing integrations can benefit too– switching to automatic publishing is an easy way to prevent future regressions and minimize the amount of code you must maintain. For more information, see MediaPlayer on developer.apple.com. Thanks for watching.