Keep colors consistent across captures

Welcome to "Keep colors consistent across captures". Hello! My name is Yuko. We all know that the iPhone takes wonderful immersive photos. The quality of the light and how it interacts with colors and textures of the objects and people in the scene is so important and really helps you relive the precious moments you recorded with your iPhone. Lets see some examples.

What is a sunset in the African Savannah without the warm glow. The autumn leaves look gorgeous in the dappled golden hour lighting. The warm indoor light juxtaposed with the cool outdoor light makes this pub seem even cosier. The crisp directional lighting gives Matt a purposeful look.

The strong saturated light in this botanical light show, is the essence of this photo.

The reflections of the Sun setting over the water in these scenes bring a magical quality to the photos. And the contrast between light and shadow in these scenes set the mood and helps draw the eye. These are all great examples of the ambient light being an important part of the aesthetics of the photo. Normally this is exactly what I want when I take a snap with my iPhone. But sometimes, the effects of the ambient light are undesired or even a distraction. Such as in photos of your skin. For example, you may want to record how a wound is progressing and so be interested in the color of the surrounding area independent of the ambient light. In this example the full extent of the bruise is more apparent in the Constant Color image.

These are the kind of photos that the Constant Color API caters for. So lets dig in.

In the rest of this talk, I’ll cover what color constancy is, provide examples of when it could be used, explain the principles of how Constant Color works. Then we’ll go through some example code, mixed with live demos. Finally, I’ll provide some information on how the Constant Color feature was validated. So what is a color constancy? It is a property of how human visual perception works.

If you know the color of an object, you see it as that color almost regardless of the ambient lighting under which you view that object. For example, a bright green granny smith apple remains green to you whether viewed under daylight, or viewed under the warm glow of a camp fire. This is because, humans adapt to the lighting condition around them automatically and correct for some of the cast of the ambient light. The human brain is able to recognize objects that have been seen before and remember their color. The Constant Color feature provides a new capture mode, which determines the color of objects, people, and materials without having to memorize and recognize them from previous captures. There are many use cases where I want to capture the colors of the people and objects in the scene consistently, regardless of the ambient illumination. I will highlight two of these here: product photography and photos of skin.

Constant Color imaging is important for product marketing photos, especially when consumers are buying something they haven’t seen in real life. I see two types of photos when I browse products in online shops. The first type is beautiful images of the product being used and enjoyed by consumers. The iPhone already caters superbly for these photos with its normal photo and portrait modes.

The second type of photos are to help people gauge the properties of the products such as size and color. Constant Color can help your customers understand the colors better because it produces photos with consistent lighting conditions. These consistent images with known lighting conditions work with users color managed displays on their mobile phones, tablets and computers. This allows the customer to interpret the product colors regardless of the viewing conditions.

All of this allows people to better visualize the products in their home, office, vehicle or workplace. This enables customers to make informed purchasing decisions, which can help with customer satisfaction.

We have all been there, we ordered some clothes online and the item that arrived isn’t quite what was expected. I will demonstrate this using a potted plant as example.

Large manufacturers and professional product photographers take these photos in a professional studio, with multiple bulky lighting rigs, photographic backdrops and digital cameras. Smaller companies and home creators may have a lightbox which they use to control the lighting for their product photos. I am pleased to say that we have been able to pack the power of a photography studio and lighting rig into your iPhone, thanks to Portrait Mode and the new Constant Color API. Lets look at an example.

Here are some potted violets and I would like to capture the gorgeous colors. As you can see, this is just a normal room with warm white indoor lighting and no special lighting rig. Lets look at some of the photos.

On the left is a normal iPhone 15 photo, on the right is the Constant Color image, which has a more neutral cast and even tone across the image. Lets make it more challenging and add another strong colored light in the scene. Now it is unclear whether the pot is a neutral color or a terracotta color.

If we look at the Constant Color image on the right, we see that the foreground with the violets is largely unchanged in both tone and cast compared to the previous photo.

In fact, I can add saturated light of any color to the scene and get great results. The left image only has a fairly neutral indoor lighting. In the other images I have added red, green and blue saturated light to the indoor lighting respectively. The aesthetic pipeline images depict the cast caused by the ambient light. This is expected and desired for the capture mode.

Adding the Constant Color images on the bottom, we get much more consistent renderings across the lighting conditions. These are extreme examples of strongly coloured light, but don’t worry: Constant Color corrects subtle casts just as well.

In case you think we’re fudging the images for this presentation, take a closer look in the highlighted areas. You can see the cast is not fully corrected in the shadow of the leaves. We’ll discuss why this happens a little later.

Of course, it is also possible to capture accurate colors of people and skin tones, which is hugely beneficial for monitoring changes in skin hue over time. As you can see under the same lighting conditions that we saw earlier, Constant Color renders these four different skin tones far more consistently across the different lightings. It can be difficult to remember how skin color changes over time especially if the change is slow, or if you view it under different light conditions.

So if you have ever wondered whether a bruise was starting to fade or was still darkening, then a Constant Color capture is more useful than an aesthetic photo. If you have wondered whether a rash or insect bite was getting redder, then Constant Color imaging provides a better photo for that purpose than a regular capture. As we can see in this example, where the very warm indoor light at 2800K makes the cupping mark very severe in the aesthetic photo. Both the absolute color in a single image, as well as the progression of color changes over time are informative.

This is especially true for people with darker skin tones, where the color changes may be more subtle than in people with lighter skin tones.

And of course performance across skin tones is essential in our products, so we will talk about validation later in the presentation.

Lets dig a bit deeper in to how the Constant Color feature generates these images.

The Constant Color API makes use of the improved flash hardware first launched in the iPhone 14 family of phones. The properties of each individual iPhone’s flash are precision measured during the production process. Similarly, the properties of each individual camera are carefully measured too.

In a completely dark room, the flash would be the only light source in a photo. In this scenario, we know the properties of the camera and the light source, so we can accurately determine the colors of the objects and the materials in the photo. In order to capture images outside of a dark room, the API analyses the brightness increase that the flash makes and uses that to predict what the equivalent dark-room image would be. The API does this by taking a flash image and a non-flash image in rapid succession to minimize change in the scene or lighting conditions.

The ambient image is then registered to the flash image to mitigate motion and to ensure that the corresponding regions in the image have the same content.

Then, in the linear scene-referred domain the two images are normalized for their relative exposures. This allows for comparisons between the two frames to be performed.

Using computational photography and machine learning, these two images are jointly processed along with the precision factory calibration of both the flash and the camera. The outcome of this is a prediction of what the equivalent dark room image would be.

Finally, the dark room image is rendered as if it were lit by a standard D65 illuminant. It then receives global tone mapping and a 2.2 gamma encoding. In order to preserve subtle changes in material lightness across the field of view, no local tone mapping is applied.

Now, lets look at how you can capture Constant Color images in your app. There are six things you need to be aware of. First, check that your device supports the Constant Color API. Second, the pipeline must be configured to expect potential Constant Color requests. Third, to trigger a Constant Color photo an AVCapturePhotoSettings property must be set. The API provides two mechanisms to know which parts of the Constant Color image have high accuracy colors and which parts may have more variance in the color rendering. The first is a 2D confidence map, which I will speak about in greater detail later. The second is a single summary confidence value. Finally, there is an option to provide a fallback frame for when your app decides not to use the Constant Color image. Cool, lets look at some example code, starting with checking device support.

This is a snippet from the new Constant Color camera sample app, available on Apple’s developer website. I will refer to this sample app throughout the rest of this presentation. In order to check if your device supports the Constant Color API, we need to check one of the AVCapturePhotoOutput properties. Specifically, the isConstantColorSupported property. Note that this property can change if you switch cameras or formats. In terms of devices, currently the iPhone 14 family, iPhone 15 family and 2024 iPad Pros are supported.

After checking to see if the device supports Constant Color, the next step is to configure the AVCapture pipeline to be able to deliver Constant Color images.

Constant Color is a photo only mode due to the pre flash sequence required. The device type needs to be builtInWideAngleCamera or builtInDualWideCamera. The AVCaptureSession sessionPreset needs to be set accordingly.

Since the Constant Color uses the flash, it will only work if the AVCaptureFlashMode is set to auto or to on. Setting the flashMode to off will cause an exception when a Constant Color image is requested.

It is also not possible to get RAW images with the Constant Color API, so the rawPhotoPixelFormatType must be set to zero. The Constant Color API is a repeatable capture mode, and so does not support different quality settings. It has the same runtime and output for all of the prioritization settings. Looking back at our code snippet, you can see where the sessionPreset is configured. The flash mode is configured in the AVCapturePhotoOutput settings, which we’ll look at shortly. Finally the isConstantColorEnabled photoOutput property must be set. In the sample code the isConstantColorEnabled property is set using the is ConstantColorSupported property to only enable Constant Colors when the device supports it. Note that changing the isConstantColorEnabled property invokes a lengthy reconfiguration of the render pipeline, so I suggest setting this once at the start of your app along with any other pipeline settings you need.

Now that we have confirmed that the Constant Color API is supported, and have configured the pipeline, we have to specify when we want a Constant Color image.

Here is a new snippet from the Constant Color camera sample app. This is where the per-capture settings in AVCapturePhotoSettings are set. The flash mode is set according to a UI toggle. A Constant Color image is requested by setting the isConstantColorEnabled property. In the snippet the isConstantColorEnabled property is hooked up to a UI toggle. If we don’t set this property, the pipeline defaults to a regular aesthetic photo.

Alright, its live demo time! I can make things interesting by setting up a challenging lighting scene.

Lets compare outputs from Constant Color imaging and the aesthetic photography modes.

Using the Constant Color camera sample app, I will take a regular photographic capture.

The iPhone has captured the subtle change in colors across my hand and color checker that was caused by the multiple light sources. Aesthetically this is great, and exactly what I want when capturing memories, such as a picture of my son illuminated by candle light while blowing out the candles of his birthday cake. Now I will activate Constant Color in the sample app.

Lets take a picture. Since we’re using the flash, there is the normal pre-flash and flash capture sequence, and once that completes we get our Constant Color image.

See how my skin tone is even across the image and the color checker is rendered with a neutral cast throughout.

Of course we don’t need the color checker for the Constant Color API to work. In fact I don’t need to place any reference material or chart in the photo.

I can take another image without it and my skin tone and the desk surface are still rendered consistently.

I will dig into how the API’s consistency was confirmed and validated a little later.

As you have seen, Constant Color uses the flash. Specifically, it looks at how much the flash brightens the scene and then analyses that brightness increase.

The iPhone flash has been carefully optimized to balance its light output power, physical size and battery usage. This means that under certain capture conditions, the flash may not be powerful enough to brighten the scene sufficiently for the Constant Color API to be able to analyse the difference. When this occurs the consistency of the colors produced by the API may decrease. Examples of when the flash may not be bright enough include capturing landscape scenes, astrophotography, or taking images in direct sunshine. The Constant Color API provides the tools to detect when the rendered colors may be less accurate, so you can have an alternative flow through your app For every Constant Color photo, the API analyzes each portion of the image to see if the brightness increase from the flash has sufficient signal to noise ratio, and if there is any clipping from reflections. This creates a confidence map, which is provided as a second CVPixelBuffer output that your app can analyse to determine if your regions of interest within the image have high confidence. The confidence map is a 2D map of floating point values, in the range of zero to one. Each pixel in this map corresponds to a portion of the Constant Color photo. A value of one in the constantColorConfidenceMap corresponds to full confidence in the accuracy at that point in the image. A value of zero, corresponds to no confidence. The threshold for your app will be unique to your use case. I suggest starting with a threshold of 0.8 to 0.9 and refining from there. We’ll look at some of these maps in the next demo shortly. The confidence map is the most detailed information provided to determine if a region of interest has accurate color. However, there may be occasions when you prefer to check a single value rather than a 2D map. For that we have created a summary statistic of the confidence map.

The constantColorCenterWeightedMeanConfidenceLevel property could be used when the region of interest is unknown, or as an indicative metric to determine whether a more detailed analysis of the confidence map should be performed. This is a weighted average, where the center pixels contribute more, because that’s where people tend to frame the primary subject. There are other ways to reduce the 2D confidence map to a single value. If your app needs one then the 2D map is available for you to calculate it from. In the Constant Color camera sample app, the center weighted confidence value is displayed above the confidence map.

What do you do if the confidence of a capture does not meet your app’s threshold? You could retake the image, or you can use the optional ambient fallback photo. Let’s check it out. To opt in to using the fall back all you have to do is set the isConstantColorFallbackPhotoDeliveryEnabled property of AVCapturePhotoSettings prior to taking the capture. In the sample code this property is also controlled with a UI toggle.

Opting in to receiving the fallback frame will cause the didFinishProcessingPhoto callback to be triggered twice. Here is the code for the delegate in the sample app. You can see within the callback that the isConstantColorFallbackPhoto property is checked to determine whether the fallback photo or the Constant Color photo is being received.

Lets check it out in our second demo.

First, I will repeat the earlier experiment with my hand on the desk.

Here we see a similar image as before with my hand rendered neutrally on the desk surface. If I swipe to the confidence map. We see that the map is almost completely white, with some minor shadowing where my hand casts a flash shadow due to the baseline between the flash and the camera. Now lets see what happens when there is some distance, so the flash doesn’t brighten the background.

Here is the Constant Color output. The flash strength was automatically set to illuminate my hand with out blowing it out. So the background is not brightly illuminated by the flash. My hand’s skin color still looks consistent. The confidence map, as expected, shows that my hand has high confidence but the background has low confidence. You can use the confidence map in your app to determine if the portion of the image that is important to you has accurate colors.

It’s worth going back to the Constant Color output briefly. Note that even though there is low confidence in the background, the entire image still has a pleasing rendering without weird colors or excessive noise. This is because the API uses information from the flash before the brightness difference was calculated to fill in the gaps. So even though there is low confidence in some of the regions of the Constant Color output, the entire image still remains viewable.

Now, lets see what happens when there are some specular reflections from the flash.

In the Constant Color image, we can clearly see there are some strong reflections from the flash on the spoon. In these areas we cannot accurately measure the brightness difference because the flash illumination is scattered or pixel values have clipped. This is evident in the confidence map as there is no spoon. This is because the shiny reflections have been reported to have low confidence. The reason they have low confidence is that the scattering and clipping prevents the exact increase in brightness from being determined.

By now, you’re probably wandering how accurate the Constant Color API images are. We tested the API under a range of indoor lighting scenes ranging from 10 Lux to 800 Lux. This corresponds to a darkness where it is hard to frame a photo up to very bright indoor conditions similar to an operating room. At each lux level, the color temperature was varied from a warm 3000K to a cool 7500K. At each brightness and lux level, we tested the Plankian white point, a magenta tint and a green tint straddling the locus. For each lighting condition we captured an image of a Calibrite ColorChecker containing strong saturated colors, neutral tones, foliage and skin tones. We also captured Pantone’s SkinTone Guide which consists of 138 skin tones covering a wide range of both albedo and undertone.

And of course, we took photos of real people too.

For this data we converted the P3 RGB output of the Constant Color API in to the CIEL*a*b* color space and calculated the variability of each color across the entire envelope of lighting conditions.

Compared to normal photography-focused iPhone captures, the Constant Color API has drastically increased the repeatability of the image rendering. The variance of surface colors due to ambient light has been decreased by 87% for the iPhone 15 family. Please see the upcoming white paper for a detailed breakdown of the validation methodology and accuracies. Oh, and one more thing. A consequence of Constant Color predicting the dark room flash image without ambient light, is that shadows in the scene cast by the ambient light are reduced or removed entirely. On the left is a normal iPhone capture where the shadow cast by my hand and the iPhone can clearly be seen. On the right is the Constant Color output, with the shadow removed. Those of you who watched the launch of the 2024 iPads, may recognize these results. Because Constant Color is an integral part of the shadow removal in the new iPad Pro document scanning feature. For reference, the doc scanning features uses a threshold of 0.9 for the constantColorCenterWeightedMeanConfidenceLevel.

Now, let’s summarize what we have covered. Constant Color is a new capture mode. This mode prioritizes preserving the color of items in the photo over the mood and ambiance created by the light. We are able to measure color without putting a known material or card in the scene. Constant Color images are highly repeatable and can mitigate the effect of varying indoor lighting on images. Shadows caused by the ambient light are reduced or even entirely removed in Constant Color images. The API provides a detailed confidence map and a fall back image, giving you the control and certainty to use Constant Color images in your app.

We are very proud to be able to release this new capture mode and can’t wait to see what you do with it. Thank you!