Simulating & tools for testing pre-warming state

Hi there

I have a large codebase with many dependencies. I have a bug which I suspect is caused by accessing the keychain when the app has launched for pre-warming and as a result the keychain is inaccessible.

Are there any recommendations for simulating or testing this app state, and for identifying any static initialisers in my dependencies that could be contributing to my issue?

Thanks

Answered by DTS Engineer in 805814022

Does this sound like prewarming behaviour or is this likely something else?

So, the first thing here is that the vocabulary here is often pretty imprecise, with "Prewarm" basically being used as short hand for "my app ran in the background for reason I didn't expect".

Adding a bit of precision here, there are basically 3 different cases at play here:

  1. Your app was launched into the background because the system "anticipated" that the user was likely to use it "soon". This is "Prewarming".

  2. Your app was launched into the background because some other system service you interact with did that deliberately. As a trivial example, PushKit launches voip apps into the background when the receive a voip push. Note that in this case didFinishLaunchingWithOptions SHOULD be called, since the system goal is to "get your app running" so that your app can then do whatever it was "supposed" to do.

  3. Your app was woken in the background by the system because of something it asked/wanted to do.

Within that framework, (post-iOS 15) #1 does in fact guarantee that "didFinishLaunchingWithOptions" will not be called. As a practical matter, that's because prewarming was always intended to be relatively "invisible" and the simplest fix was to suspend apps in way that avoided the entire problem.

The problem here is that, over time, the number of situations that will trigger #2 and ESPECIALLY #3 has MASSIVELY increased. That's because:

  • We've added more and more APIs that are designed to wake apps in the background.

  • The increasing resources of the system mean that it's more "willing" to wake apps in the background than it might otherwise have been.

As a concrete example of that second point, in my experience, an app that uses the background NSURLSession "today" is significantly more likely to wake up in the background than it was 7 years ago. NSURLSession primarily works through #3 and increases in memory and better resource management mean your app is more likely to be suspended in the background... which means it's more likely to be woke up by #3.

Strictly speaking, prewarming also has a small role here, not because it actively wakes your app because it increases the chance that something else COULD wake your app again. As a concrete example of this, image that you use the same app "in the morning" but NEVER use that app after lunch. You usage patterns also mean that it's always pushed out of memory and terminated before bed time.

In the simple case, that means the app will always be launched into the foreground by "you" when you wake up at 7am.

Now add prewarming into the mix. The system notices you always run the app at 7am, so prewarming starts launching it earlier (making up a time, lets say 6am). Now your process was created at 6am but applicationDidFinishLaunching isn't called until 7am.

Then you add your NSURLBackgroundSession in. You schedule it's earliestStartDate at 6am so the data will there at 7am. That download finishes at 6:30... and your app is now woken and applicationDidFinishLaunching is called at 6:30. You can call that a "prewarm" launch, however, prewarming actually occurred WELL before your app was woken and, more importantly, what actually woke your app was an API that we clearly documented "could do that".

In any case, my experience has been that unless you're VERY aware of exactly what your code and EVERY library you include "does", it's very easy to end up with an app that periodically wakes up in the background.

Returning to here:

What I'm observing is that if I open the app, then lock my device, then sometimes after a period of being in the background it starts executing code again.

Looking at the log data, what jumps out at me here is the very short timing sequence:

2024-09-20 14:58:02 +0000, 💣 errSecInteractionNotAllowed, Background
2024-09-20 14:58:02 +0000, 💣 errSecInteractionNotAllowed, Background
2024-09-20 14:58:03 +0000, 0, Background
2024-09-20 14:58:04 +0000, 0, Inactive
2024-09-20 14:58:04 +0000, 0, Active

Basically, your app was woken up and in the foreground within at MOST 2 seconds. I suspect what's actually happening here is that because you're app is the foreground app (that the device is unlocking "too"), it's actually being woken at the same time the device is unlocking.

However, if you want to see exactly what's happening, there are two things I'd focus on here:

  1. Make sure you can CLEARLY differentiate between "my app was launched" (#2) and "my app was woken" (#3). I actually like to log the pid as part of every log message so that you can always "know" whether or not two log messages are from the same app instance. Being launched is very different than being woken and it's important to differentiate those two case.

  2. Since you're able to reproduce the issue relatively easily, you can use system log to sort out what's going on. The basic process here is:

  • Reproduce the issue, ideally with your own log data so you can easily determine the time your app woke up.

  • Capture a sysdiagnose. Friends don't let friends work out of the live console log.

  • Open up the console log archive, find the time your app was woken, then work "back" in time to find out who/why your app was woken. "runningboardd" is the daemon that manages the process assertions used to wake/suspend apps and it always logs what it's doing and who "asked" for it to be done.

Next, a warning here:

All it does is write a string to the keychain on the first loop with kSecAttrAccessible: kSecAttrAccessibleWhenUnlockedThisDeviceOnly

The "WhenUnlocked" attributes are trickier to use than they look. The problem here is that the keychain's lock state ISN'T really tied to your apps own lifecycle- the keychain could lock within seconds of entering the background (user backgrounds they're app and locks their device) or not lock for HOURS (the user switches over to a different app and continues doing there own thing).

That makes it VERY easy to write code that "seems" correct (but isn't) and works fine lots of the time... until it doesn't. The ONLY time it's actually safe to access at "WhenUnlocked" is when you're app is the foreground, but it will work "often enough" in the background that you won't realize there is a problem.

Practically speaking, if you're doing to use "WhenUnlocked" then your app needs to either:

  1. ONLY access the key(s) when your app is the foreground.

  2. Gracefully "handle" any access failure, typically by just ignoring the error unless you're in the foreground.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Re-ordering things, as the second part is going to be much longer than the first...

Are there any recommendations for simulating or testing this app state, and for identifying any static initialisers in my dependencies that could be contributing to my issue?

What's the issue you're actually having?

Pre-warming isn't the actual trigger/issue here (see below for why), but if you can provide more details about what's actually failing I might be able to point you toward the real problem.

I have a large codebase with many dependencies. I have a bug which I suspect is caused by accessing the keychain when the app has launched for pre-warming and as a result the keychain is inaccessible.

Issues around prewarming were very visible early in iOS 15's release and then basically "went away". The reason for that it very simple. We originally documented prewarming as:

"Prewarming executes an app’s launch sequence up until, but not including, when main() calls UIApplicationMain."

In terms of the actual implementation, what that actually meant was that the first thing "UIApplicationMain" actually did was release the process assertion which was created for the app launch which, in theory, would mean that the process would immediately suspend.

This is also why static initializers and other "at load" code became the problem. They run before main(), which means they could take out their own process assertions and those new assertions were what then allowed the app to remain awake and continue into the full launch sequence.

Ultimately, we resolved these issue by changing prewarm's implementation. Since mid/late iOS 15, the initial launch assertion is actually ended early in the load process of the first library dyld loads (libsystem). This occurs before ANY other library can load, ensuring that a prewarmed app will NEVER remain awake unexpectedly.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin

I appreciate the response, and thanks for the insight into pre-warming - I noticed there's been less noise about pre-warming since iOS 15 so that makes some sense given what you've mentioned. It's also pushed me to try and understand what exactly is the root cause of my issue, with the hope that it's not related to this feature!

What's the issue you're actually having?

I have a sample app (it's a little contrived so bear with me) which has been able to demonstrate the issue on an iOS 18 14 Pro physical device produced from the Xcode 16 Release build.

The app has no special UIBackgroundModes or Push notifications etc. All it does is write a string to the keychain on the first loop with kSecAttrAccessible: kSecAttrAccessibleWhenUnlockedThisDeviceOnly, and then every subsequent loop reads this data. This repeats infinitely every 0.5 seconds and is started just before returning true in didFinishLaunchingWithOptions. The status of the read query is then logged to a file with a timestamp and the UIApplicationState.

What I'm observing is that if I open the app, then lock my device, then sometimes after a period of being in the background it starts executing code again. In this case, it tries to run my looping code, which reads the keychain which then fails with errSecInteractionNotAllowed, probably because the keychain is locked as the app is in the background.

Does this sound like prewarming behaviour or is this likely something else?

Here's some logs that were produced from the sample app:

date, keychain query status (OSStatus), UIApplication.State
2024-09-20 14:57:27 +0000, 0, Active
2024-09-20 14:57:28 +0000, 0, Active
2024-09-20 14:57:28 +0000, 0, Active
2024-09-20 14:57:29 +0000, 0, Active
2024-09-20 14:57:29 +0000, 0, Inactive
2024-09-20 14:57:30 +0000, 0, Inactive
2024-09-20 14:57:30 +0000, 0, Background
2024-09-20 14:58:02 +0000, 💣 errSecInteractionNotAllowed, Background
2024-09-20 14:58:02 +0000, 💣 errSecInteractionNotAllowed, Background
2024-09-20 14:58:03 +0000, 0, Background
2024-09-20 14:58:04 +0000, 0, Inactive
2024-09-20 14:58:04 +0000, 0, Active
2024-09-20 14:58:05 +0000, 0, Inactive
2024-09-20 14:58:05 +0000, 0, Inactive
2024-09-20 14:58:06 +0000, 0, Inactive
2024-09-20 14:58:06 +0000, 0, Background

Does this sound like prewarming behaviour or is this likely something else?

So, the first thing here is that the vocabulary here is often pretty imprecise, with "Prewarm" basically being used as short hand for "my app ran in the background for reason I didn't expect".

Adding a bit of precision here, there are basically 3 different cases at play here:

  1. Your app was launched into the background because the system "anticipated" that the user was likely to use it "soon". This is "Prewarming".

  2. Your app was launched into the background because some other system service you interact with did that deliberately. As a trivial example, PushKit launches voip apps into the background when the receive a voip push. Note that in this case didFinishLaunchingWithOptions SHOULD be called, since the system goal is to "get your app running" so that your app can then do whatever it was "supposed" to do.

  3. Your app was woken in the background by the system because of something it asked/wanted to do.

Within that framework, (post-iOS 15) #1 does in fact guarantee that "didFinishLaunchingWithOptions" will not be called. As a practical matter, that's because prewarming was always intended to be relatively "invisible" and the simplest fix was to suspend apps in way that avoided the entire problem.

The problem here is that, over time, the number of situations that will trigger #2 and ESPECIALLY #3 has MASSIVELY increased. That's because:

  • We've added more and more APIs that are designed to wake apps in the background.

  • The increasing resources of the system mean that it's more "willing" to wake apps in the background than it might otherwise have been.

As a concrete example of that second point, in my experience, an app that uses the background NSURLSession "today" is significantly more likely to wake up in the background than it was 7 years ago. NSURLSession primarily works through #3 and increases in memory and better resource management mean your app is more likely to be suspended in the background... which means it's more likely to be woke up by #3.

Strictly speaking, prewarming also has a small role here, not because it actively wakes your app because it increases the chance that something else COULD wake your app again. As a concrete example of this, image that you use the same app "in the morning" but NEVER use that app after lunch. You usage patterns also mean that it's always pushed out of memory and terminated before bed time.

In the simple case, that means the app will always be launched into the foreground by "you" when you wake up at 7am.

Now add prewarming into the mix. The system notices you always run the app at 7am, so prewarming starts launching it earlier (making up a time, lets say 6am). Now your process was created at 6am but applicationDidFinishLaunching isn't called until 7am.

Then you add your NSURLBackgroundSession in. You schedule it's earliestStartDate at 6am so the data will there at 7am. That download finishes at 6:30... and your app is now woken and applicationDidFinishLaunching is called at 6:30. You can call that a "prewarm" launch, however, prewarming actually occurred WELL before your app was woken and, more importantly, what actually woke your app was an API that we clearly documented "could do that".

In any case, my experience has been that unless you're VERY aware of exactly what your code and EVERY library you include "does", it's very easy to end up with an app that periodically wakes up in the background.

Returning to here:

What I'm observing is that if I open the app, then lock my device, then sometimes after a period of being in the background it starts executing code again.

Looking at the log data, what jumps out at me here is the very short timing sequence:

2024-09-20 14:58:02 +0000, 💣 errSecInteractionNotAllowed, Background
2024-09-20 14:58:02 +0000, 💣 errSecInteractionNotAllowed, Background
2024-09-20 14:58:03 +0000, 0, Background
2024-09-20 14:58:04 +0000, 0, Inactive
2024-09-20 14:58:04 +0000, 0, Active

Basically, your app was woken up and in the foreground within at MOST 2 seconds. I suspect what's actually happening here is that because you're app is the foreground app (that the device is unlocking "too"), it's actually being woken at the same time the device is unlocking.

However, if you want to see exactly what's happening, there are two things I'd focus on here:

  1. Make sure you can CLEARLY differentiate between "my app was launched" (#2) and "my app was woken" (#3). I actually like to log the pid as part of every log message so that you can always "know" whether or not two log messages are from the same app instance. Being launched is very different than being woken and it's important to differentiate those two case.

  2. Since you're able to reproduce the issue relatively easily, you can use system log to sort out what's going on. The basic process here is:

  • Reproduce the issue, ideally with your own log data so you can easily determine the time your app woke up.

  • Capture a sysdiagnose. Friends don't let friends work out of the live console log.

  • Open up the console log archive, find the time your app was woken, then work "back" in time to find out who/why your app was woken. "runningboardd" is the daemon that manages the process assertions used to wake/suspend apps and it always logs what it's doing and who "asked" for it to be done.

Next, a warning here:

All it does is write a string to the keychain on the first loop with kSecAttrAccessible: kSecAttrAccessibleWhenUnlockedThisDeviceOnly

The "WhenUnlocked" attributes are trickier to use than they look. The problem here is that the keychain's lock state ISN'T really tied to your apps own lifecycle- the keychain could lock within seconds of entering the background (user backgrounds they're app and locks their device) or not lock for HOURS (the user switches over to a different app and continues doing there own thing).

That makes it VERY easy to write code that "seems" correct (but isn't) and works fine lots of the time... until it doesn't. The ONLY time it's actually safe to access at "WhenUnlocked" is when you're app is the foreground, but it will work "often enough" in the background that you won't realize there is a problem.

Practically speaking, if you're doing to use "WhenUnlocked" then your app needs to either:

  1. ONLY access the key(s) when your app is the foreground.

  2. Gracefully "handle" any access failure, typically by just ignoring the error unless you're in the foreground.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Simulating & tools for testing pre-warming state
 
 
Q