Xcode Cloud build phase has massively slowed down

Hi, recently something happened in Xcode Cloud, and the build times increased massively. In our CI workflow we run test, and one of the phases executes xcodebuild build-for-testing: the duration of this phase moved from about 8 minutes to about 36 minutes, without any major changes on our side.

We observed this:

  • in different apps;
  • even when running a build on a old tag.

All other phases don't seem affected, it's just the xcodebuild build-for-testing phase.

I'm attaching 2 screenshots of the logs from AppStoreConnect that show the duration of each phase, where build 7600 is older and 7700 is more recent (but was run on a tag from a few weeks ago, to exclude issues related with the code base itself).

We had not other option than to disable the CI checks for PRs, because they would take too long, and use up too much of Xcode Cloud time.

What's happening?

Thanks

Answered by Developer Tools Engineer in 789798022

To summarize, a recent performance issue with Xcode Cloud caused some projects to take an unusually long time to complete. We have corrected this issue, and added extra compute time to the affected accounts.

We understand the impact this had on developers and are reviewing our process so that we can respond faster in the future. The developer system status page is the fastest way to see details about known issues with Xcode Cloud.

Please continue to file feedback requests and post on these forums if you are having issues with our service.

Thank you all for your feedback.

Same here. Our builds became about two times slower (10 - 15 vs. 30 minutes).

I think it started about a week ago.

Same here, it's very very slow. It's using all of our hours

Likewise - this is really unacceptable Apple... come on!

Also happening to me, there's something wrong with the builds. How can we rely on Apple if things like this keep happening?

Same! We went from 10 min to 60 min+. It eats up all our compute hours real fast now.

It seems to have happened sometime on May 21st for us. I've filed a bug in Feedback Assistant, but no response yet.

This has been happening to me as well. My builds have jumped from around 9 minutes, to around 17-20 minutes, so about a 2x increase in build times. I'm using the free tier of Xcode Cloud though, so I feel like I can't quite complain about it.

From the log's timestamp, you can see that after xcodebuild-archive finishes, the next step should be development-export-archive. However, the start time of development-export-archive is 50 minutes later. There is no log for these 50 minutes in

Noticed the same but was unaware of this post so just adding another datapoint. Noticed been happening about a week ago

It appears to be fixed now, at least for us. Our last few builds were back to normal.

Accepted Answer

Thank you everyone for the feedback. The performance issue is mitigated and your build times should be back to normal. If you are still experiencing slower than expected builds please let us know.

thanks @Developer Tools Engineer for the support, I can confirm that our builds are back to normal

this was a pretty severe disruption for us, as we had to disable the CI checks on our PRs, due to excessive build duration, and excessive usage of Xcode Cloud time

we must be able to rely on Xcode Cloud as a stable tool in our workflows, but we understand that some issue can always happen: what we would have expected though is better communication (in fact, any communication at all)

in case things like these happen again in the future, I'd suggest to at least send an email to subscribers to communicate that you're experiencing issues with the system, so you're aware of it and you're working on a fix, and maybe have some suggestion for a temporary mitigation

as an act of good will, it would be nice if you could give a few hours back to subscribers, to compensate for the wasted time

thanks

To summarize, a recent performance issue with Xcode Cloud caused some projects to take an unusually long time to complete. We have corrected this issue, and added extra compute time to the affected accounts.

We understand the impact this had on developers and are reviewing our process so that we can respond faster in the future. The developer system status page is the fastest way to see details about known issues with Xcode Cloud.

Please continue to file feedback requests and post on these forums if you are having issues with our service.

Thank you all for your feedback.

Hi team, this issue seems to be back, we just noticed a massive spike in the build hours.

Can someone please help?

I'm attaching two builds of the same PR with not changes between them.

Xcode Cloud build phase has massively slowed down
 
 
Q