Fix failures faster with Xcode test reports

♪ ♪ Sonu: Hi, my name is Sonu, and I am a human interface designer working on Xcode. And I am thrilled to give you a tour of the new Test Report in Xcode 15. The test report is where you go to view results for test runs that happen locally, in Xcode Cloud, or on another machine. test runs range from a single test you're running while working on a piece of code to an entire suite with thousands of tests running in CI. The test report organizes your test results in a way that helps you understand the health of your project, identify problem areas, and ultimately fix failures faster. Before we start exploring the new test report, I'd like to take a step back and define some key terms and concepts related to testing in Xcode. It's useful to understand the organizing concepts behind the structure of your tests. After I run through the basics, we will take a look at the test report and explore how it can help you. Let's begin with test methods. Test methods are the individual tests or methods which validate your source code and produce test results. Next, there are test classes. Test classes are groups of test methods and are usually grouped based on the area that's being tested. After that, there are test bundles. Test bundles are composed of one or more test classes. Each bundle houses a single type of test, either Unit or UI. Unit tests help verify a single piece of code, generally a function. Unit tests are short, simple, and run very quickly. UI tests observe the user-facing behavior of your app. These tests make sure your app truly does what you expect it to. For our purposes, let's say, our test bundle contains UI tests. And at the highest level, there are test plans. Test plans contain one or more test bundles, which means a test plan can contain both Unit and UI tests. And with Test plans, you can set up configurations to efficiently run your tests under several conditions. Configurations are an important aspect of test plans. Configurations tell Xcode how to set up the runtime environment for your tests. For example, you can test your app in varying Languages and Locations, so you can be sure your app works in every part of the world. You can test with code coverage, so you can keep track of the quality and coverage of your code as you continue to develop. You can even set up your tests to run many times. This ensures all elements of your app are reliably working, regardless if it's the first or hundredth time a user has used it. Alongside configurations, there are run destinations. Run destinations are the devices where your tests run. When running tests in Xcode's IDE, you can select a single run destination. With Xcode Cloud and xcodebuild command, your tests can have multiple run destinations. So now that we've covered the basics, let's see how tests, configurations, and run destinations work together. Let's say, we are using the same test plan as I spelled out before. Lately, I've been working on supporting many languages in my app. So to make sure my app is working as expected, I've created configurations for the languages I want to support, the first being English. And on top of that, let's say I'm running this test plan on three run destinations. The test plan runs on each device once with a configuration enabled. And each method will exit with a test result status, either passed, failed, skipped, or expected failure. For my app, I've also been working on supporting a few more languages, and I've made test plan configurations for each language. This means, Xcode runs the full test plan once for each configuration and run destination, resulting in a whole matrix filled with results. Zooming in, a result is produced for every test method, configuration, and run destination combination. This individual instance is called a test method run.

Test runs range from a single test you're running while working on a piece of code to an entire suite with thousands of tests running multiple configurations on multiple destinations, similar to my previous example. The new test report gives you tools to help you understand your test run, regardless of the number of tests. To do that, the new test report provides a high level summary of your test run, so you can see the big picture before digging into the details. It highlights important patterns, so you quickly know where to start investigating. It gives you a single place to see test activity, failure information, screenshots, and more. And lastly, we've improved our UI test debugging tools, giving you richer failure information. Okay, so we've covered a ton of concepts around testing and the benefits of the test report. Now let's see it in action. I've been working on a new feature for my iOS app, Backyard Birds. Similar to my previous example, I am doing work to make Backyard Birds available in many languages. As I've been adding support for more languages, I've been testing locally and in Xcode Cloud to make sure I'm not breaking anything. Let's take a look at some of my CI runs. I have a workflow set in Xcode Cloud to automatically run a handful of unit tests when I open a pull request. In the Build Overview, I can quickly understand what started my workflow, what code changes were built, and the workflow this build came from. I can also find out how my actions performed in this build. And oof, some of my tests have failed. Let's check them out.

When checking out the test action, I come across the new test report. Specifically, I find the test summary. The test summary gives me an overall understanding of what happened in this test run.

I can quickly understand my testing environment. I can also explore any notable patterns found in my test results using Insights. Insights are the patterns Xcode found while analyzing my results across all configurations and run destinations. It groups results based on certain criteria. Today, we have two types of Insights: Common Failure Patterns and Longest Test Runs. The Common Failure Pattern insight groups tests based on similar failure messages, while “Longest Test Runs” clues you in on which tests in your test bundles are taking longer than the others. Within the test section, I can understand how my tests performed during this run. I can also get more details about my test plan. I can understand what special traits my test plan has, like test repetitions or performance metrics. When testing with many run destinations and configurations, it can be hard to understand how each run performed. With this heat map, I can quickly digest how my tests did on each device and configuration. And the colors and test result counts help me understand how this run did when compared to the others. And if I have any test failures, I have quick access to them on the test summary. If a particular failure message catches my attention, I can use this section to start investigating. Speaking of attention-grabbing failures, when I was checking out the insights earlier, the first insight stuck out to me. It mentions I have three tests which failed with a similar failure message. Let's find out why these tests failed. After choosing a specific insight, I can use this view to learn more about it. I can find the failure message found in all of the tests, and I can see the tests that failed. Additionally, I can understand which configurations and run destinations produced these unit test failures. And with this information, I can get a signal as to why these tests failed with very similar failure messages. To keep investigating, let's check out one of the test method runs. After choosing a test method run, I find the test details view for testGardenSupplies(). The test details view creates a dedicated space to focus on the results of a single test method. This is where I can find test details and failure information, so I can start debugging. I can use the header to get full context of what happened across all configurations and run destinations. And in the runs view, I can explore how this method performed on each configuration and run destination. I can also find the failure message and call stack for the selected test method run. The failure message and call stack are valuable tools for debugging. They are the key pieces that help identify the source of the failure. The call stack also gives me entry points back into my source code, where I can continue investigating the failure. Alongside my unit tests, I've been running a slew of UI tests locally to make sure Backyard Birds meets customer expectations and performs as it should across different platforms and devices. Let's review a local test run from earlier. For this test run, I can still use the test summary to understand what happened. I have environment details at the top, Insights, and then the test section at the bottom. This first failure in the test section caught my attention. The failure message suggests that the test was unable to locate one of the buttons. Let's check it out. After choosing a failure, I find the test details view for testClickTabsAfterSearch(). Since I am looking at a UI test, navigating to the test details lands me on the Activities tab. The test report has a ton of awesome new features to help me understand and resolve my failures. The activities tab contains three major sections: Test activity, automation explorer, and the scrubber.

The test activity lays out my test in a timeline format, where the top-most row is the start of the test, and the bottom row is the end, and each row in between is an event which took place in the test. There is the automation explorer. This is where I can find moments of video playback related to the selected test activity. This allows me to see a full replay of my test. And lastly, there is the scrubber, which a linear representation of my test run. I can use the scrubber to locate test events, like taps, swipes, and clicks. The scrubber even highlights when the device-under-test changes orientation. And the failure icon above the scrubber notes where in the test the failure occurred. I can use these indicators to quickly find interesting moments in the test run and ensure interactions in my app are working as expected. With these new pieces, the test report has a test debugging experience that is interactive. Let's check out how it can help me solve this error. Clicking on an event in the activities pane updates the automation explorer with the corresponding frame from video playback, so I can visually understand what's happening at each moment of the test. It also moves the scrubber to the right spot, so I have context as to where events are taking place in relation to the full test run. When I'm debugging failures, I like to see the moments leading up to the failure.

I can watch my test run to see if anything unexpected happened that could have caused the error. After watching the video playback, everything seems good.

Based on this failure message, it seems the “Account” button wasn't found in the UI. The automation explorer shows me what was happening on the device when the failure occurred. The “Account” button is visible and has a bounding box over it. Clicking on the bounding box gives me more information about this specific UI element. Here is where I can find identifier and hierarchy information related to this element. Based on the information I have here, it seems like I labeled the “Account” button incorrectly within my app. From here, I can navigate to my project source and fix the error. The new Test Report made it very easy for me to understand my test run. Within a few moments, I was able to find which tests failed, find a solution, and fix the failure. Failures can happen locally and in CI, and it's best practice to test in both places. So I am happy to say the test report is available in Xcode & Xcode Cloud. So time is precious, y'all. The new features added to the test report will help you quickly understand your test run, identify problem areas, and ultimately, fix failures faster. Thank you so much for tuning in, and I hope you enjoy the rest of WWDC. ♪ ♪