Run, Break, Inspect: Explore effective debugging in LLDB

Hello, My name is Felipe, and I am an engineer on the debugging technologies team at Apple. In this talk, we are going to cover debugging techniques allowing you to explore code with ease and find bugs faster. Lldb is the underlying debugger that ships with Xcode, capable of pausing your program at any time, inspecting the state of variables, evaluating expressions, and much more. During this talk, we will cover the main tools provided by lldb, while also showing new features and advanced techniques that you may not be familiar with. We will start by defining a debugging model to guide us when using the debugger. Then, we will cover a different way of debugging using crashlogs. We will explore different methods of pausing program execution using breakpoints, and how certain coding patterns interact with the debugger. We will then look at the main tool to inspect program state: the `p` command. We will end by introducing a new feature from Swift 6 that let’s us customize how data types appear in the debugger. When debugging an issue with a program, we are typically provided with a point in time where the program is doing something wrong. This could be a crash, an incorrect value being displayed, or even the program hanging. Somewhere between the start of the program execution, and the point in which the incorrect behavior is observed, faulty code was executed. Our goal is find that piece of code. The bug can usually be found by inspecting the state of the program at different points in time, with each inspection bringing us closer to the problematic code. We are probably familiar with a few different techniques to inspect state of the program. For example, Some codebases make use of log statements. Reading an entry of the log file, in this case, is akin to inspecting the program at a point in time. If the log is detailed enough, that might be all we need to pinpoint the bug in the code This requires foresight by the programer to determine what is useful information to log, but can be a powerful technique to transmit app diagnostics between users and developers. A different technique frequently used is print debugging, which is probably the first debugging method we all learn. With print debugging, we insert print statements in the program, recompile the code, run the program, and reproduce the bug. Finally, we inspect the printed messages. If we need to print something new, we repeat the whole process. Eventually, we will have examined enough program state to fix the bug. But we’d better not forget to remove the print statements once we are done debugging, as we all have funny stories of print statements that made their way into production. This whole process can be very time consuming and error-prone. In this talk, we're going to show how you can navigate the search space faster by using the debugger. To do so, we're going to talk about the main tools provided by lldb: Backtraces, variable inspection, breakpoints, and expression evaluation. We're also going to show how lldb can help you investigate issues without even running a program. When working with a debugger, we are consistently repeating 3 different actions: Running the program, breaking at interesting points, and inspecting program state. After inspecting the program, we can proceed to a later point in program execution, or, if we need to inspect a previous point in time, we may relaunch the program. Repeating the run, break, inspect loop efficiently is key to an effective debugging session. Let's put this into practice! Most debugging sessions start by compiling the code and running the application under the debugger. Very often, all we need to do is hit the start button in Xcode or launch lldb through the command line using the target executable and its arguments. However, the first step in debugging an issue is being able to reproduce the bug; lldb can help with that too, using a technique that doesn’t even require starting the application.

In Apple platforms, whenever a program crashes, information about the state of the program at the time of the crash is collected and a crashlog is created. Lldb is able to consume crashlogs and present them in a form that resembles a debugging session, allowing developers to perform an initial investigation into how the crash happened. Sometimes, this might be all we need to figure out the source of a bug. I have a crashlog sent to me by a colleague who is testing Destination Video, a multi-platform video-playback app. Let’s open it with lldb! To open a crashlog, we can secondary click the file and open it with Xcode. Xcode then asks us if we would like to open the file in the context of a project. Let’s choose Destination Video.

And now Xcode uses lldb to create a debugging session with the state of the program at the time of the crash. The line of the crash is highlighted, which tells us that the code is failing to open a JSON file. Immediately before the crash, the program logged the filename that it was trying to open, so I might reach out to my colleague and ask for their log file. But how did the program reach this point? The debugger provides a tool to answer this question: the backtrace. The backtrace describes the sequence of function calls, or stack frames, that led to this program state. It provides a view into what each function was doing, where they were called, and also where each of them was going to return to.

We can find the current backtrace in the Debug Navigator of Xcode. So what was the program doing when it crashed? The current frame is for a JSON loading function. An earlier frame reveals that video metadata was being imported And this happened during program initialization.

Backtraces are a powerful tool to understand the control flow of the program, both with crashlogs and within a regular debugging session. When combined with crashlogs, backtraces can help us build intuition about how a crash happened. In our example, the crashlog also helped us identify information that might have been logged by the program, providing another avenue to investigate the problem. To get correct line number information with crashlogs, make sure the project is checked out on the same commit as the version of the app that created the crashlog, and that the dSYM bundle for that build is available. We cover dSYM bundles in greater detail in "Symbolication: beyond the basics".

Destination video is built with SwiftUI, which I am trying to learn more about, so I have prototyped a feature where users can add videos to a "Watch Later" list. My goal is to understand when some lines of code get executed, let's explore how lldb and breakpoints can help with that. From the app's main screen, if we select a video, we are presented with a DetailView of its contents.

In this view, we can find the "Add to Watch Later” button that I created, and its text changes when I click it.

To help me understand how it works, I will create a breakpoint where that button is created. This is the code that I prototyped, with the constructor call for the Button class; let's set a breakpoint there by clicking on the relevant line number.

Our new breakpoint is now displayed on the breakpoint navigator. But notice what happens as soon as we launch the application: Lldb has resolved the line breakpoint into 3 separate locations.

This is indication that we may stop at this breakpoint through different code paths. Let's test this hypothesis by navigate to the DetailView of a video again: The debugger stops the program. Xcode has highlighted the line in which the program is stopped, and it is about to call the constructor of Button. By inspecting the backtrace, we can confirm that we are in the middle of nested calls constructing UI elements.

For example, this frame creates a vertical stack of elements. Let’s see what the previous frame contains.

This one is creating a ScrollView.

We’ve stopped at one breakpoint, but lldb had identified a total of three locations associated with that line. We can get more information about them with the `breakpoint list` command. It describes the breakpoint we set on line 70, but it also describes the 3 locations associated with that breakpoint using line and column numbers. The first location on the list is where the program is currently stopped: the call to Button's constructor on line 70. We can observe this from the line and column information, but we can also use breakpoint identifiers, or IDs.

lldb assigns an ID to each breakpoint location, in this case 1.1. This is the same identifier used by Xcode when it highlights the breakpoint line. The second breakpoint on the list, with identifier 1.2, refers to first argument of the constructor, the action closure. This breakpoint should get triggered once we click on the button. The final breakpoint location, whose identifier is 1.3, refers to the trailing closure in the constructor call. This location is actually resolved to the next line, even though the body of the closure starts with the curly brackets in the line above. Let’s try reaching those breakpoints! From this initial constructor call, let's continue execution.

The program stops inside the trailing closure, on breakpoint 1.3. Using the backtrace, we find the constructor as an earlier frame: In other words, it was the constructor itself that called this closure.

To reach the final breakpoint, we have to click on “Add to watch later”.

We stopped once again, but this time we are inside the action closure! This example illustrates how even the most basic breakpoint can lead to breaking in interesting ways. We had three separate code regions attributed to the same line, but reached through different code paths: The call to the constructor, the trailing closure called by the constructor itself, and the action closure, which is called only when the button is clicked. This is a common scenario in declarative code making heavy use of closures, like SwiftUI. We don’t always know when a closure may get called, so a line breakpoint inside the closure's body is a good way to pause when it does. Pausing the application is an important part of the debugging cycle, but we can improve the debugging experience by combining it with program inspection. As an example, let’s try to learn how UI elements interact with the program, and focus on that first breakpoint, the one calling the Button constructor. To break only when the button is created, let's disable the last two breakpoint locations.

And now let's click the button, which should trigger a UI update and our breakpoint.

Using lldb, we can inspect the size of the Watch Later list with the `p` command, which we will explore in greater detail later.

We have one video in the list, the one we just added. We can even check its title.

In our Break/Inspect debugging cycle, it can be tedious to repeat the same commands over and over again. Using the concept of breakpoint actions, the debugger can help us by running commands when breakpoints are reached. Let’s change our example to print entries of the Watch Later list whenever we hit the breakpoint! We can find the Edit Breakpoint menu by secondary clicking a breakpoint: Let's add a Debugger Command action printing the name of the most recent video in the list, if it exists: We can even continue execution after hitting the breakpoint.

We now get information about the queue every time the constructor is called! This is a way of leveraging the debugger to print information without recompiling code. So far, we've been using Xcode’s graphical user interface to interact with the debugger, but lldb provides a rich command line interface that can be used instead. Let's repeat our steps from the previous example, but using the command line this time. To get access to the Debugger Command line, we first pause the application: Now we can set a breakpoint with the `b` command, which is a shorter alternative to the more general `breakpoint set` command.

The command line can also be used to add breakpoint actions, but note that this will overwrite any actions set through Xcode. Let’s use "break command add" to print the name of the last video added to the Watch Later list: Like before, this continues execution after printing. This command affects the most recent breakpoint, but it can modify a different breakpoint if provided with the optional breakpoint identifier argument.

Lldb provides detailed description of all of its commands by using the help command. You can also get help for any option of a specific command. To discover more lldb features, a great tool is the apropros command, which searches lldb's help text for a keyword and returns any commands or options that are described by that keyword. For example, a search for commands related to backtraces finds frame select and its alias, the f command. It also finds the thread backtrace command. When debugging, we often create a breakpoint that is triggered many times, but we are only interested in a subset of those. A common example is when a breakpoint is placed inside a loop and, instead of breaking on every iteration, we only want to break when certain events happen. Let's go over the three main techniques that can be used to handle high-firing breakpoints. Consider this code snippet iterating over videos in a collection, loading videos if they are in a remote location, and processing them. We may be interested in stopping at the loadRemoteMedia function only when the video is very long. We can accomplish this by setting a line breakpoint and modifying it with a breakpoint condition, which defines a rule for whether the debugger should stop the program or not. On the command line, we would use the break modify command, providing it with a breakpoint ID and a condition. Any code valid on the breakpoint location can be used as the condition expression. In our example, we could modify the breakpoint to only stop the program if the current video is longer than 60 seconds. In Xcode, we would secondary click on the breakpoint, navigate to Edit Breakpoint, and populate the Condition field.

Going back to our example, we may be interested in stopping at the processVideo call only if we also executed the loadRemoteMedia function. In this case, we could once again set a line breakpoint, but this time add a breakpoint action. We've used breakpoint actions before to print variables, but they can also be used to create new breakpoints. With the `tbreak` command, we can create a temporary breakpoint, which causes the program to stop only once at that location. In our example, we can set an auto-continue breakpoint on loadRemoteMedia, with an action creating a temporary breakpoint on processVideo.

The third technique consists of ignoring breakpoints for a fixed number of times, and stopping on subsequent hits. For example, we can ignore the first ten videos in our collection. To do so, we would modify the breakpoint, as we did before, but now we use the --ignore-count flag. Using Xcode, the same option is available on the Edit Breakpoint interface. In extreme cases, where a line of code is executed millions of times, the previous techniques can noticeably slow down program execution, as the debugger still needs to stop every time and decide whether to continue or not. This is a situation in which recompiling code is advisable. For example, we can compute our stop condition, and set a breakpoint inside an if statement that executes only when the condition is true. A nice trick is to use the raise function with the SIGSTOP signal: this instructs the application to stop and, if you’re running it through Xcode or lldb, the debugger will take over as if a breakpoint had been reached. We've focused on two components of the debugging cycle so far: starting the program and breaking at interesting locations. But we only glanced at the main tool for inspecting program state: the `p` command. In a previous example, we used the `p` command as our primary way of looking at variables and evaluating expressions lldb provides many other commands to accomplish this, and they all have their uses. It can be overwhelming to understand all of these tools, but since Xcode 15, `p` is the right command for most situations in which you need to inspect a variable or evaluate an expression. It has been reworked as an alias to the "do what I mean" print command, which allows you to save time by combining many different tools under a single command. We cover this in detail in "Debug with structured logging". Let's give it a try! I am now trying to add a new video into the app, and to do so I edited the JSON description of the videos. However, as soon as I launch the app, it crashes.

This is the JSON file that I edited with my new video. Let’s launch the app so that the debugger can stop at the moment of the crash.

By inspecting the console, we notice that the developers of DestinationVideo are using logging in their workflow.

The last piece of information logged was that the Videos JSON file was being loaded.

Something went wrong between the start of the program and the exception being reported. At this point, a good first choice is to set a breakpoint right before calling the JSON loading function to see if that brings us any closer to the bug. Let's give it a try.

By secondary clicking the log message, we can quickly navigate to the corresponding source code location.

Let’s set a breakpoint at the end of this function.

Since we need to break in a previous point in time, we will relaunch the program. Because no code was modified, we can relaunch without recompiling using control + click to save time.

Let’s look at the URL and filename local variables: Both of these look ok. We can also visualize them on the variable viewer.

Or simply hover over them on the source code.

Some types even have a quicklook button, providing additional details for the variable.

This looks like the file that I edited.

We have inspected the JSON loading function and it looks correct, so the bug is probably somewhere after that. Let’s look at another part of our program: the Video constructor that takes a JSON decoder.

I suspect one of those try statements are problematic, but if we break on every call to the Video constructor, we will have an issue: There are too many Videos in the application! Thankfully, we covered techniques that can deal with high-firing breakpoints like this one. I know that my new video is the 12th video in the app, so I could apply an ignore count to this breakpoint. Instead, let's try something new: a Swift Error breakpoint. This type of breakpoint instructs lldb to stop the application as soon as a Swift Error is thrown.

Let's click continue! The program stopped in our Error breakpoint while trying to decode the imageName JSON key. Something is wrong there. Let’s use the backtrace to go up to an earlier frame, where the input data is.

As a programmer, I like to code my way out of this type situation, and `p` gives me a lot of freedom. Let's do some programming to figure out how many imageNames exist inside this `data` array.

Oops, we need to create a string instead.

This is too much output. Let's search for imageName.

Closer, but still too much output. Let's check the count property.

Ah, I expected to have 13 videos, but I only have 12 keys called imageName. Something is wrong. Let’s look at our JSON file.

Ah, I made a typo when writing imageName! Let me fix this.

There we go! With this example, we’ve explored how the `p` command is able to inspect variables and to evaluate complex expressions; it can do so in any frame of the backtrace. We were able to gradually build a complex expression, printing intermediate results every step of the way without having to recompile any code. This debugging session also gave us insight into how we can enhance the logs of our application, by including which key was missing in the JSON file. Most variables we’ve inspected so far were fairly simple. However, types that contain too much data are cumbersome to inspect during a debugging session, and they may benefit from having a short description in the variable viewer or the `p` command. These types usually have many properties or are frequently stored inside collections. For example, a collection of WatchLaterItem in the variable viewer does not reveal any information about each entry, unless we manually expand them. Lldb has always provided a mechanism to customize the output of p and the variable viewer Swift 6 introduces a mechanism to do this directly from source code using the new @DebugDescription macro. Besides annotating the type with this macro, we must also create a DebugDescription string property summarizing the type. It must be created using string interpolation, and stored properties. Let's implement this for our WatchLaterItem type! Our data structure contains three relevant data members: a video, a name and the date it was added on the list. We start by marking the type with the DebugDescription macro. Then, we create a debugDescription string computed property. And, in this case, we will use the name and addedOn date as a summary for the type. Now, if we inspect the variable viewer again, the summary is displayed in each item of the collection! This example may have looked familiar if you have used the CustomDebugStringConvertible protocol before. If you were using it, you were probably printing types with the `po` command. In those cases, check the implementation of the protocol; if it is only using string interpolation and computed properties, you can adopt the macro instead of the protocol. Those types will integrate much better with the debugger and you can focus on a single command for debugging: `p`.

We have gone through the debugging cycle multiple times now, and we have explored the main concepts required to effectively use the debugger. We've covered how debugging can be treated as a search problem. With conditional breakpoints and variable inspection, Lldb is a powerful tool to perform this search effectively. It is also a great tool to help us understand codebases we are not familiar with. With breakpoint actions and expression evaluation, we can leverage our coding skills to execute code while debugging. And we don’t need to recompile our projects to do that! Hopefully you'll find bugs much faster by employing the ideas we've covered today. Once you identify the root cause of bugs, remember to add test coverage for that scenario, which was likely missing before. Thank you for watching!