Squeeze the most out of Apple Pencil

Hi, I’m Sam, a software engineer working on drawing features. In this video, I’ll go over advancements to the PencilKit tool picker and Apple Pencil APIs to help you create expressive drawing experiences in your apps. PencilKit is the easiest way for you to get started building a rich drawing experience. And the PencilKit tool picker gives you integration with all the latest platform features. It works great on iPad, providing a responsive picker that can be adjusted to fit various drawing and writing workflows. On iPhone, it docks to provide all the tools and controls in the available space.

And on Apple Vision Pro, it presents in an ornament floating just outside the window. And now with Apple Pencil Pro, the PencilKit tool picker is available wherever your hand is, with just a squeeze. First, I’ll talk about new features in the tool picker that allow configuring it with the right set of tools for your app. Then, I’ll show how you can build your own custom tools right into the tool picker. And finally, I’ll highlight some new APIs to integrate your drawing experience with Apple Pencil Pro. I’ll start with a demo of the new features in the tool picker. I’ve started a drawing of my cats in an app I’ve been building that lets me stamp animal shapes alongside strokes drawn with PencilKit. The source code for this project is available in the resources for this video. In my app, I have tools like pen and eraser, but also my animal tool. I’ll tap on the animal tool to bring up the controls for color, size, and a custom shape picker. When I change the different properties of this tool, the item in the picker updates to represent what I have selected.

And it even has deep support for features like squeeze on Apple Pencil Pro, with no additional code. When I squeeze, my custom tool is displayed in the arc alongside the PencilKit tools. I’ll tweak the properties of my stamp to make perfect paw prints for each of my cats.

My stamp tool has support for barrel roll angle on Apple Pencil Pro; so I can place the paw prints with precise rotation.

Nice! Now I can tap this button to sign my drawing, right in the tool picker! Since iOS 13, PKToolPicker has provided easy initialization using the full set of PencilKit tools.

Check out the “Introducing PencilKit” video to learn more about the basics of adding PKToolPicker to your app.

New in iPadOS 18 and visionOS 2, the tool picker now lets you define the available tools.

Use the tool picker with PKCanvasView, your own drawing canvas, or a combination of both! When using the inking tool to provide tools that draw strokes on PKCanvasView, you can even include multiple tools of the same type! For example, this is great for an app offering an annotation experience to present two colors of pen, or two colors of marker.

There are many types of PKToolPickerItems to provide whatever you need for your picker. Use the new toolItems initializer to customize the order and tool set to fit your app! Inking, eraser, and lasso tool items have a corresponding PKTool.

These are the tools that PencilKit uses to draw something on its canvas. These will be set to the tool property of PKCanvasView automatically if the canvas view is set as an observer of the tool picker.

The ruler is different from other items. When tapped, it toggles on or off rather than becoming selected. The ruler item does not have a matching PKTool, as it does not draw anything on a canvas. Observing PKCanvasViews will automatically toggle their isRulerActive property when this item is tapped.

Or, use the tool picker observer to toggle the ruler for your custom canvas.

The Scribble feature is great for turning handwriting from Apple Pencil into text, automatically.

Add the Scribble item to the picker to unify your Scribble and drawing features into a single place to switch pencil modes. When selected, the Scribble tool replaces the color wells with the controls from the keyboard’s Shortcuts Bar. The Scribble tool will hide automatically, depending on the preference from Apple Pencil Settings for enabling Scribble. Check out the "Meet Scribble for iPad" video for more information about setting up Scribble in your apps. The custom item opens up PKToolPicker to any non-PencilKit tool you can imagine! For example, here’s the stamp tool I showed earlier, but this tool could be anything your app provides! I’ll go into detail on the custom item in a bit. Also new in iPadOS 18 and visionOS 2, PKToolPicker provides the ability to set an accessory UIBarButtonItem on its trailing edge. Use this button to display canvas actions that aren’t drawing tools, like inserting a text box or setting the background. The actions you provide here should also be available elsewhere in your app’s UI, since this button will be hidden when the picker is minimized.

Next, I’ll talk about how you can build tools from your drawing canvas into the tool picker.

PKToolPicker has always been the best way to pick tools for use with PKCanvasView. Now, with custom tools, the tool picker opens up for use with your drawing tools.

PKToolPicker can be used with PKCanvasView, your own drawing canvas, or a combination of both.

Custom tools are a great way to integrate system pencil tools with actions that make your canvas unique, like the animal stamp example, or photo retouching tools controlled by drawing.

Drawing is turned off on any observing PKCanvasView when a custom tool item is selected. With custom tools, your app does the rendering, and PencilKit does the tool picking.

PKToolPickerCustomItem and its Configuration let you build your own items that fit with your tool’s features. Use the Configuration struct to define the properties your tool supports.

PencilKit provides controls for color and width properties that can be configured to fit your tool’s behaviors. For additional properties, pass a view controller to display more controls. PencilKit displays your custom tool item in the picker using a pre-rendered image you provide in a closure.

PencilKit will call this closure for a new image whenever width, color, or opacity change. There is also a reloadImage method you should call when a custom attribute changes. Consider how you can represent your tool’s properties in your tool image.

UILabel.drawText(in:) is a great way to draw text indicators on your tool, like to represent an opacity value.

Finally, I’ll show how you can unlock the powerful features of Apple Pencil Pro in your apps. Apple Pencil Pro offers a huge set of new features to push your app’s drawing experience further! I’ve been planning a salsa garden here in Notes, and I want to make a few quick changes. To pick the perfect tool for underlining the cilantro label, I can just squeeze my Apple Pencil Pro to show the tool picker, right where I’m hovering. I’ll change the pen tool’s color. I’ll draw my underlines.

Hrmm, I’ll undo that and go for a highlighted look instead. I’ll squeeze my Apple Pencil Pro, and long-press on the undo button to bring up a slider. Now I can just slide to undo multiple actions.

I’ll squeeze again and switch to the marker tool. Apple Pencil makes it easy to create expressive strokes with different motions, like changing the altitude, azimuth, as well as applying different amounts of pressure.

Now, with Apple Pencil Pro, rolling the pencil’s barrel takes creations further.

If your app uses PKCanvasView, you’ll get this with the marker and fountain pen. When using Apple Pencil Pro, rolling the pencil’s barrel controls the angle of the stroke! And with all PencilKit tools, hovering with Apple Pencil Pro shows a beautiful shadow below the pencil representing the specific tool.

I’ll use the marker tool to highlight the cilantro label.

And I’ll pause at the end of my stroke to trigger shape recognition. When triggered, I get haptic feedback in my hand that the recognition was successful. Now the snap feels alive! Notes uses PencilKit to provide this deep integration, but there’s also a rich set of APIs to integrate with your own custom drawing canvas.

First, I’ll go over the new undo slider.

New on Apple Pencil Pro, the squeeze gesture can be used to access a slider for quickly working through undo/redo history. This is available in all apps; even if you don’t use PKToolPicker. If you want to suppress this behavior, install a UIPencilInteraction with a delegate that implements the method to respond to squeeze actions. For apps with drawing features, squeeze opens up new possibilities for interaction with your app without releasing the pencil.

Treat squeeze as a single gesture that performs a discrete action. This will make your app more comfortable to use. There is a device global preference for what apps should do when Apple Pencil Pro is squeezed. This includes the new options of showing a contextual tool palette or running a system shortcut.

Note that if the preference is set to run a shortcut, your app will not be sent the squeeze event.

UIPencilInteraction now also sends pencil hover pose information when sending your app a double tap or a squeeze event on supported devices. The pose describes where the pencil is positioned above the display.

This includes location above the screen, z-offset from the screen, as well as azimuth, altitude, and roll angles.

PencilKit uses this pose information to position PKToolPicker when it presents from a squeeze. You can use this information as well in your own tool picker. I have some code that’s doing that here. I’m implementing the delegate method to receive squeeze events, I check if the user wants to show a palette on squeeze, if the squeeze has ended, then send the location of the hover through to my palette presentation, if available. In addition to the updates to UIPencilInteraction, there’s also all new SwiftUI API for interacting with Apple Pencil. Here’s the same example in SwiftUI using the onPencilSqueeze modifier. I check the preference, the phase, and I get the hover position if available. Adopt these APIs in your tool picker to be consistent with what is selected in Settings. Or use PKToolPicker to get all these behaviors with no additional code.

Apple Pencil Pro provides haptic feedback using existing SwiftUI and UIKit feedback APIs.

These APIs are already used in places like providing feedback when a control changes state.

The feedback APIs are tuned to give different types of feedback in different contexts, depending on things like the characteristics of the platform and settings.

For example, feedback may be provided as haptics, audio, both, or neither.

New feedback APIs for alignment snapping and completing a path are available in both SwiftUI and UIKit.

The APIs now use location to have more context to decide the best feedback for a given event. Here is an example using the new UICanvasFeedbackGenerator class. When my custom canvas snaps to a layout guide, I tell the feedback generator that alignment occurred. And when my custom shape recognition logic is triggered to snap a drawn stroke to a clean shape, I tell the feedback generator that a path completed. UIFeedbackGenerator and all its subclasses have been updated to take a view on initialization and a point in that view when generating feedback. For the view, pick something close to the feedback you’ll be generating, like your canvas view. Be sure to update all of your uses of UIFeedbackGenerator. This will help UIKit provide the best feedback for your apps. In addition to UICanvasFeedbackGenerator, SwiftUI also offers alignment and pathComplete in the SensoryFeedback struct. Here, I have two State integers that are incremented elsewhere when aligning to a guide and when a drawn stroke snaps to a shape. I’ve added sensoryFeedback modifiers to the most specific view on my canvas for these operations. This will request feedback whenever the trigger value changes. Adopt either SensoryFeedback from SwiftUI or UICanvasFeedbackGenerator from UIKit to make your custom canvas come to life with Apple Pencil Pro. Check out the "Playing haptic feedback in your app" article in the Apple Developer documentation for more details. With barrel roll angle, strokes become more expressive.

Use barrel roll angle for providing input to drawn strokes, rather than controlling user interface elements. Make your custom strokes more responsive by using roll position.

Strokes mimicking real tools with angled tips that vary when rolled in the hand are ideal candidates.

Add the roll and azimuth angles together to respond better to different motions, and to provide a fallback on devices not supporting roll.

rollAngle is available on both UITouch, and UIHoverGestureRecognizer. This empowers you to elevate hover experiences on your custom tools, like I showed with the animal stamp preview earlier.

When receiving finger touches, direction and velocity are good fallback options for varying your stroke. Roll is estimated when first sent and later updated over Bluetooth. Be sure to implement touchesEstimatedPropertiesUpdated to get the most accurate final values for your strokes. And check out the updated Human Interface Guidelines for best practices on implementing all of these Apple Pencil features. And that’s what’s new with Pencil! So what’s next? Adopt PKToolPicker for a tool picking experience with deep system integration and less code.

Add your custom tools to PKToolPicker to offer them alongside the PencilKit tools.

And enhance your custom drawing experiences with the rich set of APIs available for integrating with Apple Pencil Pro. That’s everything you need to get rolling, and squeeze the most out of Apple Pencil!! Thanks for watching!