Inspect, modify, and construct PencilKit drawings

Hello and welcome to WWDC.

Hi, I'm Will Thimbleby. We're going to take a look inside PencilKit drawings, at what they're made out of and what you can do with them. PencilKit is super easy to adopt, provides beautiful, realistic-looking inks, the best low-latency drawing experience and some great new UI improvements in iOS 14. For more information about these improvements, see the "What's New in PencilKit" talk. In iOS 14, we're also letting you look inside PencilKit's data model: the drawings, the strokes, inks, paths and points. This will enable you to build some great new features in your apps with PencilKit. With access to the data model, you'll be able to inspect the contents of what your users drew, react to what was drawn, manipulate existing drawings or dynamically create new drawings from scratch. To give an example of what's newly possible with PencilKit, let me jump straight into a demo.

This demo is also available as sample code. I've been working on an app to help my seven-year-old son practice his handwriting. At the top of this app is a text field with the words I want him to practice. Below is a synthesized PencilKit drawing of the same text. I could change this text with the keyboard, but this is a Pencil demo. I can use the great new Scribble feature, which, by the way, also uses PencilKit, to change this text to what I want him to practice, perhaps to something more appropriate for a seven-year-old.

As I enter text, the template PencilKit drawing below is constructed from individual letters.

In the top right, I can also choose the size I want him to practice and the difficulty.

Now that I have a template, we're set up to practice handwriting. The next stroke to copy is animated by this red dot that shows how to write the next letter. All I have to do is copy it.

If I write something that's close to the template, my handwriting changes to green and we move on. If I write something badly, the stroke disappears and the animation repeats.

My score is shown in the top right. Let's see how I do.

We did well, so we get some fireworks, a key motivational tool for seven-year-olds. But I've had a few years to practice my handwriting, so why don't we try something a bit trickier? I drew a PencilKit ligature, and I've added that to the app.

Now we can practice our calligraphy.

The app is synthesizing PencilKit drawings from text, animating them and performing recognition on what I wrote. This is just a small example of what you can now do with access to the PencilKit data model. So let's look inside a PencilKit drawing. Here we have a simple drawing of a flower. If we split it up, we can see that the drawing is composed out of many PencilKit strokes. Each stroke represents an individual line that the user drew. These strokes are in the order that the user drew them, so you can see that the outline of the flower was drawn first, then the stalk, leaf, and finally the whole thing was colored in with a marker.

Here we have the data asset that I drew that contains the lowercase letters for the demo I just gave. To be able to generate text, we first want to split this drawing up into individual letters.

Later, these letters are combined to generate the template text that users use to practice. To split this drawing up, we take the lowercase alphabet drawing and get its strokes. We slice this array to get the strokes for each individual letter.

Then we can create new drawings for each letter out of that slice. This is repeated for each letter in the alphabet.

If you want to inspect or modify a drawing, you do this by accessing the drawing's array of strokes. You can also use strokes to create new drawings from scratch.

So, what about strokes? What makes up a PencilKit stroke? For a stroke, the primary feature is the path. This provides the shape of the stroke. You also have an ink, which describes the appearance of the stroke: its color and type. A transform gives the orientation and position of the stroke. Strokes can also have masks, and we'll discuss those later in this talk.

Another useful property of strokes is the renderBounds, and this is a bounding box that encompasses the entirety of the stroke when it is rendered. The renderBounds accounts for the effect of all the stroke properties including the path, ink, transform and mask.

Inks, which describe what a stroke looks like, contain the type of ink and a color. Inks do not have a width. The width of a stroke is variable along the stroke path. The stroke path describes the shape of the stroke and the appearance of that shape as it changes along the path. For example, the stroke path gives you the width of the stroke at any point. A PencilKit stroke path is a uniform cubic B-spline of PencilKit stroke points. Now, that's quite a mouthful. What does that mean? It means that the contents of a path are, in fact, the control points for the B-spline. So if we iterate over the points in a path...

and draw each one in turn...

the resulting points are not actually on the stroke. These points are the B-spline control points and probably not what you want to draw. To get points on the actual path, we need to interpolate the spline.

To interpolate the spline, we access the points using interpolatedPoints strideBy. This provides a sequence of points that we can iterate over like before. Drawing these gives us a series of points on the path. There are a couple of things to notice here. They're on the path, there are more of them and they have uniform spacing, in this case, a distance of 50 points, which is the stride argument passed in to the method.

You might also notice that the spacing of the last point is uneven. This is because the last point on a stroke is always generated regardless of the stride. You can stride by distance, as in this example, time or parametric value. Distance and time are self explanatory. Distance is points in drawing coordinate space, and time is duration in seconds, which depends on how fast the user drew. Parametric value relates to the parametric interpolation of the B-spline. To explain what the parametric value is, let's bring back drawing the control points. This is the same code that we used earlier. If instead of drawing the control points, we iterate over the indices of the path, which go from zero to the control point count, and at each iteration we get the point using interpolatedPoint(at, for the parametric value 0, 1, 2, 3 and so on and draw that...

we get the equivalent points to the control points. But these points are actually on the path. Why is this useful? Let's number the points so you can see what's going on. The parametric value is useful because it is a floating point value. That means you can ask for the interpolated point for any value, including non-integer values between control points like 2.4 or 4.8... and so on. This gives you the flexibility to interpolate the stroke path any way you want. All the interpolation we've seen so far has been interpolating the path with a uniform step. Using the parametric value, PencilKit also provides the ability to step along a path by an arbitrary distance, using parametricValue, offsetBy.

This function allows you to offset a parametric value on a path forwards or backwards any step in time or distance.

One of the places where non-uniform stepping is useful is when animating. The demo I gave earlier uses this ability to animate the red marker dot along the strokes. Each frame, the current marker position on a stroke, is offset by the exact time duration since the last frame.

Non-uniform stepping is necessary because we're not always guaranteed a uniform amount of time between animation frames.

To animate in the demo, first we get the delta time, the time that has elapsed between the current frame and the previous frame.

We use that to offset the current animation parametric value along the path by the same amount of time. This animates along the stroke path with the same velocity as when the user drew it.

Finally, we update the marker position, getting the new location on the path from the new parametric value.

So that's the path. Both the control points and the interpolated points along the path are PencilKit stroke points. These are the atomic building blocks of paths and strokes. They capture both the appearance and touch information of a stroke at a particular location. These points are stored in a lossily compressed format, so any points you create will not capture the values you use with perfect precision. Let's take a closer look at one of these points in a stroke.

A PencilKit stroke point has several appearance attributes.

The first of those is the location of the point.

A point also has a size, which for marker strokes won't be square. A rotation angle, or azimuth.

And finally, the opacity. These attributes combine to describe how a stroke appears at a certain location.

Stroke points also have a couple of properties that are not appearance attributes. Force and altitude match the same values from UITouch when the stroke was drawn. Time offset is the offset in seconds from the creation date of the stroke path that the point belongs to. This provides timing information for how the user drew the stroke.

We'll jump back now to talk about the last property of PencilKit strokes that we haven't covered... and that is masking.

Masked strokes are typically created when the pixel eraser is used to erase only a portion of a stroke. Most strokes are not usually erased, but when they are, the mask is used to clip these strokes in rendering and adjust how the user can interact with them on the canvas. Masks can have holes.

Or they can cut a stroke into multiple pieces. In this example, using the eraser, the stroke has been split into two separate strokes.

These become unique, independent strokes and behave as such to the user and in the API. For example, each of these two new strokes has its own separate transform and mask. The user can select one of the strokes and move it around without affecting the other stroke.

PencilKit strokes are masked, but stroke paths are not. This means that if we take the code we were using earlier to draw a stroke path and use that code to draw a masked stroke...

we're going to get a much longer path than we wanted.

Instead, we want to use the maskedPathRanges property of the stroke. This is an array of parametric value ranges on the stroke path when it is clipped to the mask. Here we iterate over the maskedPathRanges...

and interpolate the points in each of those ranges.

This correctly gives an interpretation of the stroke path in a way that makes sense for a masked stroke.

Strokes can have zero masked ranges. For example, if the user erases all but a fraction of a stroke, and that fraction does not intersect the path spline, then the resulting masked stroke will have zero maskedPathRanges.

Strokes can also have multiple masked ranges. In this case, a stroke with holes in it has four individual ranges.

Recognition is a building block for many great features that you can build with PencilKit. Spline-based recognition can make use of these maskedPathRanges to provide a sensible interpretation of masked strokes, and this is what we do for handwriting recognition in Notes.

When interpreting strokes, you can use maskedPathRanges to get a range of points, interpolate them how you want and use non-appearance attributes like time and force to supplement the shape of the path.

The demo you saw earlier provides a simple example of spline-based recognition. It uses a matching algorithm to compare and score the user on the similarity between what they drew and the template they're trying to copy. If you want to do image-based recognition, use the rendering API on PKDrawing to generate images. PencilKit provides a super easy way to add great Pencil support to your app. Now that you can look inside drawings and access the strokes, inks, paths and points, inspect what the user wrote and drew to build features like the new Scribble experience which uses PencilKit to enable handwriting in text fields across the whole of iPadOS. Modify drawings to create interactive drawing experiences that respond to the user's actions. And create new drawings procedurally, like the sample code does to generate handwriting templates for practicing.

Adding support for Pencil, one of our most expressive input devices, is a great addition to almost any app. PencilKit has always been a fantastic way to add drawing to your app, and now that you can look inside drawings, it's also an incredibly powerful foundation for new Pencil-focused experiences that you want to build.