Support semantic search with Core Spotlight

Hi, I’m Jennifer, an engineer working on the Spotlight team. We care deeply about search, and today, I’m super excited to introduce all the new APIs from Spotlight that will let you build a powerful search experience in your own app. CoreSpotlight is a framework that allows your app to donate searchable content to Spotlight, and then retrieve that content with a query. It’s a great way to enhance your app’s persistent storage solution for search. Searchable content that your app donates is stored in a private, entirely local index, that never leaves the device. Users can search for your content in Spotlight, but no other apps will be able to see the data. CoreSpotlight makes it easy to provide search results and suggestions in your app from the natural language search terms that come directly from a search bar. This year, CoreSpotlight supports query understanding with semantic search. Up until now, Spotlight searched for content in your app, but search terms had to match exactly. With semantic search, people can search for content in your app in their own way, with search terms that can be similar in meaning. And with Spotlight’s query understanding models, you can be sure to find the right results no matter how you search for it. Today, we’ll walk through how to build a full search experience in your app, starting with providing content to the search index, and following best practices for data migration and recovery. We’ll look at how to retrieve results and suggestions, and then how to boost the ranking of search results that are most relevant to the user. We’ll be building an app that lets a person search for journal entries that they’ve written. You can find the full code for this sample app at the link below.

The first step to building a great search experience is to donate searchable content to Spotlight that represents what people are going to want to search in your app. In the journaling app, people can search over all journal entries, so each journal entry will be a searchable item. With that in mind, you’ll want to index items in a way that can they can be retrieved with a query to directly populate the views of your user interface. Currently, semantic search works best on text or media assets, such as images and videos. So to get the best results, you’ll want to ensure your searchable items have the appropriate content type. And, you’ll want to use system-defined attributes, whenever possible. Create a CSSearchableItem by providing a unique identifier, an optional domain identifier, and the attribute set. The unique identifier should be stored in your app’s persistent storage solution so that the full item data can be recovered.

Next, create a CSSearchableAttributeSet, and be sure to always set a valid content type Check out our documentation to see the full list of supported UT types, or learn how to create your own custom content type.

When donating searchable items with text, make sure to set the title and textContent. These attributes will be processed in the semantic index. If your item represents images or video assets, be sure to set the contentURL with the path to the asset. This ensures the asset can be processed in the semantic index from within your app’s sandboxed container.

And then, if your item references attachments or web content, consider donating these as separate items to the index, with their own content type and attributes. You can use the relatedUniqueIdentifier to maintain a relationship with the source item.

Once you’ve designed your searchable items, you’ll want to create an searchable index in Spotlight and donate them. We have some great new APIs to make donations more efficient with your app’s design. Including batch indexing with client state, and item updates.

To start, create a named CSSearchableIndex fetch and validate the last client state that you sent to Spotlight, and then, index your searchable items. Notice here that the indexing call is wrapped by calls to begin a batch and end a batch of items donated, where the next client state can be sent.

Client state is useful for both managing a large catalog of items, as well as maintaining data integrity between your app and Spotlight. Client state is also useful in preventing over-donation of items, which can have an impact on performance in your app. And when paired with the new isUpdate flag, your app can be sure to always donate only what is needed.

Migration and recovery are an important part of building a consistent search experience. Your spotlight index is completely local and private. That’s why you’ll need to take steps to ensure that your searchable content stays up-to-date in Spotlight. When Spotlight needs to migrate the index, or otherwise recover from data corruption, interruption, or other issues, it will make a request to your app to re-index all items, or a specific set of items. Your app can respond to these requests by both adopting the delegate protocol, for when your app is running live, and implementing the delegate extension, that Spotlight can call up separately from your app.

An index delegate extension allows Spotlight to schedule the requests during favorable device conditions, such as when the device is sleeping or idle, so that items can be migrated over time, with little change to your app’s existing search functionality. Setting up an index delegate extension is super simple with the new Xcode file targets. Let’s take a look at how to do this. Starting in the journaling app, which has already been configured to donate searchable content with our bundle ID: The first thing to do, is create a new target: Navigate to the File menu, and select New, Target.

Select the platform for the extension, here we're starting with macOS, and then search for the new CoreSpotlight Delegate extension template.

Click Next to configure this new target, and click Finish to add this new target to our project.

Once it's added, activate the new extension.

And we can now take a look at our stab implementation.

Now, since Spotlight will be the one to launch this process on our behalf, we have a handy command line utility to help us debug. So, let's go ahead and try that out.

First, set a breakpoint in the reindex all method, so we can try to catch this point in the code. Make sure to call the acknowledgment handler to avoid blocking the caller.

Next, we'll need to rebuild our app to include this new app extension, so select the app's target, then head to the Product menu, and select Build.

Then we'll switch back to our extension target, and from the Debug menu, select Attach to Process.

We're now ready to debug. At this point, we'll want to bring up the Terminal app to run the utility command.

The mdutil tool allows us to simulate these requests to our bundle ID for debugging. So we can run it.

And then hop back to our project in Xcode to see that we've hit the break point. We're now in a good spot to complete our implementation for reindexing all items, reindexing some items, adding support for drag and drop, and responding to throttling scenarios for critical donation paths. Now that you’ve indexed your searchable items, it’s time to support the search experience in your user interface. Queries can be configured to best support your user interface needs. Semantic search is enabled by default, but queries can also be configured to return results that are ranked using the same state of the art machine learning models that Spotlight uses, and you can also configure your query to support a suggestions menu in your app.

Use CSUserQueryContext to configure the query that is right for your user interface. Make sure to set the list of attributes to be fetched for each item returned from the query. These fetched attributes typically include only what is needed for display in your user interface. Ranked results can be enabled with a flag and can be restricted in number. These results are often shown in a separate Top Hits section. Results must be sorted in your app once all results are returned, which can be done using the new compareByRank comparator.

Suggestions can be configured or disabled by number as well. Suggestions are returned in ranked order, but the order can also be recovered when sorting. Structured queries, using the metadata syntax, are the best way to tailor the search results to your app’s user interface. As an example, if a person selects a tab to show only images, a filter query like this could be used to specify that the result set only contain images. Filter queries are also useful for content that you want to only show up in Spotlight, such as breadcrumbs that lead to specific parts of your app. Check out our documentation on using metadata syntax to build custom filter queries.

The last step is to create a CSUserQuery with the user’s query string from the search bar, and your query context. Results and suggestions are returned in async responses. Item results are returned in async batches so if ranking is enabled, be sure to sort the items before display. Suggestions are typically returned as completions of the user’s typed string. CSSuggestion provides an attributed string that can be displayed in a suggestions menu. If the user engages on a suggestion, replace the search bar text with this string, to trigger a new search.

Semantic search requires machine learning models that must be downloaded to the device, and will be run in your app’s process. These models may be loaded or unloaded at any time, to preserve your app’s memory space while running. That’s why you’ll want to call this class method just before your search interface appears, every time, to ensure that all resources are available as soon as a person starts searching. Now that search is working, you’ll want to think about improving the search experience over time, by donating signals that can improve the ranking of content that the user cares about most. Engagement and freshness are important signals in providing an adaptive ranking experience. The user might be browsing content related to a searchable item. Or they might initiate a search, then scroll through items in the result set, and finally, engage on a result to see its detail view. In each of these cases, your app can send an engagement signal to Spotlight to improve its ranking in future searches.

When the user is browsing content related to a CSSearchableItem, set the lastUsedDate property on the item attribute set, and donate it back to the index as an update.

And when a user engages with a result or engages with a suggestion from a query, you can mark the interaction associated with that query.

And that’s it! People can now enjoy a full-fledged search experience that seamlessly handles diverse search content on multiple platforms and locales. We saw how to donate content to an entirely private, on-device search index. we reviewed how to retrieve results and suggestions, and how to boost the most relevant search results over time. Adopting Core Spotlight is a great way to help people find content in your app. And now with semantic search it’s more powerful than ever.

For more information on other kinds of integrations with Spotlight, take a look at the session on leveraging App Intents, and how to get CoreSpotlight donations for free with CoreData. Thanks for watching, we'll see you in Spotlight!