We're using App Intents to launch are control our app via Siri. Siri's responses have been fairly random, some with a "Done" popup, others with a verbal confirmation, others saying "I'm sorry, there's been a problem". The latter is bogus and doesn't look good to potential investors when the app is actually working fine.
There appears to be no way in code that I've been able to find so far that would have been tell Siri to STFU. Let us handle our own errors.
Otherwise is there a means to supply Siri with a dictionary of restored messages that could be triggered inside the app?
Explore the power of machine learning and Apple Intelligence within apps. Discuss integrating features, share best practices, and explore the possibilities for your app here.
Post
Replies
Boosts
Views
Activity
Hi,
I have a AppEnum and I try to use a custom SF Symbol as DisplayRepresentation.Image but it's not working. I get a blank image when AppEnum picker appears.
The custom SF Symbol is stored in the target's asset catalog and it works in the target (eg: Image(named: "custom_sfsymbol"). The issue occurs when I try to use it in a DisplayRepresentation
static var caseDisplayRepresentations: [Self: DisplayRepresentation] = [
.sample : DisplayRepresentation(title: "sample_title", image: DisplayRepresentation.Image(named: "custom_sfsymbol")),
Can we use a custom SF Symbol in a DisplayRepresentation.Image?
I have followed the SoupChef example in migrating Custom Intents from SiriKit to AppIntents. However, we only require one iOS release back, so we can require iOS 17. Thus, I eliminated everything that was strictly for backwards compatibility, most notably the SiriKit Extension that required enormous amounts of code to try to coordinate with the real app which worked poorly anyway.
I tested for example that an NFC tag Automation created in Shortcuts works to execute an AppIntent while the app is backgrounded.
I am now receiving a beta report that indicates someone trying to execute one of our migrated AppIntents from their HomePod is not working, and they say it used to work sometimes (not all the time). I'm sure most such cases used to require the SiriKit Extension in the old SiriKit world. I am terrified that I may need to rebuild that monster once again when the new (to me) AppIntent API seemed so beautiful without it and seemed to work without it. The AppIntent API documentation seems to indicate that SiriKit Extensions are no longer related or required. What is the truth here? Do I need to re-implement everything twice in the SiriKit Extension like a barbarian, or can we live in the new world with AppIntents?
Thank you.
I am very new to App Intents and I am trying to add them to my On Device LLM ChatBot app so my users can get answers to any questions anywhere in iOS.
I have the following code and it is working wonderfully in the Shortcuts app.
import AppIntents
struct AskAi: AppIntent {
static var openAppWhenRun: Bool = false
static let title: LocalizedStringResource = "Ask Ai About"
static let description = "Gets an answer from Ai for your question."
@Parameter(title: "Question")
var question: String
static var parameterSummary: some ParameterSummary {
Summary("Ask Ai About \(\.$question)")
}
@MainActor
func perform() async throws -> some IntentResult & ReturnsValue<String> {
let bot: Bot = Bot()
await bot.respond(to: self.question)
return .result(
value: bot.output
)
}
}
class AppShortcuts: AppShortcutsProvider {
static var appShortcuts: [AppShortcut] {
AppShortcut(
intent: AskAi(),
phrases: [
"Ask \(.applicationName) \(\.$question)",
"Get \(.applicationName) answer for \(\.$question)",
"Open \(\.$question) using \(.applicationName) ",
"Using \(.applicationName) get help with \(\.$question)"
],
shortTitle: "Ask Ai",
systemImageName: "sparkles"
)
}
}
I can create a shortcut for this AppIntent and that allows me say speak the response.
I can call my shortcut via iOS 18 Beta 1 by the Shortcut name I set in the Shortcuts app and that allows it to work.
It does not work at all by just Asking Siri any of the phrases I have defined.
The info.plist has an app name alias defined just to be sure.
I even added the Siri capability in Xcode-beta.
I also tried using the ProvidesDialog return type too.
Whatever I do the AppIntent is invisible to Siri.
Siri tries to search the web, looking for my app name in the contacts or have an error Apple Cash which has nothing to do with what I was talking about.
Is there anything else I am missing for setting up iOS AppIntents to work with Siri?
I try to use Create ML Spatial template. but unexpected error is occured in 1-3 minitues. I try some times and same results. Spatial template is not available on an M1 mac ?
My development environment is
Apple M1 Pro
macOS: 15.0
Xcode: 16.0 beta
CreateML: 6.0 beta
Enhance Dialogue not showing for me as an option anywhere .I’m using an Apple TV 4K 1st Generation running IOS 18 public beta ,JBL soundbar and LG OLED all have eARC.
We can use the CreateML App to build object tracking model in Xcode 16, but is it possible to use CreateML framework as well?
No documentation of Create ML object tracking is found yet. The latest documentation I can found is Xcode 15.
https://developer.apple.com/documentation/CreateML?changes=latest_minor
Really apricated the new feature of object tracking, thank you Apple Team.
The Translation API introduced at Session 10117 is impressive, but limiting it to SwiftUI is restrictive.
This API works great in the demo, but for more complex apps, it lacks flexibility because it is bound to SwiftUI Views.
Please consider making it available in non-SwiftUI environments.
When I add AppEnity to my model, I receive this error that is still repeated for each attribute in the model. The models are already marked for Widget Extension in Target Membership. I have already cleaned and restarted, nothing works. Will anyone know what I'm doing wrong?
Unable to find matching source file for path "@_swiftmacro_21HabitWidgetsExtension0A05ModelfMm.swift"
import SwiftData
import AppIntents
enum FrecuenciaCumplimiento: String, Codable {
case diario
case semanal
case mensual
}
@Model
final class Habit: AppEntity {
@Attribute(.unique) var id: UUID
var nombre: String
var descripcion: String
var icono: String
var color: String
var esHabitoPositivo: Bool
var valorObjetivo: Double
var unidadObjetivo: String
var frecuenciaCumplimiento: FrecuenciaCumplimiento
static var typeDisplayRepresentation: TypeDisplayRepresentation = "Hábito"
static var defaultQuery = HabitQuery()
var displayRepresentation: DisplayRepresentation {
DisplayRepresentation(title: "\(nombre)")
}
static var allHabits: [Habit] = [
Habit(id: UUID(), nombre: "uno", descripcion: "", icono: "circle", color: "#BF0000", esHabitoPositivo: true, valorObjetivo: 1.0, unidadObjetivo: "", frecuenciaCumplimiento: .mensual),
Habit(id: UUID(), nombre: "dos", descripcion: "", icono: "circle", color: "#BF0000", esHabitoPositivo: true, valorObjetivo: 1.0, unidadObjetivo: "", frecuenciaCumplimiento: .mensual)
]
/*
static func loadAllHabits() async throws {
do {
let modelContainer = try ModelContainer(for: Habit.self)
let descriptor = FetchDescriptor<Habit>()
allHabits = try await modelContainer.mainContext.fetch(descriptor)
} catch {
// Manejo de errores si es necesario
print("Error al cargar hábitos: \(error)")
throw error
}
}
*/
init(id: UUID = UUID(), nombre: String, descripcion: String, icono: String, color: String, esHabitoPositivo: Bool, valorObjetivo: Double, unidadObjetivo: String, frecuenciaCumplimiento: FrecuenciaCumplimiento) {
self.id = id
self.nombre = nombre
self.descripcion = descripcion
self.icono = icono
self.color = color
self.esHabitoPositivo = esHabitoPositivo
self.valorObjetivo = valorObjetivo
self.unidadObjetivo = unidadObjetivo
self.frecuenciaCumplimiento = frecuenciaCumplimiento
}
@Relationship(deleteRule: .cascade)
var habitRecords: [HabitRecord] = []
}
struct HabitQuery: EntityQuery {
func entities(for identifiers: [Habit.ID]) async throws -> [Habit] {
//try await Habit.loadAllHabits()
return Habit.allHabits.filter { identifiers.contains($0.id) }
}
func suggestedEntities() async throws -> [Habit] {
//try await Habit.loadAllHabits()
return Habit.allHabits// .filter { $0.isAvailable }
}
func defaultResult() async -> Habit? {
try? await suggestedEntities().first
}
}
How do I directly input landmarks to the activity classifier rather than inputting an image/video?
I create an app that allow user tap on button in widget but when app is running it's got an error when tap: Could not find an intent with identifier AppIntentsIdentifier , mangled TypeName: Optional("Somethings")
Hi, the following model does not run on ANE. Inspecting with deCoreML I see the error ane: Failed to retrieved zero_point.
import numpy as np
import coremltools as ct
from coremltools.converters.mil import Builder as mb
import coremltools.converters.mil as mil
B, CIN, COUT = 512, 1024, 1024 * 4
@mb.program(
input_specs=[
mb.TensorSpec((B, CIN), mil.input_types.types.fp16),
],
opset_version=mil.builder.AvailableTarget.iOS18
)
def prog_manual_dequant(
x,
):
qw = np.random.randint(0, 2 ** 4, size=(COUT, CIN), dtype=np.int8).astype(mil.mil.types.np_uint4_dtype)
scale = np.random.randn(COUT, 1).astype(np.float16)
offset = np.random.randn(COUT, 1).astype(np.float16)
# offset = np.random.randint(0, 2 ** 4, size=(COUT, 1), dtype=np.uint8).astype(mil.mil.types.np_uint4_dtype)
dqw = mb.constexpr_blockwise_shift_scale(data=qw, scale=scale, offset=offset)
return mb.linear(x=x, weight=dqw)
cml_qmodel = ct.convert(
prog_manual_dequant,
compute_units=ct.ComputeUnit.CPU_AND_NE,
compute_precision=ct.precision.FLOAT16,
minimum_deployment_target=ct.target.iOS18,
)
Whereas if I use an offset with the same dtype as the weights (uint4 in this case), it does run on ANE
Tested on coremltools 8.0b1, on macOS 15.0 beta 2/Xcode 15 beta 2, and macOS 15.0 beta 3/Xcode 15 beta 3.
I would like to split up my intents into smaller intents with more atomic pieces of functionality that I can then call one intent from another. For example:
struct SumValuesIntent: AppIntent {
static var title: LocalizedStringResource { "Sum Values" }
let a: Int
let b: Int
init(a: Int, b: Int) {
self.a = a
self.b = b
}
init() {
self.init(a: 0, b: 0)
}
func perform() async throws -> some IntentResult {
let sum = a + b
print("SumValuesIntent:", sum)
return .result(value: sum)
}
}
struct PrintValueIntent: AppIntent {
static var title: LocalizedStringResource { "Print Value" }
let string: String
init(string: String) {
self.string = string
}
init() {
self.init(string: "")
}
func perform() async throws -> some IntentResult {
print("PrintValueIntent:", string)
return .result()
}
}
What is the best way to chain intents like these? I tried
.result(opensIntent: PrintValueIntent(string: String(describing: sum)))
as the return type of SumValuesIntent.perform but that doesn't seem to work. Then I tried
try await PrintValueIntent(string: String(describing: sum)).perform()
as the return type and that works but I'm not sure that's the correct way to do it.
The Keras Embedding layer cannot be calculated on Metal because of the missing Op:StatelessRandomGetKeyCounter, as shown in this error message:
tensorflow.python.framework.errors_impl.InvalidArgumentError: Could not satisfy device specification '/job:localhost/replica:0/task:0/device:GPU:0'. enable_soft_placement=0. Supported device types [CPU]. All available devices [/job:localhost/replica:0/task:0/device:GPU:0, /job:localhost/replica:0/task:0/device:CPU:0]. [Op:StatelessRandomGetKeyCounter]
A workaround is to enable soft placement, but this obviously is slower:
tf.config.set_soft_device_placement(True)
Reporting it here as recommended by the TensorFlow Plugin Metal team.
I'm trying to use a custom system image for my App Shortcut:
AppShortcut(
intent: AppOpenIntent(),
phrases: ["Open \(.applicationName)"],
shortTitle: "Open App",
systemImageName: "custom.image"
)
I've added custom.image to Images.xcassets for my project, but the shortcuts icon is always blank in the Shortcuts app. Is it possible to use custom system images for app shortcuts?
I'm getting widespread reports from users trialling iOS 17.6 public beta that Siri Shortcuts are failing whenever they enter any text that looks like a URL.
It's getting reported to me because my app happens to have an app intent with a string parameter which can contain a URL in some circumstances.
However it's easily reproducible outside of my app: just create a 2 line shortcut like the one below. If you change "This is some text" to "https://www.apple.com" the shortcut below will fail:
In iOS 17.5 entering "https://www.apple.com" works fine.
I've raised feedback on this (FB14206088) but can anyone confirm that this is indeed a bug and not some weird new feature of Shortcuts where the contents of a variable can somehow change the type of a variable?
It would be very, very bad if this were so.
iOS 18 App Intents while supporting iOS 17
Hello,
I have an existing app that supports iOS 17. I already have three App Intents but would like to add some of the new iOS 18 app intents like ShowInAppSearchResultsIntent.
However, I am having a hard time using #available or @available to limit this ShowInAppSearchResultsIntent to iOS 18 only while still supporting iOS 17.
Obviously, the ShowInAppSearchResultsIntent needs to use @AssistantIntent which is iOS 18 only, so I mark that struct as @available(iOS 18, *). That works as expected. It is when I need to add this "SearchSnippetIntent" intent to the AppShortcutsProvider, that I begin to have trouble doing. See code below:
struct SnippetsShortcutsAppShortcutsProvider: AppShortcutsProvider {
@AppShortcutsBuilder
static var appShortcuts: [AppShortcut] {
//iOS 17+
AppShortcut(intent: SnippetsNewSnippetShortcutsAppIntent(), phrases: [
"Create a New Snippet in \(.applicationName) Studio",
], shortTitle: "New Snippet", systemImageName: "rectangle.fill.on.rectangle.angled.fill")
AppShortcut(intent: SnippetsNewLanguageShortcutsAppIntent(), phrases: [
"Create a New Language in \(.applicationName) Studio",
], shortTitle: "New Language", systemImageName: "curlybraces")
AppShortcut(intent: SnippetsNewTagShortcutsAppIntent(), phrases: [
"Create a New Tag in \(.applicationName) Studio",
], shortTitle: "New Tag", systemImageName: "tag.fill")
//iOS 18 Only
AppShortcut(intent: SearchSnippetIntent(), phrases: [
"Search \(.applicationName) Studio",
"Search \(.applicationName)"
], shortTitle: "Search", systemImageName: "magnifyingglass")
}
let shortcutTileColor: ShortcutTileColor = .blue
}
The iOS 18 Only AppShortcut shows the following error but none of the options seem to work. Maybe I am going about it the wrong way.
'SearchSnippetIntent' is only available in iOS 18 or newer
Add 'if #available' version check
Add @available attribute to enclosing static property
Add @available attribute to enclosing struct
Thanks in advance for your help.
Hi everyone, I attempted to use the MultivariateLinearRegressor from the Create ML Components framework to fit some multi-dimensional data linearly (4 dimensions in my example). I aim to obtain multi-dimensional output points (2 points in my example). However, when I fit the model with my training data and test it, it appears that only the first element of my training data is used for training, regardless of whether I use CreateMLComponents.AnnotatedBatch or [CreateMLComponents.AnnotatedFeature, CoreML.MLShapedArray>] as input.
let sourceMatrix: [[Double]] = [
[0,0.1,0.2,0.3],
[0.5,0.2,0.6,0.2]
]
let referenceMatrix: [[Double]] = [
[0.2,0.7],
[0.9,0.1]
]
Here is a test code to test the function (ios 18.0 beta, Xcode 16.0 beta)
In this example I train the model to learn 2 multidimensional points (4 dimensions) and here are the results of the predictions:
▿ 2 elements
▿ 0 : AnnotatedPrediction<MLShapedArray<Double>, MLShapedArray<Double>>
▿ prediction : 0.20000000298023224 0.699999988079071
▿ _storage : <StandardStorage<Double>: 0x600002ad8270>
▿ annotation : 0.2 0.7
▿ _storage : <StandardStorage<Double>: 0x600002b30600>
▿ 1 : AnnotatedPrediction<MLShapedArray<Double>, MLShapedArray<Double>>
▿ prediction : 0.23158159852027893 0.9509953260421753
▿ _storage : <StandardStorage<Double>: 0x600002ad8c90>
▿ annotation : 0.9 0.1
▿ _storage : <StandardStorage<Double>: 0x600002b55f20>
0.23158159852027893 0.9509953260421753 is totally random and should be far more closer to [0.9,0.1].
Here is the test code : ( i run it on "My mac, Designed for Ipad")
ContentView.swift
import CoreImage
import CoreImage.CIFilterBuiltins
import UIKit
import CoreGraphics
import Accelerate
import Foundation
import CoreML
import CreateML
import CreateMLComponents
func createMLShapedArray(from array: [Double], shape: [Int]) -> MLShapedArray<Double> {
return MLShapedArray<Double>(scalars: array, shape: shape)
}
func calculateTransformationMatrixWithNonlinearity(sourceRGB: [[Double]], referenceRGB: [[Double]], degree: Int = 3) async throws -> MultivariateLinearRegressor<Double>.Model {
let annotatedFeatures2 = zip(sourceRGB, referenceRGB).map { (featureArray, targetArray) -> AnnotatedFeature<MLShapedArray<Double>, MLShapedArray<Double>> in
let featureMLShapedArray = createMLShapedArray(from: featureArray, shape: [featureArray.count])
let targetMLShapedArray = createMLShapedArray(from: targetArray, shape: [targetArray.count])
return AnnotatedFeature(feature: featureMLShapedArray, annotation: targetMLShapedArray)
}
// Flatten the sourceRGBPoly into a single-dimensional array
var flattenedArray = sourceRGB.flatMap { $0 }
let featuresMLShapedArray = createMLShapedArray(from: flattenedArray, shape: [2, 4])
flattenedArray = referenceRGB.flatMap { $0 }
let targetMLShapedArray = createMLShapedArray(from: flattenedArray, shape: [2, 2])
// Create AnnotatedFeature instances
/* let annotatedFeatures2: [AnnotatedFeature<MLShapedArray<Double>, MLShapedArray<Double>>] = [
AnnotatedFeature(feature: featuresMLShapedArray, annotation: targetMLShapedArray)
]*/
let annotatedBatch = AnnotatedBatch(features: featuresMLShapedArray, annotations: targetMLShapedArray)
var regressor = MultivariateLinearRegressor<Double>()
regressor.configuration.learningRate = 0.1
regressor.configuration.maximumIterationCount=5000
regressor.configuration.batchSize=2
let model = try await regressor.fitted(to: annotatedBatch,validateOn: nil)
//var model = try await regressor.fitted(to: annotatedFeatures2)
// Proceed to prediction once the model is fitted
let predictions = try await model.prediction(from: annotatedFeatures2)
// Process or use the predictions
print(predictions)
print("Predictions:", predictions)
return model
}
struct ContentView: View {
var body: some View {
VStack {}
.onAppear {
Task {
do {
let sourceMatrix: [[Double]] = [
[0,0.1,0.2,0.3],
[0.5,0.2,0.6,0.2]
]
let referenceMatrix: [[Double]] = [
[0.2,0.7],
[0.9,0.1]
]
let model = try await calculateTransformationMatrixWithNonlinearity(sourceRGB: sourceMatrix, referenceRGB: referenceMatrix, degree: 2
)
print("Model fitted successfully:", model)
} catch {
print("Error:", error)
}
}
}
}
}
I‘ve created text classification project and selected BERT algorithm With 100 iterations for json file. Json file is valid but training always cancels on 37 iteration…
Because tool does not provide any cancellation reasons I have no clue why it happens. Can I check reasons somehow? Or do anyone knows possible reasons or solutions for this?