We have an application that receives a message (through MQTT) from an external system to snap a photo, runs a CoreML vision request on the image, and then sends the results back. The customer has 100s of devices and recently on a couple of those devices (13 pros), the customer encountered an issue in which the devices were not responding in time. There was no crash, just some individual inferences were slowed down. The device performs 1000s of requests per day. Upon further evaluation of the request before and after in the device logs, I noticed that Apple loads the following
default 2024-09-04 13:18:31.310401 -0400 ProcessName Processing image for reference: ***
default 2024-09-04 13:18:31.403606 -0400 ProcessName Found matching service: H1xANELoadBalancer
default 2024-09-04 13:18:31.403646 -0400 ProcessName Found matching service: H11ANEIn
default 2024-09-04 13:18:31.403661 -0400 ProcessName Found ANE device :1
default 2024-09-04 13:18:31.403681 -0400 ProcessName Total num of devices 1
default 2024-09-04 13:18:31.403681 -0400 ProcessName (Single-ANE System) Opening H11ANE device at index 0
default 2024-09-04 13:18:31.403681 -0400 ProcessName H11ANEDevice::H11ANEDeviceOpen, usage type: 1
In a good scenario (above), these actions will performed very quickly (in a split second). The app doesn't do anything until coreml inference result is returned. In the bad scenario (below), there is a delay of about 4 seconds from app passing the control to vision request and then getting the response back (leading to timeouts with the customer)
default 2024-09-04 13:19:08.777468 -0400 ProcessName Processing image for reference: ZZZ
default 2024-09-04 13:19:12.199758 -0400 ProcessName Found matching service: H1xANELoadBalancer
default 2024-09-04 13:19:12.199800 -0400 ProcessName Found matching service: H11ANEIn
default 2024-09-04 13:19:12.199812 -0400 ProcessName Found ANE device :1
default 2024-09-04 13:19:12.199832 -0400 ProcessName Total num of devices 1
default 2024-09-04 13:19:12.199834 -0400 ProcessName (Single-ANE System) Opening H11ANE device at index 0
default 2024-09-04 13:19:12.199834 -0400 ProcessName H11ANEDevice::H11ANEDeviceOpen, usage type: 1
The logs are in order, I haven't removed anything. The code is fairly simple, it's just running a vision request without doing much. Has anyone encountered this before?