I am working on an application where we are planning to use Metal for directly rendering custom content. When user looks at something on the rendered image, I want to get the position or ray of cursor (the point where the user is currently looking at) to render something else like a crosshair. Is it possible to get the cursor position information on VisionOS to accomplish this? How can I know if something is being hovered on by the eyes?
I'm afraid eye tracking information isn't currently available at all, regardless of the immersion level of your app. When creating an app in the mixed space, Apple handles eye tracking for you (but it's abstracted away), but when creating a fully immersive app you get nothing. So you'll need to come up with an alternative mechanism of input, for example using a game controller to manipulate an onscreen cursor.
It's a shame, and is the biggest limitation of the system right now IMO.
The sensor data are processed internally by system. The processing results are provided via DataProvider.
https://developer.apple.com/documentation/arkit/dataprovider
We have two alternatives to eye-tracking in visionOS:
ARKitSession - DataProvider - Anchor.
I hope this helpful.
To help protect people’s privacy, visionOS ARKit data is available only when your app presents a Full Space and other apps are hidden.
https://developer.apple.com/documentation/visionos/setting-up-access-to-arkit-data
Your device (iPhone, iPad, Vision Pro), hands, and fingers are visible to others, and there are fewer privacy issues. But the directions of your eyeballs are virtually invisible to others, potentially leading to privacy issues.
To protect people's privacy, data are selectively shared with developers.
Data provided to iOS/iPadOS developers but not to visionOS developers:
Data provided by visionOS to developers via DataProvider:
https://developer.apple.com/documentation/arkit/dataprovider
Data NOT provided to developers by either visionOS or iOS/iPadOS:
Under given conditions, we like to draw a ray pointing a digital content in space. Of course, eye-tracking data would be the first choice but not accessible for developers.
Alternatives to eye tracking data are DeviceAnchor and HandAnchor.
visionOS includes internally the hovering-framework for matching the ray of DeviceAnchor or HandAnchor (and eye-tracking) with a digital content in space.
It would be great if Apple made the hovoring API available to developers. How to identify the digital content in the scene that is closest to the device and next to the device's line of sight.