Retrieve AnchorEntity (Hand Locations) Position through update(context: Scene) function

Question

XWDev OP

Created Aug ’24

Replies 2

Boosts 0

Participants 4

Hello there,

I'm currently working on a Hand Tracking System. I've already placed some spheres on some joint points on the left and right hand. Now I want to access the translation/position value of these entities in the update(context: Scene) function. Now my question is, is it possible to access them via .handAnchors(), or which types of .handSkeleton.joint(name) are referencing the same entity? (E.g. is AnchorEntity(.hand(.right, location: .indexFingerTip)) the same as handSkeleton.joint(.indexFingerTip). The goal would be to access the translation of the joints where a sphere has been placed per hand and to be able to update the data every frame through the update(context) function.

I would very much appreciate any help!

See code example down below:

ImmersiveView.swift

import SwiftUI
import RealityKit
import ARKit

struct ImmersiveView: View {
    
    public var body: some View {
        RealityView { content in
            
            /* HEAD */
            let headEntity = AnchorEntity(.head)
            content.add(headEntity)
            
            /* LEFT HAND */
            let leftHandWristEntity = AnchorEntity(.hand(.left, location: .wrist))
            let leftHandIndexFingerEntity = AnchorEntity(.hand(.left, location: .indexFingerTip))
           
            let leftHandWristSphere = ModelEntity(mesh: .generateSphere(radius: 0.02), materials: [SimpleMaterial(color: .red, isMetallic: false)])
            let leftHandIndexFingerSphere = ModelEntity(mesh: .generateSphere(radius: 0.01), materials: [SimpleMaterial(color: .orange, isMetallic: false)])
            
            leftHandWristEntity.addChild(leftHandWristSphere)
            content.add(leftHandWristEntity)
            
            leftHandIndexFingerEntity.addChild(leftHandIndexFingerSphere)
            content.add(leftHandIndexFingerEntity)
           
        }
        
    }
}

TrackingSystem.swift

import SwiftUI
import simd
import ARKit
import RealityKit

public class TrackingSystem: System {
    static let query = EntityQuery(where: .has(AnchoringComponent.self))
    
    private let arKitSession = ARKitSession()
    private let worldTrackingProvider = WorldTrackingProvider()
    private let handTrackingProvider = HandTrackingProvider()
    
    public required init(scene: RealityKit.Scene) {
            setUpSession()
    }
    
    
    private func setUpSession() {
        Task {
            do {
                try await arKitSession.run([worldTrackingProvider, handTrackingProvider])   
            } catch {
                print("Error: \(error)")
            }
        }
    }

    public func update(context: SceneUpdateContext) {
        guard worldTrackingProvider.state == .running && handTrackingProvider.state == .running else { return }
        
        let _ = context.entities(matching: Self.query, updatingSystemWhen: .rendering)
        if let avp = worldTrackingProvider.queryDeviceAnchor(atTimestamp: currentTime) {
                let hands = handTrackingProvider.handAnchors(at: currentTime)
                ...
            }
        }
    }

Boost

Answer 1

Vision Pro Engineer OP

Apple

3w

@XWDev

As of visionOS 2.0 you can access an AnchorEntity's transform/position after you start a SpatialTrackingSession. The snippet below starts a SpatialTrackingSession that enables you to access the transform/position of hand anchor entities.

.task {
            let configuration = SpatialTrackingSession.Configuration(
                            tracking: [.hand])
            session = SpatialTrackingSession()
            await session.run(configuration)
        }

After you start the session you can access leftHandWristEntity's transform or position using the corresponding properties or methods on Entity. For example, leftHandWristEntity.transform or leftHandWristEntity.position(relativeTo: nil).

You can also access this information via the anchors, but it's likely more convenient to run a spatial tracking session and query the entities directly. Here's a snippet to read the transform information from the left hand wrist joint.

if anchor.chirality == .left{
    // get hand origin 
    let handOrigin = anchor.originFromAnchorTransform 
    // get wrist transform relative to hand origin
    let wristTransformMatrix = anchor.handSkeleton?.joint(.wrist).anchorFromJointTransform      
}

0

Answer 2

Vision Pro Engineer OP

Apple

3w

Hi @XWDev

If your goal is to add an entity to every joint, I have a snippet for that. I'd be happy to share it with you.

0