Convert CGPoint into SCNVector3

Question

abhijit.pargaonkar OP

Created Jul ’24

Replies 3

Boosts 0

Participants 2

I want to convert CGPoint into SCNVector3. I am using ARFaceTrackingConfiguration for face tracking.

Below is my code to convert SCNVector3 to CGPoint

let point = faceAnchor.verticeAndProjection(to: sceneView, facePoint: faceAnchor.geometry.vertices[0])

print(point, faceAnchor.geometry.vertices[0])

which prints below values

CGPoint = (350.564453125, 643.4456787109375) 
SIMD3<Float>(0.014480735, 0.01397189, 0.04508282)


extension ARFaceAnchor{
	// struct to store the 3d vertex and the 2d projection point
	struct VerticesAndProjection {
   		 var vertex: SIMD3<Float>
    	var projected: CGPoint
	}

	// return a struct with vertices and projection
	func verticeAndProjection(to view: ARSCNView, facePoint: Int) -> CGPoint{
    
   	 let point = SCNVector3(geometry.vertices[facePoint])
    
    	let col = SIMD4<Float>(SCNVector4())
    	let pos = SIMD4<Float>(SCNVector4(point.x, point.y, point.z, 1))
    	let pworld = transform * simd_float4x4(col, col, col, pos)
    	let vect = view.projectPoint(SCNVector3(pworld.position.x, pworld.position.y, pworld.position.z))
    	let p = CGPoint(x: CGFloat(vect.x), y: CGFloat(vect.y))
    	return p
	}
}

extension matrix_float4x4 {

	/// Get the position of the transform matrix.
	public var position: SCNVector3 {
   		 get{
   	    	 return SCNVector3(self[3][0], self[3][1], self[3][2])
   		 }
	 }
}

Now i want to convert same CGPoint to SCNVector3.

I tried using below code but it is not giving expected values, which is SIMD3<Float>(0.014480735, 0.01397189, 0.04508282)

    let projectedOrigin = sceneView.projectPoint(SCNVector3Zero)
    let unproject = sceneView.unprojectPoint(SCNVector3(point.x, point.y, CGFloat(projectedOrigin.z)))
    let vector = SCNVector3(unproject.x, unproject.y, unproject.z)

Is there any way to convert CGPoint to SCNVector3? I cannot use hitTest because this CGPoint is not present on the node. It is present somewhere on the face area.

Answered by DTS Engineer in 797122022

Hello @abhijit.pargaonkar,

In your example, “Original Position” is in local space, but “unprojectedPosition” is in world space. This could explain the difference you have observed.

Best regards,

Greg

Boost

Answer 1

DTS Engineer OP

Apple

Jul ’24

Hello @abhijit.pargaonkar,

unprojectPoint is expecting the z-value passed in to be the projected depth value that corresponds to the original point.

If you run this code, you will see that if you preserve that projected z-value, you can unprotect back to the original point:

let firstVertex = faceAnchor.geometry.vertices[0]
        
        let localPosition = SIMD4(firstVertex.x, firstVertex.y, firstVertex.z, 1)
        
        let worldPosition = faceAnchor.transform * localPosition
        
        node.simdWorldPosition = worldPosition[SIMD3(0,1,2)]
        
        print(node.position)
        
        let projectedPosition = sceneView.projectPoint(node.position)
        
        print(projectedPosition)
        
        let unprojectedPosition = sceneView.unprojectPoint(projectedPosition)
        
        print(unprojectedPosition)

Best regards,

Greg

0

Answer 2

abhijit.pargaonkar OP

Jul ’24

Above solution does not work. It gives different values

func realWorldPosition(for originalVertex: SCNVector3, point: CGPoint, faceAnchor: ARFaceAnchor) -> (CGPoint?, SCNVector3?)? {
    
    let firstVertex = originalVertex
    print("\n\n\nOriginal position: ", firstVertex)
    
    let localPosition = SIMD4(firstVertex.x, firstVertex.y, firstVertex.z, 1)
            
    let col = SIMD4<Float>(SCNVector4())
    let worldPosition = faceAnchor.transform * simd_float4x4(col, col, col, localPosition)
    
    let projectedPosition = sceneView.projectPoint(worldPosition.position)
    
    print("projectedPosition: ", projectedPosition)

    let unprojectedPosition = sceneView.unprojectPoint(projectedPosition)
    
    print("unprojectedPosition: ", unprojectedPosition)

    let vec = SCNVector3(x: unprojectedPosition.x, y: unprojectedPosition.y, z: unprojectedPosition.z)
    
    let p = CGPoint(x: CGFloat(projectedPosition.x), y: CGFloat(projectedPosition.y))
    return  (p, vec)
    
}

Original position:  SCNVector3(x: 0.015014233, y: 0.01659775, z: 0.04286898)
projectedPosition:  SCNVector3(x: 405.9854, y: 609.32275, z: 0.9972021)
unprojectedPosition:  SCNVector3(x: -0.0027770074, y: 0.101203665, z: -0.3426712)

0

Answer 3

DTS Engineer OP

Apple

Jul ’24

Accepted Answer

Hello @abhijit.pargaonkar,

In your example, “Original Position” is in local space, but “unprojectedPosition” is in world space. This could explain the difference you have observed.

Best regards,

Greg

1