Get distance from uvd and intrinsic matrix?

Question

mingm OP

Created May ’22

Replies 5

Boosts 0

Views 2.2k

Participants 5

Hello!

I am having trouble calculating accurate distances in the real world using the camera's returned intrinsic matrix and pixel coordinates/depths captured from the iPhone's LiDAR. For example, in the image below, I set a mug 0.5m from the phone. The mug is 8.5cm wide. The intrinsic matrix returned from the phone's AVCameraCalibrationData class has focalx = 1464.9269, focaly = 1464.9269, cx = 960.94916, and cy = 686.3547. Selecting the two pixel locations denoted in the image below, I calculated each one's xyz coordinates using the formula:

    x = d * (u - cx) / focalx
    y = d * (v - cy) / focaly
    z = d

Where I get depth from the appropriate pixel in the depth map - I've verified that both depths were 0.5m. I then calculate the distance between the two points to get the mug width. This gives me a calculated width of 0.0357, or 3.5 cm, instead of the 8.5cm I was expecting. What could be accounting for this discrepancy?

Screen Shot 2022-05-02 at 12.45.46 PM.png

Thank you so much for your help!

Boost

Answer 1

jcell OP

Jul ’22

I encountered the same issue, and the key seems to be this:

var intrinsicMatrixReferenceDimensions: CGSize The image dimensions to which the camera's intrinsic matrix values are relative.

If it has a different value from the image dimensions you're using (photo or depth data), then the intrinsic matrix is incorrect and needs to be scaled.

One my iPhone 13 Pro, image size, depth data size and intrinsicMatrixReferenceDimensions are all different.

0

Answer 2

jcell OP

Jul ’22

In addition, this seems to confirm my answer: https://github.com/flutter/flutter/issues/78617

0

Answer 3

masaldana2 OP

Jan ’23

same here

0

Answer 4

zqj2000 OP

Mar ’24

I used to have this problem too. But later I figured out that cause the resolution of YUV and depth image are not the same. For example, your YUV image is 4 times wider and 4 times higher, so when you using the cameraIntrinsics from YUV on depth image, which is like 200 * 250, you are supposed to divide the intrinsic parameters by 4. eg, focalx = 1464.9269 / 4, focaly = 1464.9269 / 4, cx = 960.94916 / 4, and cy = 686.3547 / 4

0

Answer 5

kc_jmx OP

Jun ’24

Hey @zqj2000 I'm getting hung up on this problem as well. Most likely in the rectification of the depth data so I can use 2 points in space to calculate the distance. Same as the OP.

My depth image coords (640,360), video image (1280,720) and intrinsicReference (4032,2268). If the depth is .389 (which is in meters) and my 2 points are (421.62951779, 475.694515546) and (428.601582527, 921.03613217) then I expect a length around 0.1651 meters. But I'm around .33 meters using the following code:

    let xrwPoint1 = depthValue1 * (Float(convertedCGScreenPoint1.x) - adjustedCx) / adjustedFx

    let yrwPoint1 = depthValue1 * (Float(convertedCGScreenPoint2.y) - adjustedCy) / adjustedFy

    let xrwPoint2 = depthValue2 * (Float(convertedCGScreenPoint2.x) - adjustedCx) / adjustedFx

    let yrwPoint2 = depthValue2 * (Float(convertedCGScreenPoint2.y) - adjustedCy) / adjustedFy

    let point1In3D = simd_float3(xrwPoint1, yrwPoint1, distanceValue1)
    let point2In3D = simd_float3(xrwPoint2, yrwPoint2, distanceValue2)
     
    let dist = simd_precise_distance(point1In3D, point2In3D)

I should note that I do NOT get the same values on an iPhone 12 and iPad Pro. So the math must be inaccurate somewhere. If I was getting the same value but it was off, that would be encouraging.

There has to be more rectifying of the depth data, correct? Thank you for any advice!

0