DockKit tracking becomes erratic with increased zoom factor in iOS app

Question

Created 1w

Replies 0

Boosts 0

Participants 1

I'm developing an iOS app using DockKit to control a motorized stand. I've noticed that as the zoom factor of the AVCaptureDevice increases, the stand's movement becomes increasingly erratic up and down, almost like a pendulum motion. I'm not sure why this is happening or how to fix it.

Here's a simplified version of my tracking logic:

func trackObject(_ boundingBox: CGRect, _ dockAccessory: DockAccessory) async throws {
    guard let device = AVCaptureDevice.default(for: .video),
          let input = try? AVCaptureDeviceInput(device: device) else {
        fatalError("Camera not available")
    }
    
    let currentZoomFactor = device.videoZoomFactor
    let dimensions = device.activeFormat.formatDescription.dimensions
    let referenceDimensions = CGSize(width: CGFloat(dimensions.width), height: CGFloat(dimensions.height))
    
    let intrinsics = calculateIntrinsics(for: device, currentZoom: Double(currentZoomFactor))
    
    let deviceOrientation = UIDevice.current.orientation
    let cameraOrientation: DockAccessory.CameraOrientation = {
        switch deviceOrientation {
        case .landscapeLeft: return .landscapeLeft
        case .landscapeRight: return .landscapeRight
        case .portrait: return .portrait
        case .portraitUpsideDown: return .portraitUpsideDown
        default: return .unknown
        }
    }()
    
    let cameraInfo = DockAccessory.CameraInformation(
        captureDevice: input.device.deviceType,
        cameraPosition: input.device.position,
        orientation: cameraOrientation,
        cameraIntrinsics: useIntrinsics ? intrinsics : nil,
        referenceDimensions: referenceDimensions
    )
    
    let observation = DockAccessory.Observation(
        identifier: 0,
        type: .object,
        rect: boundingBox
    )
    let observations = [observation]
    
    try await dockAccessory.track(observations, cameraInformation: cameraInfo)
}

func calculateIntrinsics(for device: AVCaptureDevice, currentZoom: Double) -> matrix_float3x3 {
    let dimensions = CMVideoFormatDescriptionGetDimensions(device.activeFormat.formatDescription)
    let width = Float(dimensions.width)
    let height = Float(dimensions.height)
    
    let diagonalPixels = sqrt(width * width + height * height)
    let estimatedFocalLength = diagonalPixels * 0.8
    
    let fx = Float(estimatedFocalLength) * Float(currentZoom)
    let fy = fx
    let cx = width / 2.0
    let cy = height / 2.0
    
    return matrix_float3x3(
        SIMD3<Float>(fx, 0, cx),
        SIMD3<Float>(0, fy, cy),
        SIMD3<Float>(0, 0, 1)
    )
}

I'm calling this function regularly (10-30 times per second) with updated bounding box information. The erratic movement seems to worsen as the zoom factor increases.

Questions:

Why might increasing the zoom factor cause this erratic movement?
I'm currently calculating camera intrinsics based on the current zoom factor. Is this approach correct, or should I be doing something differently?
Are there any other factors I should consider when using DockKit with a variable zoom?
Could the frequency of calls to trackRider (10-30 times per second) be contributing to the erratic movement? If so, what would be an optimal frequency?

Any insights or suggestions would be greatly appreciated. Thanks!

Boost