I am working with MeshAnchors, and I am having troubles getting to the classification of the triangles/faces. This post references the MeshAnchor.Geometry, and that struct does have a property named "classifications", but it is of type GeometrySource. I cannot find any classification information in GeometrySource. Am I missing something there? I think I am looking for something of type MeshAnchor.MeshClassification, but I cannot find any structs with this as a property.

Hello. I'm developing the app using ARKit and RealityKit. The purpose of the app is to scan the apartment and put furniture next to the walls. It works good, but if AR session takes more than 3 mins at some point app is crashed. According to crash report it's not something related to my code. I'm attaching crash report (company data is hidden). Any help is appreciated. Thanks in advance.

I see example code converting the results of a SpatialTap to a SIMD3 location. For example, from WWDC session Meet ARKit for spatial computing: let location3D = value.convert(value.location3D, from: .global, to: .scene) What I really want is a simd_float4x4 that includes orientation of the surface that the tap gesture/cast collided with? My goal is to place an object with its Y-axis along the normal of the surface that was tapped. For example, in the referenced WWDC session, they create a CollisionComponent from the MeshAnchor data. If that mesh data is covering a curved couch cushion, I would like the normal from that curved cushion (i.e., the closest triangle approximating it). Is this possible? My planned fallback is to only use planes for collision surfaces for tap gestures, extract the tap gesture value's entity (which I am hoping is the plane), and grab its transform for the orientation information. I am hoping Apple has a simple function call that is more general than my fallback approach.

Im working on the following problem: For a measurment application i want to take a picture of something laying on the ground, and given i will have the floor plane detecred, i plan to raycast 4 points from the corners of the screens, given the raycast land on this plane, i want to use those coordinates to do a perspective transform (warp) of the camera-image onto the new coordinates. This way i should be able to perform pixel-per-cm measurments. The prolem i have is the screen coordinates dont seem to reflect the camera-frame coordinates, and im not sure how to go from one to another.

Why does PhotogrammetrySession.isSupported return true if Object Capture is supported? It would be great if you could use PhotogrammetrySession on iOS devices without lidar and feed it a folder of pictures to make a 3D model. Thanks!

Hello, I'm interested in using the iOS on-device object capture API for photogrammetry, however I would like to integrate it in a web app. I understand that web apps cannot usually access system-level APIs, so I am unsure of whether or not this would be feasible to implement. I would greatly appreciate for any pointers in the right direction. Thank you!

I'd like to grab the current camera frame in visionOS. I have a Swift file (am new to Swift) that looks like this: import ARKit import SwiftUI class ARSessionManager: NSObject, ObservableObject, ARSessionDelegate { var arSession: ARSession override init() { arSession = ARSession() super.init() arSession.delegate = self } func startSession() { let configuration = ARWorldTrackingConfiguration() configuration.planeDetection = .horizontal arSession.run(configuration) } // ARSessionDelegate method to capture frames func session(_ session: ARSession, didUpdate frame: ARFrame) { // Process the frame, e.g., capture image data } } and I get errors including "Cannot find type 'ARSessionDelegate' in scope". Help? Is ARFrame called something different for Vision Pro?

I am running a modified RoomPllan app in my test environment I get two ARSessions active, sometimes more. It appears that the first one is created by Scene Kit because it is related go ARSCNView. Who controls that and what gets processed through it? I noticed that I get a lot of Session Interruptions from Sensor Failure when I am doing World Tracking and the first one happens almost immediately. When I get the room capture delegates fired up I start getting images to the delegate via a second session that is collecting images. How do I tell which session is the scene kit session and which one is the RoomCapture session on thee fly when it comes through the delegate? Is there a difference in the object desciptor that I can use as a differentiator? Relying on the Address of the ARSession buffer being different is okay if you get your timing right. It wasn't clear from any of the documentation that there would be TWO or more AR Sessions delivering data through the delegates. The books on the use of ARKIT are not much help in determining the partition of responsibilities between the origins. The buffer arrivals at the functions supported by the delegates do not have a clear delineation of what function is delivered through which delegate discernible from the highly fragmented documentation provided by the Developer document library. Can someone give me some guidance here? Are there sources for CLEAR documentation of what is delivered via which delegate for the various interfaces?

Hi, I want to develop an AR App for construction site on which i need to prove the calibration quality of the 3D model on plane. For that i have already retrieve informations like TrackingState, points cloud, confidence map... I would like to know if the ConfidenceLevel, that appears to be an enumeration, is available or if I need to analyse the points cloud to make my own confidence level. And also if you have informations on how can I know the precision of the 3D map on real life.

When I use LiDAR, AVCaptureDeviceTypeBuiltInLiDARDepthCamera is used. As AVCaptureDeviceTypeBuiltInLiDARDepthCamera is A device that consists of two cameras, one LiDAR and one YUV. I found that the LiDAR data is 30fps, even making the YUV data 30 fps. But I really need the 240fps YUV data. Is there a way to utilize the 30fps LiDAR with 240fps YUV camera? Any reply would be appreciated.

Hi, I have a code that uses ImageTrackingProvider. I am experimenting with glyphs of various complexity and structure to understand which ones would be more superior for recognition. Due to the absence of a color printer, I am mostly experimenting with monochrome glyphs as well as some color-paper squares. I am getting mixed results and would like to validate whether what I got are the expected results for the current capabilities of ARKit & VisionPro, or if there is still an opportunity for improvement by selecting different glyphs. So far, I have used a colored square of size 5x5 cm, as well as two glyphs provided below. ARKit Glyph Abstract Glyph The ARKit Glyph is not recognizable by ARKit or VisionPro at all, no matter the lighting conditions or the angles from which I view it. The Abstract Glyph is recognizable consistently at a 90-degree angle, and sometimes at other angles too. The maximum distance at which I was able to detect it was around 15cm, maybe less. I am really curious if there is any specification that I can check against to understand whether my glyphs are good or not, and at what maximum distance such glyphs can be recognized if they were 5x5cm in size. I am also curious whether ARKit is capable of recognizing images of 5x5cm size at a distance between 2 and 3 meters, and if so, how I should prepare the glyph for such requirements. Thanks in advance, Nikita ps I am skipping a question about yaw angles of image, as well as angel between normal of an image & camera view but I guess they also have their impact on ability to recognize original image.

According to https://developer.apple.com/documentation/visionos/bringing-your-arkit-app-to-visionos#Isolate-ARKit-features-not-available-in-visionOS, Body Tracking and several other features are not available on VisionOS. So is there any ETA for these ARKit's features to be supported in VisionOS? Thanks.

let apple = try Entity.load(named: "apple", in: realityKitContentBundle) works, but let apple = try Entity.loadModel(named: "apple", in: realityKitContentBundle) does not work ie. (error.localizedDescription = Failed to find resource with name "apple" in bundle) I am unsure what is causing the problem, apple.usda was created in Reality Composer Pro from primitives and has a single apple object (no root). When I load with Entity.load and print apple, I get: ▿ 'apple' : Entity, children: 1 ⟐ Transform ⟐ SynchronizationComponent ▿ 'apple' : ModelEntity ⟐ ModelComponent ⟐ Transform ⟐ CollisionComponent ⟐ PhysicsBodyComponent ⟐ SynchronizationComponent This nested hierarchy seems redundant to me, is it preferred in AR kit to have such a structure? Why am I unable to load usda directly as a ModelEntity?

I am working on a sports training app for VisionOS that requires recognition of fast-moving objects. Currently, I am using ImageTrackingProvider to tag the objects I need. I have noticed that while recognition works well for stationary objects, it does not perform well in tracking moving objects. I assume there are a mix of factors at play: I am not sure if ARKit is actually built for tracking moving objects, so there could be a refresh rate limit enforced to save battery. My reference image could be suboptimal/too complex to recognize quickly. I am not sure if ARKit is actually built for tracking moving objects, so there could be a refresh rate limit enforced to save battery. My reference image could be suboptimal/too complex to recognize quickly. While I can't do anything about #1, I am curious about recommendations for #2. Are there recommendations for the best size of a reference image, its color (would black and white work better?), and its complexity? Also, since the ARKit Resource Group seems to support JPEG & PNG, is there any specific preference for one over the other? Should I prepare the images in any special way to achieve the best possible performance? Thanks.

I am working on a sports training app for VisionOS that requires recognition of fast-moving objects. Currently, I am using ImageTrackingProvider to tag the objects I need. I have noticed that while recognition works well for stationary objects, it does not perform well in tracking moving objects. I assume there are a mix of factors at play: I am not sure if ARKit is actually built for tracking moving objects, so there could be a refresh rate limit enforced to save battery. My reference image could be suboptimal/too complex to recognize quickly. While I can't do anything about #1, I am curious about recommendations for #2. Are there recommendations for the best size of a reference image, its color (would black and white work better?), and its complexity? Also, since the ARKit Resource Group seems to support JPEG & PNG, is there any specific preference for one over the other? Should I prepare the images in any special way to achieve the best possible performance? Thanks.

I am trying to implement a game where the character walks on the scene mesh. I am controlling the character with a game controller. I noticed there is a character controller component in Reality Composer Pro, I am aware that when this component is added, the player cannot have a collision or a physics component. I need an example that covers adding an entity with the character controller component to the scene and then moving the character using the moveCharacter function. I was also looking at the documentation https://developer.apple.com/documentation/realitykit/entity/movecharacter(by:deltatime:relativeto:collisionhandler:) Here it is also looking for deltaTime. Where do we get the deltaTime from? does it come from a system's update function, does that also mean that the character controller needs to be moved in the update method? Thanks, Sarang

I am new to visionOS development, just slowly figuring out the difference in immersion styles to figure out how I want my app to behave. It seems that when you use a progressive immersive space the minimum immersion level (set via the digital crown) is not 0? Meaning, there is no way to go from mixed to full by using the Digital Crown. Even when I try to set it to 0 (such as in the Destination Video sample), it pops back up to around 30-40%, and I always see the background. Is this expected behavior, or are there some settings that allow me to change this minimum immersion level? Further, in the video 'Meet ARKit for spatial computing', it is stated that to get access to ARKit tracking data you must use a 'Full Space', not the 'Shared Space'. This wording is confusing to me. Is an ImmersiveSpace set to the .mixed (or .progressive) immersion style still a 'Full Space' (because it isn't in the shared space, with other apps)? OR, is ARKit only available in an ImmersiveSpace with the .full immersion style? Just feels like maybe 'full' is being used in two different ways here... Thanks in advance, -pj

I am using the room plan api to implement the function of multiple space merging, but I found that after performing multiple space merging, the generated json would miss some of the newly added areas, but the usd file and plist file were complete.Does anyone have this problem? Look forward to official support this is my code: public func mergeScan(_ data:String,_ scanName:String,_ directoryName:String){ var capturedRoomArray: [CapturedRoom] = [] //解析主结构 let jsonURL = getRootURL().appending(path: "/\(directoryName)/\(scanName)/scan.json") guard let mainStructureRoom = try?loadCapturedRoom(from: jsonURL) else { return } capturedRoomArray.append(mainStructureRoom) // 添加子结构 if let subStructureRoom = try? loadCapturedRoom(from: data) { os_log("loadCapturedRoom string data success: %@", type: .error, String(describing: data)) capturedRoomArray.append(subStructureRoom) } os_log("merge scan capturedRoomArray: %@", type: .error, String(describing: capturedRoomArray.count)) //合并 Task { do { finalStructureResults = try await structureBuilder.capturedStructure(from: capturedRoomArray) }catch { print("Merging Error:\(error.localizedDescription)") return } do{ //保存 //导出json guard let finalStructureResults else { return } try exportJson(from: finalStructureResults, to: jsonURL) //导出usd let meshDestinationURL = jsonURL.deletingPathExtension().appendingPathExtension("usdz") //导出plist let metadataDestinationURL = jsonURL.deletingPathExtension().appendingPathExtension("plist") try finalStructureResults.export(to: meshDestinationURL, metadataURL: metadataDestinationURL, exportOptions: [.mesh]) } catch { print("Merge Error:\(error.localizedDescription)") return } } } func exportJson(from capturedStructure: CapturedStructure, to url: URL) throws { let encoder = JSONEncoder() encoder.outputFormatting = [.prettyPrinted, .sortedKeys] let data = try encoder.encode(capturedStructure) try data.write(to: url) } Note: Only json is missing the content of this or the next scan, usdz and plist are complete

Hello, I want to be able to tap on a previously-placed ModelEntity box and add a dot or a text at that location on the box (kind of like I'm adding an annotation on the box) I have something like this, but not sure how I should do it correctly: class MyARView: ARView { // ... private func didTap(_ gestureRecognizer: UITapGestureRecognizer) { let pos = gestureRecognizer.location(in: self) if !didPlaceCube { placeCube(pos) return } let hitTestResult = self.hitTest(pos) guard let firstResult = hitTestResult.first else { return} let entity = firstResult.entity let textEntity = ModelEntity(mesh: .generateText("Hello there", extrusionDepth: 0.4, font: .boldSystemFont(ofSize: 0.05), containerFrame: .zero, alignment: .center, lineBreakMode: .byWordWrapping)) textEntity.setPosition(entity.position + firstResult.position, relativeTo: entity) entity.addChild(textEntity) } // ... }

I have a main app window that presents an Immersive style in Mixed Reality. I am trying to determine the anchor/position of this glass window in the 3D space and place a Sphere entity right next to it. The goal is to ensure that if the user moves the window, the Sphere entity remains attached to it. Does anyone have insights on how to achieve this? The below code snippet provides the position of the device, and I have positioned it 0.5 meters away from the z-axis. However, my objective is to obtain the position of the glass window and anchor the sphere to it. Any guidance on achieving this would be appreciated. import RealityKit import RealityKitContent import ARKit struct ImmersiveView: View { let visionProPose = VisionProPose() var body: some View { RealityView { content in Task { await visionProPose.runArSession() } // Add the initial RealityKit content if let scene = try? await Entity(named: "Immersive", in: realityKitContentBundle) { content.add(scene) } } update: { content in if let scene = content.entities.first { if let sphere = scene.findEntity(named: "Sphere") as? ModelEntity { Task { let transfrom = await visionProPose.getTransform() sphere.position = [Float((transfrom?.columns.3.x)!), Float((transfrom?.columns.3.y)!), Float((transfrom?.columns.3.z)!) - 1 ] } } } } } } @Observable class VisionProPose { let session = ARKitSession() let worldTracking = WorldTrackingProvider() func runArSession() async { Task { try? await session.run([worldTracking]) } } func getTransform() async -> simd_float4x4? { guard let deviceAnchor = worldTracking.queryDeviceAnchor(atTimestamp: 1) else { return nil } let transform = deviceAnchor.originFromAnchorTransform return transform } }