Discuss spatial computing on Apple platforms and how to design and build an entirely new universe of apps and games for Apple Vision Pro.

All subtopics

Post

Replies

Boosts

Views

Activity

[ARKit, Reality Composer Pro] Is it possible loading Immersive Scene after recognizing preregistered images?
Hi! I wanna know that if it's possible that loading Immersive Scene after scanning(recognizing) preregistered images or objects? I tried to load the Immersive scene after scanning image and objects, it didn't work well. Please let me know about the solution if it's possible. Here the ImmersiveView.swift code i tried. // ImmersiveView.swift import SwiftUI import RealityKit import RealityKitContent // Using the RealityKitContent module struct ImmersiveView: View { @ObservedObject var viewModel: TrackingViewModel @State private var immersiveScene: Entity? @State private var isToggleOn: Bool = false // Variable for toggle state var body: some View { ZStack { // Overlay RealityView and UI elements RealityView { content in if let scene = immersiveScene { content.add(scene) print("Immersive scene successfully added.") if let moneyGunsEntity = scene.findEntity(named: "MoneyGuns") { NotificationCenter.default.post( name: Notification.Name("RealityKit.NotificationTrigger"), object: nil, userInfo: [ "RealityKit.NotificationTrigger.Scene": scene, "RealityKit.NotificationTrigger.Identifier": "PlayTimeline" ] ) print("PlayTimeline notification sent.") } else { print("MoneyGuns entity not found.") } } } .onAppear { Task { if let scene = try? await Entity(named: "Immersive", in: realityKitContentBundle) { immersiveScene = scene } else { print("Failed to load immersive scene.") } } } VStack { Spacer() Toggle(isOn: $isToggleOn) { // Add toggle button Text("Toggle Option") .foregroundColor(.white) } .padding() .background(Color.black.opacity(0.7)) .cornerRadius(8) .padding() } } } }
1
0
264
2w
RoomPlan Framework v2 - Stairs missing
Hello Community, I'm encountering an issue with the latest iOS 17 update, specifically related to RoomPlan version-2. In iOS 16, when using RoomPlan version-1, we were able to display stairs in our app. However, after upgrading to iOS 17 and implementing RoomPlan version-2, the stairs are no longer visible. Despite thorough investigation, I couldn't find any option within the code to show or hide stairs, or any other objects for that matter. It seems like a specific issue with the update rather than a coding error on our part. Has anyone else encountered a similar problem? If so, I would greatly appreciate any insights or solutions you might have. It's crucial for our app functionality to have stairs displayed accurately, and we're currently at a loss on how to address this issue. Thank you in advance for any assistance you can provide. Best regards
2
1
533
Mar ’24
How to present an alert in visionOS immersive space?
My visionOS app uses an immersive view. If the app encounters an error, I want to present an alert. I tried in a demo app to present such an alert, but it is not shown. Nearly the same code on iOS presents an alert window. Here is my demo code, based on Apple's Immersive Environment App template: import SwiftUI import RealityKit import RealityKitContent struct ErrorInfo: LocalizedError, Equatable { var errorDescription: String? var failureReason: String? } struct ImmersiveView: View { @State private var presentAlert = false let error = ErrorInfo( errorDescription: "My error", failureReason: "No reason" ) var body: some View { RealityView { content, attachments in let mesh = MeshResource.generateBox(width: 1.0, height: 0.05, depth: 1.0) var material = UnlitMaterial() material.color.tint = .red let boardEntity = ModelEntity(mesh: mesh, materials: [material]) boardEntity.transform.translation = [0, 0, -3] content.add(boardEntity) } update: { content, attachments in // … } attachments: { // … } .onAppear { presentAlert = true } .alert( isPresented: $presentAlert, error: error, actions: { error in }, message: { error in Text(error.failureReason!) }) } } Since I cannot see any alert, is something wrong with my code? How should an alert be presented in immersive space?
1
0
181
2w
[ARKit] Is it possible remembering certain room using Room Tracking?
Hi! I'm making content using Room Tracking for vision pro these days. So I searched information about it. Here the links I visited. But I could not found the info I wanted to know Apple ARKit Create enhanced spatial computing experiences with ARKit RoomTrackingProvider I wanna know that if it's possible remembering room structure that recognized before and adding contents in certain world anchor in the room space when user entered the room again? For example, a developer can save the room structure, room info (with room ID) and world anchor of the room with Room Tracking feature. After this, the developer can add entities via Xcode and Reality Composer Pro in certain position of the room to show contents to users when users enter the room. So users can see the contents whenever they visit the room. Is this possible? If there are example codes or projects about it, please let me know.
1
0
280
2w
Detecting collisions between fingertip and world mesh
I'm using hand tracking to detect collisions between fingertips and entities that I have placed in the scene. I'm using the .mixed environment. However, I want to detect when a fingertip touches a real-world object such as a wall. No matter what I try, I can't get the collision to fire. I'm using the SceneReconstructionProvider to give me world meshes, which I use to create ModelEntity objects to which I add a CollisionComponent with the shape of the object. I can render the meshes just fine, but nothing I do seems to allow collisions to work. Surely this is possible, what am I missing?
4
0
197
3w
visionOS Simulator Rotate and Scale gestures difficult to register (capture)
We were having an issue wrb the system rotate and scale gestures (two-handed gestures / RotateGesture3D and MagnifyGesture) were extremely difficult to register (make work) in the visionOS simulator. The solution we found was to: Launch your app in the simulator Move the pointer on top of the 3D object for which you are testing rotation and scaling gestures. Press and hold the Option key to display touch points (ie: the two-handed gesture points). While maintaining the option key pressed, release the pointer and re-enable it again. I am using a track pad with tap-to-click enabled and three-finger to drag enabled in accessibility, so "release the pointer and re-enable it again" translates simply to removing the three finger and placing them again on the trackpad. If you have maintained the option key pressed, then you should now be able to rotate and scale the 3D object. Context if you are interested: Our issue was also occurring in Apple's own sample project relating to gestures "Transforming RealityKit entities using gestures", at below link. On Apple's article "Interacting with your app in the visionOS simulator" at the below link, for two-handed gestures it states "Press and hold the Option key to display touch points. Move the pointer while pressing the Option key to change the distance between the touch points. Move the pointer and hold the Shift and Option keys to reposition the touch points." This simply did not work anymore for rotation and scaling gestures. These gestures used to be a lot more responsive in Sonoma. Either the article should be updated to what I described above, or there is an issue. Our colleague who is using macOS Sonoma 14.6.1 with the latest release of Xcode is not having these issues. Here is the list of configurations (troubleshooting we tried!) where it is difficult to achieve rotation and scaling gestures in the visionOS simulator: macOS Sequoia 16.1 Beta, Xcode 16.1 RC w visionOS 2.1 macOS Sequoia 16.1 Beta, Xcode 16.1 RC w visionOS 2.0 macOS Sequoia 16.1 Beta, Xcode 16.2 Beta 1 w visionOS 2.1 macOS Sequoia 16.1 Beta, Xcode 16.2 Beta 1 w visionOS 2.0 macOS Sequoia 16.1 Beta, remove all Xcodes and installed the build from AppStore (Xcode 16.1) macOS Sequoia 16.1 Beta, Xcode 16.0 w visionOS 2.0 completely wiped out, and reset entire development machine, re-installed latest releases of sequoia (15.1) and xcode (15.1)) Throughout these troubleshooting I often: restarted both xcode and sim erased all derived data erased all contents and settings from sims performed fresh git clones None of the above worked, only the workaround described above works atm. As you can maybe deduce, it was very time consuming to find the workaround, we also wasted some development effort thinking our gesture development was no-good. Hopefully this will help other devs. Article Link: https://developer.apple.com/documentation/xcode/interacting-with-your-app-in-the-visionos-simulator Gesture sample project link: https://developer.apple.com/documentation/realitykit/transforming-realitykit-entities-with-gestures
2
0
323
2w
dismissWindow() doesn't dismantle View
We have discovered that our UIViewRepresentable view isn't being dismantled after its window is dismissed via dismissWindow(). This seems to result in a leak of our custom Coordinator class. Every time the user opens a new window, a new Coordinator is created; if the user then dismisses the window manually, or we dismiss it programmatically, the Coordinator remains in memory with no way to destroy it. Is this expected behavior? How can we be sure to clean up our Coordinator when the view's window is closed? Thanks.
0
0
145
2w
Geometry recognition and measurement from MeshAnchor
FYI. The source code of the FindSurface demo app for Apple Vision Pro (visionOS) is available now. The Swift package of FindSurface™ library is required to build the source code into the demo app. https://github.com/CurvSurf/FindSurface-visionOS After starting the app, the floating panels (below) will appear on your right side, and you will see wireframe meshes that approximately describe your environments. Performing a spatial tap (pinching with your thumb and index finger) with staring at a location on the meshes will invoke FindSurface, with an indicator (blue disk) appearing on the surface you've gazed. Voice commands: “Tap” – Spatial tap (gazing & pinching). Invoke FindSurface. “Tap plane” – Plane selection. “Tap sphere” or “Tap ball” – Sphere selection. “Tap cylinder” – Cylinder selection. “Tap cone” – Cone selection. “Tap torus” or “Tap donut” – Torus selection. “Tap accuracy” or “Tap measurement accuracy” – Accuracy selection. “Tap mean distance”, “Tap average distance”, or “Tap distance” – Avg. Distance selection. “Tap touch radius” or “Tap seed radius” – Touch Radius selection. “Tap Inlier” – “Show inlier points” toggle. “Tap outline” – “Show geometry outline” toggle. “Tap clear” – “Clear Scene” click.
7
0
884
Jun ’24
Object Occlusion in Non LiDAR devices
Hi, I'm currently working on an ARKit project where I need to implement object occlusion on devices that do not have a LiDAR sensor (e.g., iPhone XR, iPhone 11). I used CoreML models like DepthAnythingV2 to create depth maps and DETRResnet50SemanticSegmentationF16P8 to to perform real-time segmentation. But these models are too heavy for devices. Much appreciated on any advice or pointers to resources.
2
0
307
3w
VisionOS 2.0 App Crashes Faster with Each Upload from XCode
We developed an app for VisionOS 2.0 Beta in XCode 16 Beta. The development was done in the beta versions since we needed features for our app which were not available in VisionOS 1.0, which was the most recent stable release at the time we developed the app. The app was fully functional on our AVP running VisionOS 2.0 Beta Version 5, and we never had any errors. We did not publish the app to the app store since we are a research lab using the app for teleoperating a custom robot. Last week, we upgraded the AVP from VisionOS 2.0 Beta Version 5 to VisionOS 2.0 (stable release). Unfortunately, once we upgraded to 2.0, we began to have an issue with the app. While the app is running, at seemingly random times, without any new functionality being used within the app (no new buttons being pressed, etc), we encounter the following console error: assertion failure: 'index < m_size' (operator[]:line 1011) Index out of range. index = 18446744073709551615, size = 0 We could re-upload the app to the AVP and successfully operate the app for several minutes until the same error occurred again. We thought to use Apple Configurator to flash VisionOS 2.0 Beta 5 to the AVP since the error wasn't happening on the previous firmware, but we were unable to flash a beta version of VisionOS via Apple Configurator, so we simply performed a factory reset of the device (on VisionOS2.0, by pressing restore in Apple Configurator with the AVP connected via the developer strap) to see if this might fix the issue. After doing a factory reset, we thought the console error completely went away. We were able to operate the app for ~3 hours on Sunday with no issues. Then, yesterday (Monday) we operated the app for another 2 hours, and at the very end of using the app, it crashed with the same error. We re-uploaded the app with XCode, and the error occurred again after about 20 mins of using the app. This cycle repeated, and every time we re-uploaded the app, the time it took for the error to occur decreased, until we uploaded the app and the error occurred in <20 seconds. We decided to test our hypothesis by upgrading VisionOS to 2.1 and using XCode 16. Similarly, we were able to run the app on the AVP for 2 hours, then the error occurred. The next time we ran the app, the error occurred within 20 minutes, then after reloading, occurred within 5 mins, then 2 mins, etc. We are pretty stumped on why the app would work after a factory reset or a firmware upgrade for hours, then fail faster and faster every time we re-upload the app from Xcode. We are not experienced in debugging Swift and ObjC, so we wanted to inquire if this is an issue that you have ran into before to point us in the right direction. We think that it could be a problem with the cached memory that persists on the device across uploads from Xcode, but that's the extent of our understanding. P.S., we also experienced this error during some of the app failures, but the one above is the most common: assertion failure: Index out of range (operatorl]:line 858) index = 576460752303423487, max = 1
3
0
219
3w
[Apple Vision Pro] Issue with startDeviceMotionUpdates in ImmersiveSpace Mode
Hello everyone, I’m developing an app for Apple Vision Pro, and I’m trying to retrieve motion data updates aligned to magnetic north by using the following method: startDeviceMotionUpdates(using: .xMagneticNorthZVertical, to: .main) { ... } The goal is to get motion data oriented to magnetic north while an ImmersiveSpace() with an immersiveStyle set to .mixed is active. However, with this setup, I receive no updates at all. If I switch to: startDeviceMotionUpdates(using: .xArbitraryZVertical, to: .main) { ... } or startDeviceMotionUpdates(to: .main) { ... } then I do receive data, but it’s not aligned as required (I specifically need .xMagneticNorthZVertical). Has anyone experienced a similar issue, or does anyone know how to enable updates aligned to magnetic north in this configuration? Thanks in advance for any insights! SDK: VisionOS 2.0
1
0
176
3w
How to Play Timeline Animations via code
Hi everyone, I need to synchronize the playback of RealityKit Timelines via SharePlay. To do this I am trying to get the references of the timelines using "AnimationPlaybackController" and "AnimationResource". In my realitykit scene I have configured both an animation (with blender), and a timeline, the animation starts correctly when the realitykit scene starts, the timeline not. Below the code: struct ContentView: View { @State private var subscriptions = [EventSubscription]() @Environment(AppModel.self) private var appModel let rootEntity = Entity() @State var testEntity: Entity? @State var testAnimation: AnimationResource? @State var testController: AnimationPlaybackController? init() { CubeComponent.registerComponent() } var body: some View { RealityView { content in content.add(rootEntity) if let scene = try? await Entity(named: "Room", in: realityKitContentBundle) { rootEntity.addChild(scene) playAnimations(from: content) } } .gesture(SpatialTapGesture().targetedToAnyEntity() .onEnded({ value in _ = value.entity.applyTapForBehaviors() if let testEntity, let testAnimation { testController = testEntity.playAnimation(testAnimation.repeat()) } }) ) } func playAnimations(from content: RealityViewContent) { subscriptions.append(content.subscribe(to: ComponentEvents.DidAdd.self, componentType: AnimationLibraryComponent.self, { event in let entity = event.entity entity.components[AnimationLibraryComponent.self]?.animations.forEach({ (key, value) in if value.definition is AnimationGroup { if key == "/Room/TestTimeline" { let controller = entity.playAnimation(value.repeat()) testEntity = entity testAnimation = value appModel.syncronizedAnimations[key] = .init(name: key, animationController: controller, entityName: entity.name) } } else { if entity.name == "SphereInteractable" { let controller = entity.playAnimation(value.repeat()) appModel.syncronizedAnimations[key] = .init(name: key, animationController: controller, entityName: entity.name) } } }) })) } } the variables testEntity, testAnimation and testController are for testing purposes only. If I try to start the animations in the playAnimations function, only the animation created via blender starts (the one related to the object "SphereInteractable"), the Timeline starts only if I save a reference and I play it with a tap gesture or with a delay of ! seconds with DispatchQueue.asyncAfter called in the onAppear. is there a better way to handle this? The goal is to have a reference of the AnimationPlaybackController of the timeline, in order to sync the animation via shareplay. Thanks
3
0
223
4w
Can we prevent Apple Vision Pro from turnning off the display and going to sleep as long as take off Apple Vision Pro?
In some cases, we will put the Apple Vision Pro back on for a short period of time after taking it off, such as less than 1 minute, and want to keep it activte when we take it off so that it can continue to work seamlessly when we put it on again. The current Apple Vision Pro turns off the display and goes to sleep whenever it is been taken off. This feature is also explained in the user guide, to save power, for safety, etc. However, we want a seamless experience! Is it possible to have a screen saver like macOS or iOS, where we can set our own delay time to go sleep? Or is there any API that can be called to prevent going to sleep?
0
0
221
2w
3D display from 3D camera
I want to pursue for a project involving 3D VR visualisation. I would like to know if there is a 3D stereoscopic camrea setup that is able to connect straight to Apple Vision Pro display yet. Of course, with no compatibility issues with MV-HEVC. Any recommendation is appreciated.
0
0
199
2w