I have multiple images that at various times I need to replace a target image for a SKSpriteNode. Each of these multiple images has a different size. The target SKSpriteNode has a fixed frame that I want to stay fixed. This target is created via: myTarget = SKSpriteNode(imageNamed: “target”) myTarget.size = CGSize(…) myTarget.physicsBody = SKPhysicsBody(rectangleOf: myTarget.size) How do I resize each of the multiple images so that each fills up the target frame (expand or contract)? Pretend the target is a shoebox and each image is a balloon that expands or contracts to fill the shoebox. I have tried the following that fails, that is, it changes the size of the target to fit the new image .. in short, it does the exact opposite of what I want. let newTexture = SKTexture(imageNamed: newImage) let changeImgAction = SKAction.setTexture(newTexture, resize: true) myTarget.run(changeImgAction) Again, keep frame of myTarget fixed and change size of newTexture to fit the above frame ..

With AVFoundation's builtInLiDARDepthCamera, if I save photo.fileDataRepresentation to heic, it only has Exif and TIFF metadata. But, RealityKit's object capture's heic image has not only Exif and TIFF, but also has HEIC metadata including camera calibration data. What should I do for AVFoundation's exported image has same meta data?

After the iOS 17 update, objects rendered in SceneKit that have both a normal map and morph targets do not render correctly. The shading and lighting appear dark and without reflections. Using a normal map without morph targets or having morph targets on an object without using a normal map works fine. However, the combination of using both breaks the rendering. Using diffuse, normal map and a morpher: Diffuse and normal, NO morpher:

hello, when I do Metal drawing into an MTKView in full screen, there's an issue with frame scheduling, it seems. There is visible stutter, and the Metal HUD shows the frame rate jittering about. happens in Ventura and Sonoma b2 on my Macbook Pro. here's a really minimal example. Not even actively drawing anything, just presenting the drawable. #import "ViewController.h" #import <Metal/Metal.h> #import <MetalKit/MetalKit.h> @interface ViewController() <MTKViewDelegate> @property (nonatomic, weak) IBOutlet MTKView *mtkView; @property (nonatomic) id<MTLDevice> device; @property (nonatomic) id<MTLCommandQueue> commandQueue; @end @implementation ViewController - (void)viewDidLoad { [super viewDidLoad]; _device = MTLCreateSystemDefaultDevice(); _mtkView.device = _device; _mtkView.delegate = self; _commandQueue = [_device newCommandQueue]; } - (void)drawInMTKView:(MTKView *)view { MTLRenderPassDescriptor *viewRPD = view.currentRenderPassDescriptor; if(viewRPD) { id<MTLCommandBuffer> commandBuffer = [_commandQueue commandBuffer]; [commandBuffer presentDrawable:view.currentDrawable]; [commandBuffer commit]; } } - (void)mtkView:(MTKView *)view drawableSizeWillChange:(CGSize)size { NSLog(@"%@", NSStringFromSize(size)); } Looks like there's some collision between display and render timer, or something. what gives? I would like to be able to render stutter free on this very nice machine? how would I go about that?

I am processing CVPixelBuffers received from camera using both Metal and CoreImage, and comparing the performance. The only processing that is done is taking a source pixel buffer and applying crop & affine transforms, and saving the result to another pixel buffer. What I do notice is CPU usage is as high a 50% when using CoreImage and only 20% when using Metal. The profiler shows most of the time spent is in CIContext render: let cropRect = AVMakeRect(aspectRatio: CGSize(width: dstWidth, height: dstHeight), insideRect: srcImage.extent) var dstImage = srcImage.cropped(to: cropRect) let translationTransform = CGAffineTransform(translationX: -cropRect.minX, y: -cropRect.minY) var transform = CGAffineTransform.identity transform = transform.concatenating(CGAffineTransform(translationX: -(dstImage.extent.origin.x + dstImage.extent.width/2), y: -(dstImage.extent.origin.y + dstImage.extent.height/2))) transform = transform.concatenating(translationTransform) transform = transform.concatenating(CGAffineTransform(translationX: (dstImage.extent.origin.x + dstImage.extent.width/2), y: (dstImage.extent.origin.y + dstImage.extent.height/2))) dstImage = dstImage.transformed(by: translationTransform) let scale = max(dstWidth/(dstImage.extent.width), CGFloat(dstHeight/dstImage.extent.height)) let scalingTransform = CGAffineTransform(scaleX: scale, y: scale) transform = CGAffineTransform.identity transform = transform.concatenating(scalingTransform) dstImage = dstImage.transformed(by: transform) if flipVertical { dstImage = dstImage.transformed(by: CGAffineTransform(scaleX: 1, y: -1)) dstImage = dstImage.transformed(by: CGAffineTransform(translationX: 0, y: dstImage.extent.size.height)) } if flipHorizontal { dstImage = dstImage.transformed(by: CGAffineTransform(scaleX: -1, y: 1)) dstImage = dstImage.transformed(by: CGAffineTransform(translationX: dstImage.extent.size.width, y: 0)) } var dstBounds = CGRect.zero dstBounds.size = dstImage.extent.size _ciContext.render(dstImage, to: dstPixelBuffer!, bounds: dstImage.extent, colorSpace: srcImage.colorSpace ) Here is how CIContext was created: _ciContext = CIContext(mtlDevice: MTLCreateSystemDefaultDevice()!, options: [CIContextOption.cacheIntermediates: false]) I want to know if I am doing anything wrong and what could be done to lower CPU usage in CoreImage?

I am working on a fully immersive RealityView for visionOS and I need to add light from the sun to my scene. I see that DirectionalLight, PointLight, and SpotLight are not available on visionOS. Does anyone know how to add light to fully immersive scene on visionOS? My scene is really dark right now without any additional light.

I have a RealityView in my visionOS app. I can't figure out how to access RealityRenderer. According to the documentation (https://developer.apple.com/documentation/realitykit/realityrenderer) it is available on visionOS, but I can't figure out how to access it for my RealityView. It is probably something obvious, but after reading through the documentation for RealityView, Entities, and Components, I can't find it.

What is the most efficient way to use a MTLTexture (created procedurally at run-time) as a RealityKit TextureResource? I update the MTLTexture per-frame using regular Metal rendering, so it’s not something I can do offline. Is there a way to wrap it without doing a copy? A specific example would be great. Thank you!

has anyone gotten their 3d Models to render in seperate windows, i tried following the code in the video for creating a seperate window group, but i get a ton of obsecure errors, i was able to get it to render in my 2d windows, but when i try making a seperate window group i get errors

Hi, I'm creating a SF Symbols image like this: var img = UIImage(systemName: "x.circle" ,withConfiguration: symbolConfig)!.withTintColor(.red) In the debugger the image is really red. and I'm using this image to create a SKTexture: let shuffleTexture = SKTexture(image: img) The texture image is ALWAYS black and I have no idea how to change it's color. Nothing I've tried so far works. Any ideas how to solve this? Thank you! Best Regards, Frank

Hi, I have an app based on Metal and it runs on VisionOS. It has a huge sphere mesh and renders video outputs (from AVPlayer) on it. What I want to do is rendering left portion of my video output on left eye, and right portion of my video output on right eye. In my fragment shader, I think I need to know that the thread in shader is for left eye or right eye. (I'm not using MV-hevc encoded video but just hevc encoded one) So, what I currently do is, I assume 'amplification_id' is for the thing which determines the side of eyes. But, I'm not sure this is correct.

I know opengl is marked as deprecated since ios12 but I have an old project using it and I want to update some feature of it then release the update version. So I'm wondering if I can still release an app using opengl to app store currently? (I know it's better to shift to MetalKit but for some reason I want to cut the cost if I can. )

So I can observe RealityKit Components by using the new @Observable or using ObservableObject, both of these require my Component to be a class instead of a struct though. I've read that making a Component a class is a bad idea, is this correct? Is there any other way to observe values of an entities' components?

I have some strange behavior in my app. When I set the position to .zero you can see the sphere normally. But when I change it to any number it doesn't matter which and how small. The Sphere isn't visible or in the view. The RealityView import SwiftUI import RealityKit import RealityKitContent struct TheSphereOfDoomRV: View { @StateObject var viewModel: SphereViewModel = SphereViewModel() let sphere = SphereEntity(radius: 0.25, materials: [SimpleMaterial(color: .red, isMetallic: true)], name: "TheSphere") var body: some View { RealityView { content, attachments in content.add(sphere) } update: { content, attachments in sphere.scale = SIMD3<Float>(x: viewModel.scale, y: viewModel.scale, z: viewModel.scale) } attachments: { VStack { Text("The Sphere of Doom is one of the most powerful Objects. You can interact with him in every way you can imagine ").multilineTextAlignment(.center) Button { } label: { Text("Play Video!") } }.tag("description") }.modifier(GestureModifier()).environmentObject(viewModel) } } SphereEntity: import Foundation import RealityKit import RealityKitContent class SphereEntity: Entity { private let sphere: ModelEntity @MainActor required init() { sphere = ModelEntity() super.init() } init(radius: Float, materials: [Material], name: String) { sphere = ModelEntity(mesh: .generateSphere(radius: radius), materials: materials) sphere.generateCollisionShapes(recursive: false) sphere.components.set(InputTargetComponent()) sphere.components.set(HoverEffectComponent()) sphere.components.set(CollisionComponent(shapes: [.generateSphere(radius: radius)])) sphere.name = name super.init() self.addChild(sphere) self.position = .zero // .init(x: Float, y: Float, z: Float) and [Float, Float, Float] doesn't work ... } }

We're getting some strange rendering crashes on various devices running both ios16 and ios17 beta. The problems all appear when compiling with any of the Xcode 15 betas, including 8. This code has worked fine for years. The clearest error we get is on the iPhone X and XR where newRenderPipelineStateWithDescriptor returns: "Inlining all functions due to use of indirect argument bufferbuffer(15): Unable to map argument buffer access to resource" Buffer 15 is where we stash our textures and looks like this: typedef struct RegularTextures { // A bit per texture that's present uint32_t texPresent [[ id(WKSTexBufTexPresent) ]]; // Texture indirection (for accessing sub-textures) const metal::array<float, 2*WKSTextureMax> offset [[ id(WKSTexBuffIndirectOffset) ]]; const metal::array<float, 2*WKSTextureMax> scale [[ id(WKSTexBuffIndirectScale) ]]; const metal::array<metal::texture2d<float, metal::access::sample>, WKSTextureMax> tex [[ id(WKSTexBuffTextures) ]]; } RegularTextures; The program we're trying to set up looks like this: vertex ProjVertexTriB vertexTri_multiTex( VertexTriB vert [[stage_in]], constant Uniforms &uniforms [[ buffer(WKSVertUniformArgBuffer) ]], constant Lighting &lighting [[ buffer(WKSVertLightingArgBuffer) ]], constant VertexTriArgBufferB & vertArgs [[buffer(WKSVertexArgBuffer)]], constant RegularTextures & texArgs [[buffer(WKSVertTextureArgBuffer)]]) { // Do things } Fairly benign as these things go. Even more curiously, a different program with the same RegularTextures argument buffer is sometimes set up first without complaint. I strongly suspect Apple introduced a bug here, but with the impending release, we're just trying to figure out how to work around it.

I am attempting to place images in wall anchors and be able to move their position using drag gestures. This seem pretty straightforward if the wall anchor is facing you when you start the app. But, if you place an image on a wall anchor to the left or the wall behind the original position then the logic stops working properly. The problem seems to be the anchor and the drag.location3D orientations don't coincide once you are dealing with wall anchors that are not facing the original user position (Using Xcode Beta 8) Question: How do I apply dragging gestures to an image regardless where the wall anchor is located at in relation to the user original facing direction? Using the following code: var dragGesture: some Gesture { DragGesture(minimumDistance: 0) .targetedToAnyEntity() .onChanged { value in let entity = value.entity let convertedPos = value.convert(value.location3D, from: .local, to: entity.parent!) * 0.1 entity.position = SIMD3<Float>(x: convertedPos.x, y: 0, z: convertedPos.y * (-1)) } }

I have generated a box in RealityKit with splitFaces property set to true to allow different materials on each cube side. Applying different SimpleMaterials (e.g. with different colors) works fine on Vision Pro simulator. But combining VideoMaterial and SimpleMaterial does not work. BTW: a 6x video cube can be rendered successfully so the problem seems to be mixing material structures. Here's my relevant code snippet: let mesh = MeshResource.generateBox(width: 0.3, height: 0.3, depth: 0.3, splitFaces: true) let mat1 = VideoMaterial(avPlayer: player) let mat2 = SimpleMaterial(color: .blue, isMetallic: true) let mat3 = SimpleMaterial(color: .red, isMetallic: true) let cube = ModelEntity(mesh: mesh, materials: [mat1, mat2, mat3, mat1, mat2, mat3]) In detail, the video textures are shown whereas the simple surfaces are invisible. Is this a problem of Vision Pro simulator? Or is it not possible to combine different material structures on a box? Any help is welcome!

I'm trying to animate a shape (e.g. a circle) to follow a custom path, and struggling to find the best way of doing this. I've had a look at the animation options from SwiftUI, UIKit and SpriteKit and all seem very limited in what paths you can provide. Given the complexity of my path, I was hoping there'd be a way of providing a set of coordinates in some input file and have the shape follow that, but maybe that's too ambitious. I was wondering if this were even possible, and assuming not, if there were other options I could consider.

Hello, I've been working on an app that involves training a neural network model on the iPhone. I've been using the Metal Performance Shaders Graph (MPS Graph) for this purpose. In the training process the loss becomes Nan on iOS17 (21A329). I noticed that the official sample code for Training a Neural Network using MPS Graph (link) works perfectly fine on Xcode 14.3.1 with iOS 16.6.1. However, when I run the same code on Xcode 15.0 beta 8 with iOS 17.0 (21A329), the training process produces a NaN loss in function updateProgressCubeAndLoss. The official sample code and my own app exhibit the same issue. Has anyone else experienced this issue? Is this a known bug, or is there something specific that needs to be adjusted for iOS 17? Any guidance would be greatly appreciated. Thank you!

Xcode complains that Model3D is not in scope despite that RealityKit is imported. I have tried with Xcode 15 & beta 8 in vain. How can I resolve this issue? import SwiftUI import RealityKit struct ContentView: View { var body: some View { Model3D(named: "Robot-Drummer") { model in model .resizable() .aspectRatio(contentMode: .fit) } placeholder: { ProgressView() } } }