Unable to update IBL at runtime

I seem to be running into an issue in an app I am working on were I am unable to update the IBL for entity more than once in a RealityKit scene. The app is being developed for visionOS.

I have a scene with a model the user interacts with and 360 panoramas as a skybox. These skyboxes can change based on user interaction. I have created an IBL for each of the skyboxes and was intending to swap out the ImageBasedLightComponent and ImageBasedLightReceiverComponent components when updating the skybox in the RealityView's update closure.

The first update works as expected but updating the components after that has no effect. Not sure if this is intended or if I'm just holding it wrong. Would really appreciate any guidance. Thanks

Simplified example

// Task spun up from update closure in RealityView
Task {
    if let information = currentSkybox.iblInformation, let resource = try? await EnvironmentResource(named: information.name) {
        parentEntity.components.remove(ImageBasedLightReceiverComponent.self)
        if let iblEntity = content.entities.first(where: { $0.name == "ibl" }) {
            content.remove(iblEntity)
        }

        let newIBLEntity = Entity()
        var iblComponent = ImageBasedLightComponent(source: .single(resource))
        iblComponent.inheritsRotation = true
        iblComponent.intensityExponent = information.intensity
        newIBLEntity.transform.rotation = .init(angle: currentPanorama.rotation, axis: [0, 1, 0])
        newIBLEntity.components.set(iblComponent)
        newIBLEntity.name = "ibl"
        content.add(newIBLEntity)
        parentEntity.components.set([
            ImageBasedLightReceiverComponent(imageBasedLight: newIBLEntity),
            EnvironmentLightingConfigurationComponent(environmentLightingWeight: 0),
        ])
    } else {
        parentEntity.components.remove(ImageBasedLightReceiverComponent.self)
    }
}