Multi-scene/window shareplay on visionOS

Hi all, been working with visionOS for a bit so far and am trying to develop a feature that allows users to shareplay and interact with a 3D model pulled from the cloud (icloud in this case, but may use a regular backend service in the future). Ideally, I would want to be able to click a custom button on a regular window that starts the group activity/shareplay with another person in the facetime call and opens the volumetric window with the model and can switch to an immersive space freely. TLDR/questions at the very end for reference point.

I was able to get this working when only working with a single window group (i.e. a volumetric window group scene and an immersive space scene). However I am running into trouble getting shareplay to correctly grab the desired scene (or any scene at all) when I have multiple window group scenes defined.

I have been referencing the following documentation in my attempts to implement this:

No luck so far however. Additionally, here is a quick breakdown of what I've done so far to attempt implementation:

  1. Define group activity that contains static var activityIdentifier: String and var metadata: GroupActivityMetadata as well as conforms to GroupActivity.

  2. Provide methods to start shareplay via a startShareplay() method that instantiates the above group activity and switch awaits on activity.prepareForActivation() to activate the activity if case .activationPreferred. I have also provided a separate group activity registration method to start shareplay via airdrop as mentioned in the Building spatial SharePlay experiences developer video (timestamped), which does expose a group activity to the share context menu/ornament but does not indicate being shared afterwards.

  3. On app start, trigger a method to configure group sessions and provide listeners for sessions (including subscribers for active participants, session state, messages of the corresponding state type - in my case it's ModelState.self, journal attachments for potentially providing models that the other user may not have as we are getting models from cloud/backend, local participant states, etc). At the very end, call groupSession.join().

  4. Add external activation handlers to the corresponding scenes in the scene declaration (as per this documentation on SceneAssociationBehavior using the handlesExternalEvents(matching:) scene modifier to open the scene when shareplay starts). I have also attempted using the view modifier handlesExternalEvents(preferring:allowing:) on views but also no luck. Both are being used with the corresponding activityIdentifier from the group activity and I've also tried passing a specific identifier while using the .content(_) sceneAssociationBehavior as well but no luck there either.

I have noted that in this answer regarding shareplay in visionOS, the VP engineer notes that when the app receives the session, it should setup any necessary UI then join the session, but I would expect even if the UI isn't being set up via the other person's session that the person who started shareplay should still see the sharing ornament turn green on the corresponding window which doesn't seem to occur. In fact, none of the windows that are open even get the green sharing ornament (just keeps showing "Not shared").

TLDR: Added external events handling and standard group activity stuff to multi-window app. When using shareplay, no windows are indicated as being shared.

My questions thus are:

  1. Am I incorrect in my usage of the scene/view modifiers for handlesExternalEvents to open and associate a specific windowgroup/scene with the group activity?

  2. In regards to opening a specific window when the group activity is activated, how do we pass any values if the window group requires it? i.e. if it's something like WindowGroup(id: ..., for: URL.self) { url in ... }

  3. Do I still need to provide UI setup in the session listener (for await session in MyActivity.sessions())? Is this just a simple openWindow?

  4. Past the initializing shareplay stuff above, what are the best practices for sharing 3d models that not all users in the session might have? Is it adding it as an attachment to GroupSessionJournal? Or should I pass the remote URL to everyone to download the model locally instead?

Thanks for any help and apologies for the long post. Please let me know if there's any additional information I can provide to help resolve this.

Answered by Vision Pro Engineer in 790616022

Thanks for the detailed question. You're doing the right thing and hitting a couple known limitations.

Do I still need to provide UI setup in the session listener (for await session in MyActivity.sessions())? Is this just a simple openWindow?

  • Scene preference is only applied to windows opened before the session is received. Simply put, all participants must have the window open before the initiator clicks the "Start [Activity]" button (that activates the activity) for SharePlay to associate them. It appears you are opening the recipient window after the session is received, but before the session is joined. Note: received is distinct from joined.

  • An app's initial window style (defined in the plist as Preferred Default Scene Session Role) needs to match the window style of the window you wish to SharePlay (in this case volumetric) for window association to work when the app is initially open (via a SharePlay link).

You can work around the limitations by adjusting the content of the initial volumetric window based on the presence of a session and the session's activity's content.

More details

Define an app whose initial window is volumetric.

struct SceneAssociationApp: App {
    var appModel:AppModel = AppModel()
    
    var body: some SwiftUI.Scene {
        WindowGroup {
            ContentView().environment(appModel)
            
        }
        .windowStyle(.volumetric)
    }
}

Define a model to hold shared state related to the SharePlay activity.

@Observable
class GroupActivitiesModel {
    var session:GroupSession<TestActivity>?
}

Define an activity with the data you need to load the desired model when a session is received. In this case I've used modelId.

struct TestActivity: GroupActivity, Transferable, Sendable {
    static let activityIdentifier = "com.yourdomain.TestActivity"

    let modelId:String 
    var metadata: GroupActivityMetadata = {
        var metadata = GroupActivityMetadata()
        metadata.type = .generic
        metadata.title = "Model sharing app"
        metadata.sceneAssociationBehavior = .content(TestActivity.activityIdentifier)
        
        return metadata
    }()
}

This is the heart of the work around. Define a view that displays HomeView when there is no session and displays ModelView when there is one.

struct ContentView: View {
    @State var groupActivitiesModel = GroupActivitiesModel()
    
    var body: some View {

        if let session = groupActivitiesModel.session {
            ModelView(modelId: session.activity.modelId)
        }
        else { 
            HomeView(groupActivitiesModel: groupActivitiesModel)
        }
    }
}

Define ModelView to display the model for the modelId associated with the incoming session.

struct ModelView: View {
    @State var modelId:String

    var body: some View {
        
        RealityView { content in
            let mesh = MeshResource.generateText(modelId,
                                                 extrusionDepth: 0.01,
                                                 font: UIFont.systemFont(ofSize: 0.07))
            let textEntity = ModelEntity(mesh: mesh, materials: [SimpleMaterial(color: .systemPink, isMetallic: true)])
            textEntity.position.x = -mesh.bounds.extents.x / 2
            content.add(textEntity)
        }
    }
}

Finally define HomeView to present the user with a button to start the activity while listening for incomming sessions. Note the use of the frame and glassBackgroundEffect modifiers on HomeView; both of those help the window “feel" more like a traditional 2D home view. That said, consider taking advantage of the volumetric window to enhance the home view with 3D content.

struct HomeView: View {
    @State var groupActivitiesModel = GroupActivitiesModel()
    
    var body: some View {
        
        VStack {
            Text("Welcome to model share").font(.headline)
            
            if groupActivitiesModel.session == nil {
                Button("Share model", systemImage: "shareplay") {
                    Task {
                        let _ = try? await TestActivity(modelId: "some id").activate()
                    }
                }
            }
        }
        .task {
            for await session in TestActivity.sessions() {
                groupActivitiesModel.session = session
                session.join()
            }
        }
        .frame(width: 800, height: 600)
        .glassBackgroundEffect()
    }
}

Once a session is established you can use GroupSessionMessenger to send and receive messages containing the data needed to update the model(s) displayed in the volume.

Accepted Answer

Thanks for the detailed question. You're doing the right thing and hitting a couple known limitations.

Do I still need to provide UI setup in the session listener (for await session in MyActivity.sessions())? Is this just a simple openWindow?

  • Scene preference is only applied to windows opened before the session is received. Simply put, all participants must have the window open before the initiator clicks the "Start [Activity]" button (that activates the activity) for SharePlay to associate them. It appears you are opening the recipient window after the session is received, but before the session is joined. Note: received is distinct from joined.

  • An app's initial window style (defined in the plist as Preferred Default Scene Session Role) needs to match the window style of the window you wish to SharePlay (in this case volumetric) for window association to work when the app is initially open (via a SharePlay link).

You can work around the limitations by adjusting the content of the initial volumetric window based on the presence of a session and the session's activity's content.

More details

Define an app whose initial window is volumetric.

struct SceneAssociationApp: App {
    var appModel:AppModel = AppModel()
    
    var body: some SwiftUI.Scene {
        WindowGroup {
            ContentView().environment(appModel)
            
        }
        .windowStyle(.volumetric)
    }
}

Define a model to hold shared state related to the SharePlay activity.

@Observable
class GroupActivitiesModel {
    var session:GroupSession<TestActivity>?
}

Define an activity with the data you need to load the desired model when a session is received. In this case I've used modelId.

struct TestActivity: GroupActivity, Transferable, Sendable {
    static let activityIdentifier = "com.yourdomain.TestActivity"

    let modelId:String 
    var metadata: GroupActivityMetadata = {
        var metadata = GroupActivityMetadata()
        metadata.type = .generic
        metadata.title = "Model sharing app"
        metadata.sceneAssociationBehavior = .content(TestActivity.activityIdentifier)
        
        return metadata
    }()
}

This is the heart of the work around. Define a view that displays HomeView when there is no session and displays ModelView when there is one.

struct ContentView: View {
    @State var groupActivitiesModel = GroupActivitiesModel()
    
    var body: some View {

        if let session = groupActivitiesModel.session {
            ModelView(modelId: session.activity.modelId)
        }
        else { 
            HomeView(groupActivitiesModel: groupActivitiesModel)
        }
    }
}

Define ModelView to display the model for the modelId associated with the incoming session.

struct ModelView: View {
    @State var modelId:String

    var body: some View {
        
        RealityView { content in
            let mesh = MeshResource.generateText(modelId,
                                                 extrusionDepth: 0.01,
                                                 font: UIFont.systemFont(ofSize: 0.07))
            let textEntity = ModelEntity(mesh: mesh, materials: [SimpleMaterial(color: .systemPink, isMetallic: true)])
            textEntity.position.x = -mesh.bounds.extents.x / 2
            content.add(textEntity)
        }
    }
}

Finally define HomeView to present the user with a button to start the activity while listening for incomming sessions. Note the use of the frame and glassBackgroundEffect modifiers on HomeView; both of those help the window “feel" more like a traditional 2D home view. That said, consider taking advantage of the volumetric window to enhance the home view with 3D content.

struct HomeView: View {
    @State var groupActivitiesModel = GroupActivitiesModel()
    
    var body: some View {
        
        VStack {
            Text("Welcome to model share").font(.headline)
            
            if groupActivitiesModel.session == nil {
                Button("Share model", systemImage: "shareplay") {
                    Task {
                        let _ = try? await TestActivity(modelId: "some id").activate()
                    }
                }
            }
        }
        .task {
            for await session in TestActivity.sessions() {
                groupActivitiesModel.session = session
                session.join()
            }
        }
        .frame(width: 800, height: 600)
        .glassBackgroundEffect()
    }
}

Once a session is established you can use GroupSessionMessenger to send and receive messages containing the data needed to update the model(s) displayed in the volume.

I appreciate the detailed response and suggested work-around! Could you confirm the following statements I've concluded based on your reply:

Scene preference is only applied to windows opened before the session is received. Simply put, all participants must have the window open before the initiator clicks the "Start [Activity]" button (that activates the activity) for SharePlay to associate them. It appears you are opening the recipient window after the session is received, but before the session is joined. Note: received is distinct from joined.

  1. The documentation for SceneAssociationBehavior mentions the scene modifier handlesExternalEvents(matching:), which is supposed to be for the app intercepting a group activity and opening the scene if it is currently not open. Should I take the above limitation to mean that this scene modifier is not properly working in regards to group activities/SharePlay? I have tested with another vision pro user having both the main window and the volume window (the targeted SharePlay window) open and the share menu did not update to show that it was sharing.

  2. So the work around essentially revolves around simply having a singular window group, but selectively showing/hiding content based on whether or not we have a GroupSession. Am I misunderstanding that the assumption here is that multiple window groups being defined (even if they are all volumetric windows as per the second limitation where the initial window style needs to match the SharePlay target window) currently does not work with SharePlay properly then?

Ideally I would want to retain the multiple windows for display purposes, but will go ahead for now and see if the proposed work-around can somehow fit my use case. Again, I appreciate the detail!

  1. Try using View.handlesExternalEvents(preferring:allowing:) instead of handlesExternalEvents(matching:). There's a snippet for this in Adding Spatial Persona support to an activity and the method is covered in detail in Build spatial SharePlay experiences around 6:18. You can retain multiple windows (volumetric and non volumetric) as long as you ensure the window you want to SharePlay is open by all participants before the session is received.

  2. Yes, this is mostly relevant when a recipient launches an (unopen) app by joining a SharePlay activity. The desired window needs to be open before the session is received or SharePlay will pick a window to attach to. When an app launches there's only one window open (the initial one) so SharePlay attaches to that.

Happy to help!

Multi-scene/window shareplay on visionOS
 
 
Q