Simultaneously interacting with SwiftUI View below a UIViewRepresentable

My problem


I have a SwiftUI app that uses PKCanvasView to allow drawing. The PKCanvasView is transparent and there are SwiftUI views (e.g. an Image) behind which can be dragged and opened.

I want the user to be able to draw on the PKCanvasView with their apple pencil and interact with the views below with their finger...

I can't figure out how to enable simultaneous interaction with the PKCanvasView (i.e. if using apple pencil draw) and the swiftUI views behind (i.e. if using finger, drag and move views).

The context

Here is my PKCanvasView wrapped in a UIViewRepresentable struct:

Code Block
struct CanvasView: UIViewRepresentable {
@Binding var canvasView: PKCanvasView
@State var toolPicker = PKToolPicker()
func makeUIView(context: Context) -> PKCanvasView {
canvasView.drawingPolicy = .pencilOnly
canvasView.backgroundColor = .clear
canvasView.isOpaque = false
canvasView.alwaysBounceVertical = true
canvasView.alwaysBounceHorizontal = true
toolPicker.setVisible(true, forFirstResponder: canvasView)
toolPicker.addObserver(canvasView)
return canvasView
}
func updateUIView(_ uiView: PKCanvasView, context: Context) {
}
}


The structure of my content view is as follows:

Code Block swift
ZStack(alignment: .topLeading){
                CanvasView(canvasView: $canvasView)
                    .zIndex(0)
                ForEach(canvas.subCanvases.indices, id: \.self){ index in
                        CanvasCardView(
                            canvas: $canvas.subCanvases[index],
                            animationNamespace: animationNamespace,
                            onReorder: {
                                reorderCard(at: canvas.subCanvases[index].zOrder)
                        })
                }
            }


The CanvasCardView are the background SwiftUI views which can be dragged and opened. The view attaches a negative zIndex to them which orders them behind the CanvasView (PKCanvasView).

My attempts


I've tried two approaches so far which don't work:
  1. Subclassing PKCanvasView to override point(inside:with:). Passing false when I want to touches to be passed through. Problem with this is that the swiftUI views never recognise the touches through the UIView. I assume as there are no UITouchs behind the scenes.

  2. Adding a @State property to my contentView that can update an allowsHitTesting() modifier dynamically when required. The issue with this is that without allowing the PKCanvasView to be "hit" I can't determine whether it is a pencil touch or not. So i can't properly interact with PKCanvasView simultaneously.

Any other ideas if this is possible? That is, passing touches through a UIViewRepresentable dynamically?

I really want to avoid shoving the SwiftUI views into a UIView (with UIHostingController) and then turning them back into SwiftUI.

Many thanks!

Did you find a way?

Having the same problem while trying to draw with the pen in PencilKit, and using fingers to manipulate a 3D shape in SceneKit. Any suggestions?

I am also trying to find a way to do this. My hope was to solve it using a hittest that distinguishes between the current event type, but I could not figure out. Did anybody solve this?

Simultaneously interacting with SwiftUI View below a UIViewRepresentable
 
 
Q