Sending events to VZVirtualMachineView

Speaking of docs, I should be using NSWindow.postEvent(_:atStart:). Doing that instead now triggers the local monitor, but modified keystrokes still appear in the VM without modifiers and scroll wheel events don't do anything. The latter may be due to how I have to start with a CGEvent since NSEvent doesn't have direct support for creating scroll wheel events.

So lemme check that I understand this properly. My reading of your post is that:

• You have a virtualisation app.

• Its UI is based on SwiftUI.

• You’re wrapping VZVirtualMachineView in NSViewControllerRepresentable.

• Everything works if you actually type on the keyboard.

• But you want to send synthetic key events to the view.

• And that works for normal key presses. So sending an F key event will generate an F key event in the VM.

• But you can’t get it working for modified key presses. So sending a command-F key event generates an unmodified F key event in the VM.

Is that right?

If so, let’s start with a quick question: Are you setting capturesSystemKeys? If you are, does not setting that improve things?

I don’t think it will, but I wanna rule that out first.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

The VZVirtualMachineView is in a view hierarchy managed by an NSViewController subclass that is wrapped in NSViewControllerRepresentable, but I doubt that matters, so yes, mostly correct.

I have tried both true and false for capturesSystemKeys and it does not change the behavior seen in the VM.

Mouse scroll wheel events are similar. Real ones from my mouse scroll content in the VM, synthetic ones never show up in the global monitor in the VM and nothing happens.

I’m gonna focus on keyboard events right now. We’re already stretching the bounds of my limited AppKit knowledge.

I have tried both true and false for capturesSystemKeys and it does not change the behavior seen in the VM.

OK.

For context, when you set capturesSystemKeys the view has to jump through more hoops to achieve its goal.

VZVirtualMachineView is handling modifiers via the flagsChanged(with:) method. I can see two potential ways of dealing with that:

• You could dispatch .flagsChanged events down the AppKit event pipeline. The main challenge there is that I’m not 100% sure how to create such events |-:

• You could use a self-targeting CGEvent tap to insert these events. That’ll insert the events before AppKit sees them.

Honestly, I’m not 100% comfortable with either of these approaches. For the first, any solution is going to be pretty tightly bound to both AppKit event routing and the implementation of this view. For the second, I thought that you could set up a self-targeting event tap without invoking the ire of TCC, but I ran a quick check today and that’s not working for me (the event tap requires TCC approval even when you target your own process).

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

This was the little push I needed. Thank you!

I had tried sythesizing .flagsChanged events equivalent to key-down and key-up for the modifier key, but it didn't make a difference. I revisited it once you confirmed that it mattered.

The flags are a 32-bit bitmask. There are the device independent bits, which are documented for each of the modifier keys plus some others, in the upper 16 bits. The lower 16 bits are not documented, but if you attach a monitor, you'll notice some are set. In particular, bits 3 and 8 (0x108) are set when the modifier key is pressed down, and bit 8 (0x100) is still set when the modifier is released.

I turned my synthesized .keyDown/.keyUp pair into a sequence of .flagsChanged with 0x108 set for each modifier, .keyDown, .keyUp, .flagsChanged with 0x100 set. Injecting this sequence of events produces the desired behavior in the VM. Success!

I don't like needing to set these undocumented bits, but my use case is for an internal tool and I can live with it. (Perhaps they can be documented and become part of the public API?)

Now I'm hoping we can find a similar breakthrough for scroll wheel events! 😄

Here is some info on where I am with scroll events. Unlike keystrokes, flag changes and mouse clicks, these cannot be created directly as NSEvents, so I have to start with a CGEvent and create the NSEvent from that. There is some math involved to create the CGEvent at the proper location in the VM view in screen coordinates, but I believe I have that worked out.

My local monitor (inside the VM host app) sees the event. Here is a dump of one:

- NSEvent: type=ScrollWheel loc=(50,300) time=903547.3 flags=0 win=0x0 winNum=0 ctxt=0x0 deltaX=0.000000 deltaY=-10.000000 count:0 phase=None momentumPhase=None #0
  - super: NSObject

The part that jumps out the most from a real scroll event is that the win and winNum values are nil/zero. CGEvents, as much as I understand, sit at a lower level of the system and have no concept of a window. The conversion to NSEvent either doesn't attempt to identify a window or fails to do so. I've tried various ObjC tricks to assign a window, since in Swift this property is read-only. So far, no luck.

Real scroll events have a window and flow into the VM just fine, where the global monitor there logs them. Synthesized events are dropped somewhere. They work in a sample app without the VM view.

Again, this is an internal tool. It'll never be distributed, Mac App Store or otherwise, so I'm not opposed to some dirty tricks to get this working, even if it means revisiting it when it breaks.

I need to correct my last reply. After looking at my sample app again, it was calling NSWindow.sendEvent(_:). That works, but I should be calling postEvent(_:atStart:). After making that change, the sample app no longer scrolls.

This doesn't appear unique to the VM view. Something more general in AppKit's event handling is rejecting my synthesized scroll event.

Sadly, I don’t think I’ll be able to help you with the scroll wheel side of this; this is way too deep into AppKit for me )-:

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"