Permission issue accessing Reminders through EventKit in MacOS app

Hello,

I have an app that I would like to have read and write access to Reminders using EventKit, but on MacOS 14 (14.2.1) when I call requestFullAccessToReminders I get error = nil, success = false (even when access is granted) and there is no permission prompt to use user when access has not yet been granted. Note this is not an issue in iOS.

Here's a stripped down example of what I'm talking about

import EventKit

struct ContentView: View {
    let store = EKEventStore()
    @State var permissionState = "Ready to Request"
    
    var body: some View {
        VStack {
            Text(permissionState).font(.title)
            
            Button("Request Access") {
                requestFullAccessToEvents()
            }.padding()
            
        }
        .padding()
    }
    
    func requestFullAccessToEvents() {
        store.requestFullAccessToReminders { (granted, error) in
            if let foundError = error {
                permissionState = "Error: " + foundError.localizedDescription
            } else if granted {
                permissionState = "Permission Granted"
            } else {
                permissionState = "Permission Denied"
            }
        }
    }
}

then the plist

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>NSRemindersFullAccessUsageDescription</key>
	<string>something meaningful</string>
</dict>
</plist>

I managed to find an error in the log saying

Received error from calaccessd connection: Error Domain=NSCocoaErrorDomain Code=4099 "The connection to service named com.apple.CalendarAgent was invalidated: failed at lookup with error 159 - Sandbox restriction." UserInfo={NSDebugDescription=The connection to service named com.apple.CalendarAgent was invalidated: failed at lookup with error 159 - Sandbox restriction.}. Attempting to call any reply handler.

So in the Signing & Capabilities tab I added Calendars to the App Sandbox, and that seemed to fix the issue, after that the user gets prompted for access to Reminders as expected or gets access depending on the access granted. Then I tried to release the app and I got rejected from the App Store because my app isn't using calendars, but of course I'm asking for access in the sandbox. So is this the wrong way to get permission for access, and if so, what is the correct way?

Thanks!