My application supports Custom URL Schema which is used to perform an open operation. My application is used as a helper app for MDM, hence it will be installed as a Managed Application. I want only the other Managed Applications to be able to invoke the Custom URL Schema and not allow it for unmanaged applications. Is there any such provision provided by Apple MDM protocol?
I suggest trying out the Managed Open In restrictions. I'm not positive, but I think that custom URL schemes honor the allowOpenFromManagedToUnmanaged and allowOpenFromUnmanagedToManaged restrictions.
Another possibility is that the app itself could enforce this with some coordination with the admin. The app that handles the custom URL scheme receives the bundle ID of the source application that invoked it. So it would be possible for the app to apply an allow list or deny list. The app could support managed app config to allow the MDM server to specify the allow list or deny list.