Issues with Embedding Python Interpreter in MacOS App Distributed via TestFlight

[Linking to your previous thread for context.]

Lemme start with this quote from Placing content in a bundle:

If you put content in the wrong location, you may encounter hard-to-debug code signing and distribution problems. These problems aren’t always immediately obvious.

You are hitting exactly this problem. You’ve structured your bundle weirdly and then you’re seeing weird results. You have two choices:

• Either use the bundle structure as it’s documented.

• Or debug these weird problems, now and in the future.

IMO the first choice is the better one.

For some reason Bundle Identifier is ignored.

Right. Well, the bundle ID isn’t being ignored, it’s that this bundle isn’t applicable to your python3.11 executable. For that to be the case:

• python3.11 must be in Python.bundle/Contents/MacOS.

• With a name that matches the CFBundleExecutable property in the Info.plist.

Neither of those apply here, so the system is treating Python.bundle as a codeless bundle and treating python3.11 as an unbundled executable.

It sounds like you want to maintain as much of Python’s traditional file system hierarchy as possible. That’s problematic because it directly conflicts with the file system hierarchy required by a bundle. You can potentially square that circle using the techniques outlined in Embedding nonstandard code structures in a bundle.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Hello, thanks a lot for your guidance.

This is the error I am facing at the moment, tried some things but I am stuck:

Prompting policy for hardened runtime; service: kTCCServiceAppleEvents requires entitlement com.apple.security.automation.apple-events but it is missing for accessing={TCCDProcess: identifier={APP_IDENTIFIER}, pid=63455, auid=502, euid=502, binary_path=[PATH_APP]}, requesting={TCCDProcess: identifier=com.apple.appleeventsd, pid=756, auid=55, euid=55, binary_path=/System/Library/CoreServices/appleeventsd},

What I did since last conversation was to move python libraries to the app and only the python executable is located in the bundle.

Python.bundle/Contents
Python.bundle/Contents/Info.plist
Python.bundle/Contents/MacOS
Python.bundle/Contents/MacOS/python3.11

Added to bundle Info.plist:

<key>CFBundleExecutable</key>
<string>python3.11</string>

Added to Python.entitlements:

com.apple.security.inherit, with a Boolean value of true.

Python bundle entitlements now looks like this:

<key>com.apple.security.app-sandbox</key>
<true/>
<key>com.apple.security.inherit</key>
<true/>

In this documentation Embedding a command-line tool in a sandboxed app says:

"Add the App Sandbox and Hardened Runtime capabilities"

I could not select capabilities in the bundle, so from Build Settings Signing section I set to yes Sandbox and Hardened Runtime.

From the same documentation:

"Set the Other Code Signing Flags (OTHER_CODE_SIGN_FLAGS) build setting to $(inherited) -i $(PRODUCT_BUNDLE_IDENTIFIER)"

I had a conflict at this point, I had to remove it because I got a CodeSign error when archiving:

 -i com.sampleApp.app.Python: No such file or directory
Command CodeSign failed with a nonzero exit code

In the same documentation, section "Embed the helper tool" says:

"Add the ToolX executable to that build phase, making sure Code Sign On Copy is checked."

I added the bundle and Code Sign On Copy is checked.

Thanks again for your help.

What I did since last conversation was to move python libraries to the app and only the python executable is located in the bundle.

OK.

How is that bundle created? Did you start with a generic bundle target, created from the the macOS > Bundle template? Or something else?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

How is that bundle created? Did you start with a generic bundle target, created from the the macOS > Bundle template?

Yes exactly, followed as much as I could what describes Create the helper tool target section from Embedding a command-line tool in a sandboxed app document.

OK. And python3.11 executable is built by an external build system, right? That is, you are not using Xcode to build it from source.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

OK. And python3.11 executable is built by an external build system, right? That is, you are not using Xcode to build it from source.

I use Xcode Command Line tools (clang 16.1) to compile the source and generate the dSYM files.

Let me bring to the conversation this document I've been going through, although maybe is not related:

Accessing files from the macOS App Sandbox

Python executable needs a folder to save cache files, at the moment the folder is created by the app and set as environment variable to the executable.

The executable also needs access to the folder where the libraries are, etc.

We already know that this setup works fine in a non sanboxed app. But now the executable is sandboxed, and the app with its libraries is sandboxed too.

Can a sandboxed process (python exec) access to a URL from another sandboxed process (app)?

The document mentioned earlier talks about bookmarks to share file access between processes, is that the way to go with all the urls the executable needs?

If so, I could imagine how to generate the bookmark in the main app, but I don't how to resolve it in the python bundle because I don't have a piece of source code.

Many thanks for your support.

I use Xcode Command Line tools (clang 16.1) to compile the source and generate the dSYM files.

OK. From Xcode’s perspective, that’s still an external build system.

Can a sandboxed process (python exec) access to a URL from another sandboxed process (app)?

Yes, but I don’t think it needs to in this case.

My understanding is that you’re running this Python executable as a child process, using posix_spawn or something layered on top of that (like NSTask or Process). Please confirm.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

My understanding is that you’re running this Python executable as a child process, using posix_spawn or something layered on top of that (like NSTask or Process). Please confirm.

Yes that's right, this is how it is running:

private let process = Process()

private let pipe = Pipe()

func startPython() async throws {


        Task {
            self.process.executableURL = URL(fileURLWithPath: try File.pathResource(.pythonBin))
            self.process.environment = environment
            self.process.arguments = arguments
            self.process.standardOutput = self.pipe
            
            try self.process.run()
            self.process.waitUntilExit()

            let data = self.pipe.fileHandleForReading.readDataToEndOfFile()
            let output = String(data: data, encoding: .utf8) ?? ""
            Log.debug("\(output)", category: .python)
        }

}

OK. In that case your tool will inherit its sandbox from your app. For more on how that works, see On File System Permissions.

For this to pass muster when you submit to the App Store, the tool must be signed with com.apple.security.app-sandbox and com.apple.security.inherit. As you’re using an external build system, that means following the process described in Embed an externally built tool.

FYI, the code you use to run the tool has some significant problems:

• By using Task and then blocking in waitUntilExit(), you are tying up a Swift concurrency thread for the duration. That’s never a good idea.

• If Python outputs a lot of text, you’ll deadlock Python as blocks trying to write to your pipe.

One day we’ll have the shiny new Subprocess type to help with this. Sadly, today is not that day so, in the meantime, you can use the code from Running a Child Process with Standard Input and Output.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Hello, sorry I am not able to follow it, I'm quite confused.

Please correct me when I am wrong.

From this document Embedding a command-line tool in a sandboxed app

Section 'Create the helper tool target'

I created a bundle and that bundle contains an executable that is located in the MacOS folder of the bundle.

Section 'Embed the helper tool'

I added a new Copy Files build phase in the app, the bundle goes to MacOS folder of the app.

Now starts the confusion...

Section 'Embed an externally built tool'

"With the app and Xcode-built helper tool working correctly, it’s time to repeat the process for a tool built using an external build system."

With 'Helper tool' means the bundle?

Section 'Embed the helper tool'

"Add the ToolX executable to that build phase, making sure Code Sign On Copy is checked."

Are we are talking about the bundle?

"In the Build Phases tab of the app target editor, add ToolC to the Embed Helper Tools build phase"

But already added the bundle that contains the executable.

I don't know if I have to add the bundle, the executable or both.

Many thanks for the clarification.

Yeah, sorry. Embedding a command-line tool in a sandboxed app is actually broken up into two separate parts based on the two common scenarios:

• Built from source code by Xcode

• Built externally

However, our documentation style doesn’t make that obvious. So, with reference to that doc:

• Everything before the Create the helper tool target section applies to both scenarios.

• From Create the helper tool target up to, but not including, Embed an externally built tool applies to the built-from-source scenario.

• From Embed an externally built tool to the end of the doc applies to the built-externally scenario.

As your tool is built externally, you want to focus on the first and the third parts.

Additionally, you don’t need to mess around with bundles (well, other than the app’s main bundle!). You can just embed a standalone executable. As long as you place it in one of the expected locations, which in this case is Content/MacOS [1], then Xcode will do the right thing.

So:

1. Build your tool using your external build system.

2. Sign it with the two entitlements needed to indicated sandbox inheritance. You can use ad hoc signing here, because Xcode will resign the tool as part of its build process. The key thing is to have the entitlements in place so that Xcode’s re-signing can preserve them.

3. Add it to your project.

4. Create a build phase that copies it to Contents/MacOS.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] Placing Content in a Bundle lists two, but in this case this one is easiest,

Hello, I still get some errors:

Process tccd

Prompting policy for hardened runtime; service: kTCCServiceAppleEvents requires entitlement com.apple.security.automation.apple-events but it is missing for accessing={TCCDProcess: identifier=com.sampleApp.app, pid=72680, auid=502, euid=502, binary_path=[PATH_TO_APP]]}, requesting={TCCDProcess: identifier=com.apple.appleeventsd, pid=831, auid=55, euid=55, binary_path=/System/Library/CoreServices/appleeventsd},

Process python3.11

flock failed to lock list file (<private>): errno = 35

Basically the changes I made are:

• Used the code from 'Running a Child Process with Standard Input and Output', it works great, thanks!

• Made a Run script to sign the executable:

codesign -s - -i com.sampleApp.app.Python -o runtime --entitlements "$ENTITLEMENTS_PATH" -f "$BINARY_PATH"

• And then created a copy build phase to place the executable in Executables destination. Code sign on copy is selected. (Verified that it is placed in MacOS folder, and correctly signed.)

These are the Entitlements used to sign the executable:

<key>com.apple.security.app-sandbox</key>
<true/>
<key>com.apple.security.inherit</key>
<true/>

And these are the Entitlements of the app:

<key>com.apple.security.app-sandbox</key>
<true/>
<key>com.apple.security.files.user-selected.read-only</key>
<true/>
<key>com.apple.security.network.client</key>
<true/>
<key>com.apple.security.network.server</key>
<true/>

The security network entries are there because the app runs a small python local web server.

I assume they are inherited to the executable, correct me if I am wrong, in that case I will add them to the entitlements of the executable, do they need to be in both?

Thanks

Is your Python code actually using Apple events? If so, what for?

If it’s not using Apple events then you can ignore this warning. It’s not exactly log noise — because some code within the process is almost certainly doing something with Apple events — but if you don’t care about that failing then there isn’t really an issue.

OTOH, if you do care about Apple events then we need to talk about the context in which that failure is occurring, because Apple events in a non-GUI process are possible, but tricky.

With regards the flock error, that’s likely to be something internal to Python. It’s hard for me to offer an opinion about that because I don’t maintain expertise in that code base (I use the Python language, but I’ve never dug into the internals).

It’s likely that this failure is caused by Python making assumptions that aren’t true when sandboxed. I have an example of that in Embedding nonstandard code structures in a bundle, and that example is even Python specific. However, that’s just one example of a wide class of problems.

If you want to understand what’s going on here, you’ll need to either dig into Python yourself or seek help from the Python community.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

First of all many many many thanks for your support, without it we could not progress the way we did. Now python actually works.

Is your Python code actually using Apple events? If so, what for?

No, it's not using Apple events, I can see that warning or error even in the working version.

This document Embedding nonstandard code structures in a bundle

helped us to find a bug that was allowing python to write bytecode (*.pyc) in not allowed folders.

We double checked all dynamic library linkage, seems all good, it's only linking to system libraries, like /usr/lib/libc++.1.dylib

Our final product is still not working, this error is still appearing:

flock failed to lock list file (<private>): errno = 35

All we know is that seems one component is reading from a place that must not be allowed.

We know it is not writing because we have the same app distributed via dmg (not sandboxed) and nothing is written out of the sandbox.

Now is about isolating python components until finding what causes the problem.

Thanks again, a lot of learnings.

Hello, we found out that the problem of our product comes when running a python web server locally.

The app accesses the server via http://127.0.0.1:5000 But the server is never running. On the console there are no errors, only kTCCServiceAppleEvents requires entitlement com.apple.security.automation.apple-events

The app have these entitlements:

<key>com.apple.security.app-sandbox</key>
<true/>
<key>com.apple.security.files.user-selected.read-only</key>
<true/>
<key>com.apple.security.network.client</key>
<true/>
<key>com.apple.security.network.server</key>

Because the app is client and server at the same time I added both entitlements, but I don't know if this could create a conflict...

I also have these lines in the app Info.plist to allow access to a non secure endpoint:

<key>NSAppTransportSecurity</key>
    <dict>
        <key>NSExceptionDomains</key>
        <dict>
            <key>127.0.0.1</key>
            <dict>
                <key>NSExceptionAllowsInsecureHTTPLoads</key>
                <true/>
                <key>NSExceptionRequiresForwardSecrecy</key>
                <false/>
                <key>NSIncludesSubdomains</key>
                <true/>
            </dict>
        </dict>
    </dict>

Is there anything else that maybe I need to consider?

Many thanks in advance.