notarytool submit fails 94% of the time with Error: MultipartUploadError(error: HTTPClientError.deadlineExceeded) or other error

Is it possible to write our own S3 upload tool … ?

Yes. The notary service has its own web API. For the details, see Notary API. It’s a non-trivial amount of work though.

I’ve worked with folks who’ve been able to upload similarly large files, so I’m not sure why things are failing so badly for you. When you submit a file using notarytool, it should spit out a request UUID before commencing the upload. Is that true here?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

I have had 1 upload succeed out of countless attempts. I have tried libraries at first (electron-notarize), and then when that failed, I went to the command line. That fails as well, so it wasn't the library, and every single time I am getting the following: Sometimes it gets to 20% sometimes 50% but fails inevitably. The timeout does nothing either.

[12:18:30.326Z] Debug [UPLOAD] Completed [1/52] chunks of a multipart upload.
[12:18:30.954Z] Debug [UPLOAD] Completed [2/52] chunks of a multipart upload.
[12:18:31.621Z] Debug [UPLOAD] Completed [3/52] chunks of a multipart upload.
[12:18:32.352Z] Debug [UPLOAD] Completed [4/52] chunks of a multipart upload.
zsh: bus error  xcrun notarytool submit --apple-id "John@*******" --password  --team-id 

I am at a loss for how to continue creating a distributable application.

I am glad to hear you say to have worked with people who have been able to successfully upload large files. Can you share any of the studio or developers so that I can contact them and ask for a reference? In your previous posts on threads about notaryservice's failures, there are replies from developers reporting a failure rate above 50% on a tiny 700MB upload. I am surprised Apple is not more concerned about this situation.

My own experience, and the experience I have heard from the developer community stands out as overwhelmingly negative. I inquired with two Unity developer groups on Discord. I did not find a single developer who has had a consistently successful experience with uploading large files to notaryservice.

I did find other developers who indicated that notarization is what prevented them from releasing a Mac OS X build. I wish I had known that the notary service was so poor at handling large file uploads before we committed to a Mac OS X port of the project. I think if I had known that uploads would take 48 hours of work per update, we would have just explained the weaknesses in the Apple infrastructure to our Kickstarter backers and begged for forgiveness.

One of the developers did provide a nice bash script that wraps notaryservice and loops until it is successful. Today I was able to notarize a 6.05GB file but it took uploading 110GB of data to the S3 bucket. This does give me some hope that I can brute force my way past the issue for the 17GB file. I will start it and let it run continuously for a few days to see if I can force my way past defects in the uploader.

Every time I start an upload, with the loop script or manually, it prints something like this:

Conducting pre-submission checks for PROJECT.zip and initiating connection to the Apple notary service... Submission ID received id: f8476ccd-xxxx-yyyy-zzzz-21ae7d77581a

trops wrote:

zsh: bus error  xcrun notarytool submit --apple-id "John@*******" --password  --team-id 

This is a different problem. Please start a new thread for it.

The above indicates that notarytool crashed, so there should be a crash report. It would help if you attached that crash report to your thread. See Posting a Crash Report for advice on how to do that.

digital2noise wrote:

Can you share any of the studio or developers so that I can contact them and ask for a reference?

No.

Clearly your experience is very different from mine, to the point where I’ve never needed to research the best way to investigate problems like this. I’m going to do that now and get back to you.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Earlier I wrote:

I’ve never needed to research the best way to investigate problems like this. I’m going to do that now and get back to you

OK, I have some answers. Sorry that took so long. There was some internally juggling required.

If you encounter a performance or correctness problem with notary submissions, like the one described above, I recommend that you retry on a different network. That’ll confirm whether the problem is tied to one specific network. We do, for example, regularly see issues like this on corporate networks, where the network’s security policies cause weird problems.

Similarly, you should review any enterprise-y software installed on your Mac. I regularly see problems ilke this caused by endpoint security products. If you can get a Mac without that installed and use that on a non-corporate network, that’d be ideal.

If either, or both, of these things fix the problem, work with your organisation’s IT folks to investigate.

If neither of these things help, it’s definitely worth filing a bug about this. For best effect:

• In Feedback Assistant, follow the path Developer Tools & Resources > “Something else not on this list”.

• Start a submission and note down the request ID.

• While the upload is running slowly, trigger a sysdiagnose log. For instructions on that, see Bug Reporting > Profiles and Logs.

• Also run a few tens of seconds of packet trace. See Recording a Packet Trace.

Include the request ID, the sysdiagnose log, and the packet trace in your bug report.

Please post your bug number, just for the record.

Finally, if you’re in touch with other developers who are having similar problems, I’d appreciate you pointing them at this post. As I mentioned earlier, this isn’t an issue that I’m actively tracking, but it’s possible that’s just because I’m not moving in the right circles. It’d be great to get feedback from a wider range of folks.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Just as an update, the notary service that Apple hosts with AWS is still being targeted by an incorrectly configured SotoAWS client in notary tool.

Current success rates are around 14%, over the past 25TB of uploads. The relentless looper upload-until-it-works script runs all day and all night to get these games notarized.

It appears that Apple knows this is an issue but refuses to fix it, even after the developer of the open source library they are using pointed out the change they need to make in the code for notarytool. I suppose the AWS bill just isn't enough to make anyone at Apple do anything.

Bummer!!!!!

Are you still seeing this problem? If so, there’s an experiment I’d like you to try. In the context in which you’re running notarytool, do this:

% defaults write com.apple.gke.notary.tool nt-upload-connection-timeout <timeout>

where <timeout> is something bigger than the default timeout of 100 seconds. Perhaps you could try 300 seconds?

IMPORTANT Make sure you’re using notarytool from Xcode 16. This setting didn’t work correctly on some older releases.

If you add the --verbose flag, you should see a log message like Setting S3 timeout to 300 seconds., which confirms that the setting ‘took’.

I’m curious whether this improves your success rate.

To get back to your original state, do this:

% defaults delete com.apple.gke.notary.tool nt-upload-connection-timeout

Oh, and just as an aside, the notary service has an API: Notary API. If notarytool doesn’t work for you, that’s always an option. For example, a lot of big developers want to run their entire distribution workflow on a non-Apple platform, and thus use this API rather than notarytool.

If you want to try this out, see Submitting software for notarization over the web.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"