`cp` ( & friends ) silent loss of extended attributes & file flags

Question

Created Aug ’24

Replies 1

Boosts 0

Participants 2

Since the introduction of the siblings / and /System/Volumes/Data architecture, some very basic, critical commands seems to have a broken behaviour ( cp, rsync, tar, cpio…).

As an example, ditto which was introduced more than 10 years ago to integrate correctly all the peculiarity of HFS Apple filesystem as compared to the UFS Unix filesystem is not behaving correctly.

For example, from man ditto:

     --rsrc        Preserve resource forks and HFS meta-data.  ditto will
                   store this data in Carbon-compatible ._ AppleDouble files
                   on filesystems that do not natively support resource forks.
                   As of Mac OS X 10.4, --rsrc is default behavior.
[...]
     --extattr     Preserve extended attributes (requires --rsrc). As of Mac
                   OS X 10.5, --extattr is the default.

and nonetheless:

 # ls -@delO /private/var/db/ConfigurationProfiles/Store        
drwx------@ 5 root  wheel  datavault 160 Jan 20  2024 /private/var/db/ConfigurationProfiles/Store
                           *********
        com.apple.rootless       28 
        ***************************
# mkdir tmp
# ditto /private/var/db/ConfigurationProfiles tmp
ditto: /Users/alice/Security/Admin/Apple/APFS/tmp/Settings: Operation not permitted
ditto: /Users/alice/Security/Admin/Apple/APFS/tmp/Store: Operation not permitted
# ls -@delO tmp/Store
drwx------  5 root  wheel  - 160 Aug  8 13:55 tmp/Store
                           *
#

The extended attribute on copied directory Store is empty, the file flags are missing, not preserved as documented and as usual behaviour of ditto was since a long time ( macOS 10.5 ). cp, rsync, tar, cpio exhibit the same misbehaviour. But I was using ditto to be sure to avoid any incompatibility with the Apple FS propriaitary modifications.

As a consequence, all backup scripts and applications are failing more or less silently, and provide corrupted copies of files or directories. ( I was here investigating why one of my security backup shell script was making corrupted backups, and only on macOS ).

How to recover the standard behaviour --extattr working on modern macOS?

Boost

Answer 1

DTS Engineer OP

Apple

Aug ’24

As an example, ditto which was introduced more than 10 years ago...

This makes me feel very old. "ditto" was part of the base release of Mac OS X 10.0, which makes it 23 years old.

In any case, I want to start by commenting on this:

--extattr     Preserve extended attributes (requires --rsrc). As of Mac
                   OS X 10.5, --extattr is the default.

The idea that extended attributes are something that can simply be "preserved" is dangerously naive. What's important to understand here is that extended attributes don't represent a coherent form or metadata but are instead basically a form of "surrender" on the part of file system designers.

Historically, file systems were basically designed around the assumption that a "file" consisted of:

a) A series of block that stored it's "data".

b) A discrete collection of metadata which stored information about those blocks ("name", "times", "permissions", etc....).

The file systems job was then quite simple- provide a place to store the data ("a"), define a fixed structure to hold the metadata ("b"), dream up a smart way to organize both, implement all that, and then have a nice nap in the afternoon.

Unfortunately, the problem here turned out to be the word "discrete". In practice, there has been a long history of new and brilliant idea which would be AWESOME if the file system would just track this one more piece of metadata... which then forced the file system designer to come up with yet another silly trick to store metadata it did not in fact have space for.

Extended attributes exist because file system designer final got tired of all this foolishness. They still kept "b" around (because there is a core set of metadata every file will have), but they also include a general mechanism that could be used to store "anything". Anyone with a fancy idea can now attach whatever they want to a file and the file system designer can nap in peace.

However, here is the key point in that. Extended attributes don't have any single coherent "meaning", nor do they "do" any coherent "thing". The higher level system determines what they mean and what they'll "do", which makes blindly copying them EXTREMELY problematic.

How to recover the standard behaviour --extattr working on modern macOS?

Case in point "--extattr" has NEVER simply captured "all" extended attributes. In reality, it has only ever captured the specific attributes it recognizes, since that's the ONLY safe option.

Moving the the specific case you mentioned:

drwx------@ 5 root  wheel  datavault 160 Jan 20  2024 /private/var/db/ConfigurationProfiles/Store
                           *********
        com.apple.rootless       28

That is a datavault directory (Quinn's post here has more info). It only be accessed by specifically entitled processes, which is why nothing you've tried has been able access it. Disabling SIP will allow you to access it's contents, however, keep in mind that it's content and structure are all private, so anything you see/find could change at any time.

However, that doesn't explain this:

As a consequence, all backup scripts and applications are failing more or less silently, and provide corrupted copies of files or directories.

I'm not sure what you mean by "corrupt", but nothing stored inside a data vault should need to be backed up. It's understood that they're not generally accessible or preservable, so the system cannot rely on them persisting.

What problem(s) are these missing files actually causing?

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

0