Something odd with Endpoint Security & was_mapped_writable

I'm seeing some odd behavior which may be a bug. I've broken it down to a least common denominator to reproduce it. But maybe I'm doing something wrong.

What you're seeing may seem like odd behavior, but it's actually normal and expected. Here is what the header documentation in ESMessage.h says:

 * `was_mapped_writable` only indicates whether the target file was mapped into writable memory or not for the lifetime of the
 * vnode. It does not indicate whether the file has actually been written to by way of writing to mapped memory, and it does not
 * indicate whether the file is currently still mapped writable. Correct interpretation requires consideration of vnode lifetimes
 * in the kernel.

Note: If you weren't aware, the most of the EndpointSecurity headers are heavily commented and those comment should generally be considered as the most authoritative documentation for this framework.

The key sentence there is:

Correct interpretation requires consideration of vnode lifetimes in the kernel.

That could easily have been written "vnode caching behavior is complicated and unpredictable, so don't be surprised if the value in this flag seems weird".

You can talke a look at MFSLives for a deeper discussion of how VFS caching works, however, I think what's basically happening here is the following:

1. The file was initially opened read-only so that was the state the vnode was created with.

2. The file was mapped read/write, so the vnode became writeable.

3. Additional access continued occurring, so the vnode from #2 continued to be reused.

One thing to be clear about here is that the file systems view of writability is about how the VFS system interactions with a given file object, NOT larger system security. If 9 process open a file read-only and one process opens it read/write, then the vnode is now writable. That doesn't mean anything changed for those other 9 process.

It will remain that way until I delete the file.

I suspect this is because of how you're testing. As long as you're actively interacting with the file, the VFS system will keep pulling the same vnode out of the cache, leading to the result you're seeing. Leave it alone "long enough" and the vnode will (probably) get evicted from the cache, returning to back to state #1.

Of course, that comes with two qualifiers:

1. "Long enough"-> isn't really defined in any coherent way. If you're on an "active" file system (something like the boot volume where lots of processes are interacting with it) then I'd expect it would be evicted fairly quickly. However, it's possible for a vnode to remain in the cache "forever" (the system won't evict it unless it "has" too).

2. "probably"-> the file system driver itself also has a great deal of control over how the cache is managed. It's possible for a vnode to remain in the cached regardless of other system activity simply because that's what a particular file system is "doing".

If this makes it seem like "was_mapped_writable" isn't all that useful for determining what's actually happened to a file... well, yes, I think that's correct. There are many situations where the EndpointSecurity system gives you the information it can get, which isn't the same as the information you'd want.

I think it's main value is that was_mapped_writable==false lets you quickly know that something CAN'T have been modified. There are lots of files that the system only maps read-only (for example, executables) and this makes it easy to quickly ignore/filter those cases.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Ah. That makes sense. So I'll need to track mmap and mprotect if I want to be somewhat certain that a file changed.

So, that's the question I've been really scratching my head on here. The problem with memory mapped I/O is that it makes it possible to break the connection between "the opening of a file for writing" and "modifying a file" in a way that defies any kind of straightforward analysis.

Basically, one a file has been mapped read/write that memory can be cross between processes in a huge variety of different ways, none of which are really "visible" to your app. In concrete terms, it's possible for one process to map a file for writing, exit, and then have a totally different process modify the file through that mapping hour/days/months later and that reality isn't something the ES API (or any other API) has a great way te represent.

Here are my thoughts on this broad issues:

1. The first API this all starts with is "open". You can't modify a file that hasn't been opened R/W, so that's the key point you need to protect and monitor.

2. Once a file has been opened R/W, I think it's a mistake to assume you can "reliably" predict when/if a file has/will be modified through the auth API. Notably, the mmap/mprotect auth calls might tell you that it's become POSSIBLE for memory mapped modification to occur, but they won't tell you when it actually HAS occurred.

3. I think ES_EVENT_TYPE_NOTIFY_WRITE (and other file related notifications you feel are relevant) needs to be treated as the "final" gate/protection point. Check around #2 may have value, but I would also start with the assumption that you'll always miss "something", so ES_EVENT_TYPE_NOTIFY_WRITE is what covers the "unknown unknowns".

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware