Hi folks! I'm David Barsky and I work on rust-analyzer, which is the IDE for the Rust programming language. For a while, we've had issues with VS Code not sending the correct changed files to the language server (such as changing commits or rebasing), so I started using rust-analyzer's native, off-by-default file watching functionality that binds to FSEvent via the notify library. This has helped a bunch, but I'm not sure how completely reliable it is. Before I consider changing the default file watching behavior for our (many!) users, I wanted to check: is it possible to combine "walk & watch" into a single, atomic operation?
My goal is that upon getting a notification for a file change event, rust-analyzer can read the changed file and not worry about TOCTOU-esque conditions (rust-analyzer has a pretty robust incremental computation system, so we're able to invalidate changes pretty reliably).
That being said, based off:
- this response from Quinn "The Eskimo!" about 8 years ago, and
- FSEventStreamCallback being a bit limited in the number of args,
...it seems like the answer appears to be "no".
(I'm also familiar with Watchman, but it'd be great if the big pile of heuristics that Watchman uses were less necessary.)
Hi folks! I'm David Barsky and I work on rust-analyzer, which is the IDE for the Rust programming language. For a while, we've had issues with VS Code not sending the correct changed files to the language server (such as changing commits or rebasing), so I started using rust-analyzer's native, off-by-default file watching functionality that binds to FSEvent via the notify library. This has helped a bunch, but I'm not sure how completely reliable it is.
I think the place to start here is with what FSEvents is actually "for". Basically, it was designed so that things like backup, search, sync, etc. could quickly determine that "something has changed" without dealing with the "pain" of actively monitoring, running a daemon, etc.
However, one thing to understand about that design is that it's specifically NOT trying to notify of changes "as soon as they happen". That's pretty clear in the API design- for example, see the latency argument in FSEventStreamCreate:
"The number of seconds the service should wait after hearing about an event from the kernel before passing it along to the client via its callback. Specifying a larger value may result in more effective temporal coalescing, resulting in fewer callbacks and greater overall efficiency."
In other words, "please tell me about changes later so that I don't get as many notifications". The thing to understand here is that this behavior is basically designed as the deliberate alternative to more "realtime" monitoring methods like kqueue or DispatchSource.FileSystemEvent.
Keep in mind that in many cases these APIs are actually best used to complement each other, not as direct replacements. For example, an app might use FSEvents to determine that a large hierarchy "has changed" while there app was not running, then rely on kqueue or Dispatch for realtime monitoring while they are running.
Another key point underneath all of these APIs- their role it to tell you that something HAS changed, not reassure you that it HASN'T. By their nature, file system's are full of race conditions. Much of the way apps interact with the file system is driven by "practically true", not "ACTUALLY true". Simple questions like "does this file exist" can't ACTUALLY be answered in a reliable way- a file can easily be created or destroyed in the time it takes for your question to reach the file system and then return. In practice what prevents these issue from becoming hugely problematic is simply that user/app activity tends to constrain what actually changes from moment to moment and/or what the user consequences of those changes actually are.
However, I think it's important to understand the limits of what you're actually being told. All of these APIs are telling you about what's already happened, NOT about how things stand "now".
Before I consider changing the default file watching behavior for our (many!) users, I wanted to check: is it possible to combine "walk & watch" into a single, atomic operation?
That is a great question that I'm not (quite) ready to answer yet, but I wanted to reply with what I already had. I'll have more to say about this in the next day or two.
__
Kevin Elliott
DTS Engineer, CoreOS/Hardware