Improved "rename" support

[aktau]

As mentioned here: howeyc/fsnotify#104

Currently, it's not straightforward to detect a rename correctly. In most cases, one will see this:

RENAME file1
CREATE file2

Yet this doesn't guarantee that file1 and file2 are the same file (there could be a file creation in between, or file1 could be moved outside a watched folder and file2 could be moved in). To work around this, one could store the os.FileInfo for every watched file and use os.SameFile to compare it with the new file. This is a bit cumbersome though, as it requires keeping a lot of os.FileInfo structs in memory + bookkeeping them.

However, some platforms provide a way to atomically see what was the old name and what's the new name of a file. To quote @nathany:

On Linux there is a "cookie" used to tie the RENAME FROM and TO events together, but this isn't currently exposed by fsnotify. I still need to do more research across each platform.

This is a tracker issue to see if there is sufficient cross-platform support to enable this in the default API.

[jmhodges]

I think watchdog (https://github.com/gorakhargosh/watchdog) has figured out a cross-platform API, by the way.

[nathany]

@jmhodges Do you have any more details on "Weird Things" vim does? Is this when saving or is there a rename file function people are using in Vim?

[nathany]

ref: rjeczalik/notify#78

[omeid]

Vim uses swap and backup files, involves rename and moving.

[F21]

Do you think keeping a list of os.FileInfos in a concurrently safe map like map[string]*os.FileInfo as mentioned by @aktau is probably the best way to get this implemented?

[F21]

Having done some experimentation on Windows, the os.Samefile() method doesn't work on Windows, because it compares by using the file path. So, if you had test.txt, the fileInfo stores the path to test.txt. After renaming it to test2.txt, using os.Lstat() on the new file stores the path as test2.txt. When you do a comparison with os.Samefile(), it sees that the file paths for the 2 FileInfos are different and returns false.

[benmccann]

Once #114 is implemented, I think you can do this using fanotify:

The event FAN_RENAME may contain two information records. One of type FAN_EVENT_INFO_TYPE_OLD_DFID_NAME identifying the old directory entry, and another of type FAN_EVENT_INFO_TYPE_NEW_DFID_NAME identifying the new directory entry.

[arp242]

fanotify isn't needed. It's already implemented in inotify. See: #628.

[benmccann]

Interesting. @arp242 how do you know when to give up waiting for a matching cookie? E.g. if you move a file to a different file system, you will never get a matching cookie and should issue a delete event. Do you have to the wait until 10 other operations on the file system have been made? You could potentially be waiting days before that happens. Is there or should there also be some sort of timeout to avoid waiting too long?

Since I think you probably need a timeout anyway, would it not be sufficient to solely rely on the timeout to limit the cache size rather than limiting to ten entries?

Anyway, what I was referring to with fanotify is that it's more deterministic and does not rely on a heuristic that is "inherently racy" as the notify docs say. Although the downside is that it isn't as well supported in Docker

[arp242]

It doesn't wait for anything; you either get a IN_MOVED_TO event or you don't. It just keeps a small cache to survive to survive event reordering.

if you move a file to a different file system, you will never get a matching cookie and should issue a delete event

No, because the file is not deleted.

[benmccann]

So what do you do if you never get the IN_MOVED_TO event? Do you simply issue no event and the user is never notified that the file is no longer on the file system? Or do you issue a rename event without saying what it was renamed to? And in that case, how do you know when to issue the event?

[arp242]

IN_MOVED_FROM always triggers Rename, and IN_MOVED_TO (which may or may not arrive) triggers Create.

[benmccann]

I see. So basically the consumer is responsible for setting up the timer to decide whether or not to keep waiting for IN_MOVED_TO event with this design

[arp242]

That depends what you want I guess; in a lot of use cases just wait for a Create with RenamedFrom is fine, and/or treating Rename as Delete.

All of this has worked like this for 10 years or so; we can't easily change it. Never mind cross-platform considerations.

I don't really get what you're asking in any case; if you have a concrete problem or use case then ask that.

[benmccann]

You've already answered all my questions. Thanks!

As for my usecase, I want to handle renames to the extent possible - i.e. handling them as renames when I receive both events and otherwise handle them as create/delete when receiving only one event (depending on whether the file was moved to/from the watched location). To do so, I believe I either need to use a timer or use fanotify (#26 (comment)). I hadn't been familiar with #628 or how it worked, so thanks for explaining.

[arp242]

FAN_RENAME is only triggered inside watched directories, so that behaviour isn't any different.

And even if it would be, a fanotify backend here should not send different events; the entire point of this library is to provide a cross-platform experience, so you can develop something on Linux and be reasonably confident it works on Windows. And stuff like that. If one backend starts sending events that aren't supported by other backends then that breaks.

[benmccann]

Yes, it's only triggered inside watched directories, but the part that is different is that I believe it sends the details in a single event instead of splitting it across two events.

Fair point about needing the same functionality on Mac and Windows though.