Watchdog shortcoming with Windows

[davidelbert]

This one is complicated and comes from Tyrel looking in detail at Watchdog and Windows. The default Watchdog observer on Windows doesn't catch changes to CIFS/SMB mounted directories because it uses ReadDirectoryChangesW which has problems. There are numerous report of ReadDirectoryChangesW not working reliably on any filesystem other than NTFS (even problems with local FAT, let alone network drives). Also has other sharp edges, like if multiple files are added to the directory quickly, then it will start returning errors instead of the updates. Basically: code should not assume ReadDirectoryChangeW is authoritative, and instead be periodically iterating over files to find changes. This could be a serious pain. Not sure about issues of efficiency and the fact that we haven't had a lot of trouble with it makes me wonder about having it configurable but maybe not.

Some info here: https://devblogs.microsoft.com/oldnewthing/20110812-00/?p=9913

Microsoft API documents say as of 2023 that the only guaranteed way to know of file additions/changes/deletions in a directory is to periodically list them all and keep track yourself. Ugh. Even full fat notification tools used by the big three have this limit: https://github.com/fsnotify/fsnotify

Watchdog is aware of this, and states setting the change watching method to the poller instead when using CIFS/SMB filesystems (https://github.com/gorakhargosh/watchdog) and possibly everywhere on Windows. Right now to do this requires editing of the openmsi source code, but probably needs to be done (despite the load/performance hit) because it is apparently the only way on window>

Bottom line is this is complicated and probably needs some discussion and Tyrel should be in that. Real answer is to have Microsoft fix ReadDirectoryChangeW but no way to control that.

[eminizer]

I'm doing some research about this today. Do you @davidelbert or @tmcqueen-materials have a specific example of where this caused a problem in an OpenMSIStream deployment that I may be able to try replicating on another system in the meantime?

From the reading that I'm doing it currently seems like Watchdog should be more robust than ReadDirectoryChangesW on its own, and I've also added some (configurable) robustness to OpenMSIStream for stuff like many files being added at once or files that are being frequently and persistently updated. I have yet to run into this problem myself, so unless I can replicate it I'm loathe to switch our file monitoring back to something less robust with more edge cases.

OpenMSIStream was manually tracking all of the files in a directory tree until v1.4.0 when we switched over to Watchdog, and that decision was informed by issues with storing large lists of files in memory (or even in atomic files on disk) as well as several edge cases that had showed up.

In the meantime I'll keep researching this to see if there's an extension that can or should be made to Watchdog, or if there are better solutions that we could use in deployments that could potentially see issues like these. More to come.

[eminizer]

It looks like watchdog natively supports an alternate Observer that "polls watched directories for changes by periodically comparing snapshots of the directory tree", and it's recommended that this Observer be used when pairing watchdog on CIFS.

I believe that using that alternate Observer should solve this problem as described, and I can very easily add in a command line option to use it instead of the default Observer. I'll add some documentation for the new option, and a CI test for the alternate Observer, but I don't think I'll have a way to reliably test it where the default Observer wouldn't work but the alternate one would.

PR incoming.

[eminizer]

Yeah, this new option should solve this problem using watchdog's fallback observer, but please do get back in touch if it doesn't behave as expected.

[tmcqueen-materials]

Thank you @eminizer ! This is exactly the functionality that is needed. For posterity, while certainly required for network file systems on windows, PollingObserver may also be needed for non network file systems, for example where a large number of file events in a short time causes ReadDirectoryChangesW to fail (see gorakhargosh/watchdog#1019 and gorakhargosh/watchdog#391 ) on Windows, or inotify to overflow (see gorakhargosh/watchdog#654 ) on linux.