-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for concurrent append in out_file #3808
Commits on Jul 1, 2022
-
Add support for simple inter-worker locking
This is the first cut at a simple locking mechanism that allows Fluentd workers to coordinate each other. Here is how it works: * ServerModule prepares a lockfile directory and share that directory path with its workers (through ENV). * Now, Workers can define critical sessions by creating a lockfile in that directory. * The actual lock was held by flock, so it's automatically freed by OS even when Worker gets killed in the middle of critical session. I'm going to use this mechanism to support concurrent appends in out_file, but should be useful for other plugins as well. Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
Configuration menu - View commit details
-
Copy full SHA for dfd4fc3 - Browse repository at this point
Copy the full SHA dfd4fc3View commit details -
out_file: Add a simple lock to support concurrent appends
Previously, out_file did not support any locking on append mode. With this, out_file will acquire a worker lock when appending to a file, which makes the following configuration just works: <system> workers 2 </system> <match **> @type file append true ... </match> Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
Configuration menu - View commit details
-
Copy full SHA for 6a18e4d - Browse repository at this point
Copy the full SHA 6a18e4dView commit details -
Apply feedback from kou on GitHub#3808
Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
Configuration menu - View commit details
-
Copy full SHA for a538b9b - Browse repository at this point
Copy the full SHA a538b9bView commit details -
Use automatic cleanup with FLUENTD_LOCKDIR
This is a lot safer than doing a manual cleanup, and it also allows to clean up directories in a more rubust manner. Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
Configuration menu - View commit details
-
Copy full SHA for 75ef92f - Browse repository at this point
Copy the full SHA 75ef92fView commit details -
Raise error on configuration if lockdir is unavailable
Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
Configuration menu - View commit details
-
Copy full SHA for 2ebce28 - Browse repository at this point
Copy the full SHA 2ebce28View commit details -
Apply feedback from ashie on GitHub#3808
Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
Configuration menu - View commit details
-
Copy full SHA for 0a7a590 - Browse repository at this point
Copy the full SHA 0a7a590View commit details -
Add testcase for worker lock functions
Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
Configuration menu - View commit details
-
Copy full SHA for d31631e - Browse repository at this point
Copy the full SHA d31631eView commit details -
Apply feeback from kou on GitHub#3808
Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
Configuration menu - View commit details
-
Copy full SHA for d9c793e - Browse repository at this point
Copy the full SHA d9c793eView commit details -
Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
Configuration menu - View commit details
-
Copy full SHA for f3838cb - Browse repository at this point
Copy the full SHA f3838cbView commit details -
Add more tests for inter-worker locking
Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
Configuration menu - View commit details
-
Copy full SHA for 2c9f3f3 - Browse repository at this point
Copy the full SHA 2c9f3f3View commit details