Skip to content

Commit

Permalink
Merge pull request #3092 from fluent/fix-unstable-timekey-conparison-…
Browse files Browse the repository at this point in the history
…with-hash-method

buffer: Do more precise timekey optimization handling. Fix #3088
  • Loading branch information
repeatedly committed Aug 4, 2020
2 parents 8ebff61 + 6cc2606 commit 17aae56
Showing 1 changed file with 19 additions and 3 deletions.
22 changes: 19 additions & 3 deletions lib/fluent/plugin/buffer.rb
Original file line number Diff line number Diff line change
Expand Up @@ -143,17 +143,33 @@ def <=>(o)
end
end

# timekey should be unixtime as usual.
# So, unixtime should be bigger than 2^30 - 1 (= 1073741823) nowadays.
# We should check object_id stability to use object_id as optimization for comparing operations.
# e.g.)
# irb> Time.parse("2020/07/31 18:30:00+09:00").to_i
# => 1596187800
# irb> Time.parse("2020/07/31 18:30:00+09:00").to_i > 2**30 -1
# => true
def self.enable_optimize?
a1 = 2**30 - 1
a2 = 2**30 - 1
b1 = 2**62 - 1
b2 = 2**62 - 1
(a1.object_id == a2.object_id) && (b1.object_id == b2.object_id)
end

# This is an optimization code. Current Struct's implementation is comparing all data.
# https://github.com/ruby/ruby/blob/0623e2b7cc621b1733a760b72af246b06c30cf96/struct.c#L1200-L1203
# Actually this overhead is very small but this class is generated *per chunk* (and used in hash object).
# This means that this class is one of the most called object in Fluentd.
# See https://github.com/fluent/fluentd/pull/2560
# But, this optimization has a side effect on Windows due to differing object_id.
# But, this optimization has a side effect on Windows and 32bit environment(s) due to differing object_id.
# This difference causes flood of buffer files.
# So, this optimization should be enabled on non-Windows platform.
# So, this optimization should be enabled on `enable_optimize?` as true platforms.
def hash
timekey.object_id
end unless Fluent.windows?
end if enable_optimize?
end

# for tests
Expand Down

0 comments on commit 17aae56

Please sign in to comment.