Commits on Mar 5, 2021

1. Support multiple kind of timestamp format …

   Use Case 1: Most of timestamp format is unixtime, but some entry use %iso8601
   
    time_type mixed
    time_format_fallbacks unixtime,%iso8601
   
   Use Case 2: Most of timestamp format is %iso8601, but some entry use unixtime
   
    time_type mixed
    time_format_fallbacks %iso8601,unixtime
   
   With a micro benchmark:
   
   * normal TimeParser
   * primary iso8601 and fallback to unixtime
   * primary unixtime and fallback to iso8601
   
     Warming up --------------------------------------
       Normal time parser     1.191M i/100ms
     primary format is %iso8601, secondary is unixtime
                            408.609k i/100ms
     fallback to unixtime   302.240k i/100ms
     Calculating -------------------------------------
       Normal time parser     12.136M (± 2.3%) i/s -     60.731M in   5.007157s
     primary format is %iso8601, secondary is unixtime
                               4.074M (± 0.5%) i/s -     20.430M in   5.015248s
     fallback to unixtime      3.010M (± 0.5%) i/s -     15.112M in   5.020749s
   
   Here is the benchmark program [1]
   
   [1] fluent#3252 (comment)
   
     require 'benchmark/ips'
     require 'fluent/time'
     require 'fluent/config'
     require 'fluent/configurable'
   
     class DummyForTimeParser
       include Fluent::Configurable
       include Fluent::TimeMixin::Parser
     end
   
     def config_element(name = 'test', argument = '', params = {}, elements = [])
       Fluent::Config::Element.new(name, argument, params, elements)
     end
   
     a = Fluent::TimeParser.new('%d/%b/%Y:%H:%M:%S %z')
     i = DummyForTimeParser.new
     i.configure(config_element('parse', '',
                                {'time_type' => 'mixed',
                                 'time_format' => '%iso8601',
                                 'time_format_fallbacks' => ['unixtime']}))
     b = i.time_parser_create
     i2 = DummyForTimeParser.new
     i2.configure(config_element('parse', '', {'time_type' => 'mixed',
                                               'time_format_fallbacks' => ['unixtime', '%iso8601']}))
     c = i2.time_parser_create
     time = Time.now
     timesec = "#{time.sec}".freeze
   
     Benchmark.ips do |x|
       x.report("Normal time parser") do
         a.parse('28/Feb/2013:12:00:00 +0900'.freeze)
       end
   
       x.report("primary format is %iso8601, secondary is unixtime") do
         b.parse('2021-01-01T12:00:00+0900'.freeze)
       end
   
       x.report("fallback to unixtime") do
         c.parse(timesec)
       end
     end
   
   Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>

   

   kenhys committed Mar 5, 2021
   Configuration menu

   • View commit details
   • Copy full SHA for a71befd
   • Browse repository at this point

   Copy the full SHA

   a71befd View commit details

   Browse the repository at this point in the history