v1.11.x vs v1.12.x not reading files

[toyaser]

I have an issue with the exact same conf reading monitored files between docker fluentd v1.11.x and v1.12.x.

This is the conf:

<ROOT>
  <source>
    @type tail
    @id in_tail_console_application_log
    path "/mnt/console/*.log"
    pos_file "/mnt/logfilespos/console-auditlog.log.pos"
    pos_file_compaction_interval 24h
    tag "console-audit-logs.*"
    refresh_interval 10
    enable_watch_timer false
    enable_stat_watcher true
    read_from_head true
    <parse>
      @type "json"
      unmatched_lines
    </parse>
  </source>
  <filter console-audit-logs.**>
    @type genhashvalue_alt
    use_entire_record true
    hash_type "sha256"
    base64_enc true
    base91_enc false
    set_key "_hash"
    separator "_"
    inc_time_as_key false
    inc_tag_as_key false
  </filter>
  <match console-audit-logs.**>
    @type copy
    <store>
      @type "elasticsearch"
      host "myhost.com"
      port 9200
      scheme https
      ssl_version TLSv1_2
      logstash_prefix "my"
      logstash_dateformat "%Y.%m"
      type_name "logs"
      tag_key "@log_name"
      user "elastic"
      password xxxxxx
      id_key "_hash"
      remove_keys "_hash"
      @log_level "debug"
      logstash_format true
      include_tag_key true
      reload_connections false
      reconnect_on_error true
      reload_on_failure true
      <buffer>
        flush_thread_count 8
        flush_interval 5s
      </buffer>
    </store>
  </match>
  <source>
    @type monitor_agent
    @id monitor_agent_input
    port 24220
  </source>
  <label @FLUENT_LOG>
    <match fluent.*>
      @type stdout
    </match>
  </label>
</ROOT>

I am currently using fluentd image fluent/fluentd:v1.11.5-1.0 and I tried all different versions of v1.12 (v1.12.0-1.0, v1.12.3-1.0, v1.12.4-1.0, v1.12.4-1.1 and v1.12-1, which was 1.12.4-1.1 at the time) to no avail.

When I turn on the logging for the newest versions, all I get is this:

2021-06-07 18:05:59.335539320 +0000 fluent.trace: {"instance":1820,"message":"enqueueing all chunks in buffer instance=1820"}
2021-06-07 18:06:00 +0000 [trace]: #0 enqueueing all chunks in buffer instance=1820

The only time any changes to files are detected is on fluentd startup, after that no changes are detected and therefore nothing is shipped to elastic, but as soon as I switch back to the older version of fluentd everything starts working again and changes to monitored files are immediately picked up.

Is there something wrong with my conf that is causing the issue with the newer version of fluentd?

I also tried adding follow_inodes true but it made no difference.

[ashie]

and i tried all different versions of v1.12 to no avail.

Could you list up all v1.12.x versions you tried?

[toyaser]

Please see versions below, and I also updated the issue:
v1.12.0-1.0
v1.12.3-1.0
v1.12.4-1.0
v1.12.4-1.1
v1.12-1 (this was v1.12.4-1.1 at the time)

[toyaser]

I am having the same issue with v1.14.0-1.1.

Again will not will not read updates to files it is monitoring.

[toyaser]

Ok, I think I found the issue. Has the pos file format changed between version 11 of fluentd and 12 and above?

Prior to that version. Updating fluentd and pointing to the same pos file everything worked well with no issues, but once i updated to the 12+ it no longer works.

This is a problem as i would like to do a rolling update of my fluentd and i cannot modify my pos file.

Below is my pos file from version 1.11.5:

/mnt/sta-console/sta.20210917.log       0000000000006830        00000000000007ac
/mnt/sta-console/sta.20210919.log       0000000000002946        0000000000000752
/mnt/sta-console/sta.20210920.log       000000000000d496        00000000000002cb
/mnt/sta-console/sta.20210921.log       000000000000cb00        00000000000003de
/mnt/sta-console/sta.20210922.log       000000000001847a        000000000000078d
/mnt/sta-console/sta.20210923.log       00000000000103d7        0000000000000749
/mnt/sta-console/sta.20210924.log       0000000000008790        0000000000000789

And here is my pos file from version 1.14.0

/mnt/sta-console/sta.20210924.log       00000000000003fc        000000000000075f

[ashie]

Ok, I think I found the issue. Has the pos file format changed between version 11 of fluentd and 12 and above?

No, it's not changed, some bug fixes were made though (e.g.: remove nonexistent file entries on startup compaction).
Does it work correctly when you remove the old pos file?

[ashie]

    enable_watch_timer false
    enable_stat_watcher true

I got it, fluent/fluentd#3541 is the cause.

[ashie]

fluent/fluentd#3541 was merged and shipped as v1.14.3