Add a feature to clean up position informations on in_tail

[thkoch2001]

• fluentd version: 0.12.20 - commit aee8086
• environment: fluentd pods running on Google Container Engine / kubernetes

Google Container Engine uses fluentd pods to collect container log files via the in_tail plugin and forward the logs to Stackdriver logging.

When a container is deleted, kubernetes also deletes the containers log file and there will never be a log file at this filesystem path again.

However the position file will never clean up the obsolete line in the position file although the position value is ffffffffffffffff.

We see production clusters with position files of over 10000 lines.

Can this cause performance problems with fluentd? Should this be fixed?

The config stanza for the containers log files is:

<source>
  type tail
  format json
  time_key time
  path /var/log/containers/*.log
  pos_file /var/log/gcp-containers.log.pos
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  tag reform.*
  read_from_head true
</source>

[repeatedly]

in_tail removes untracked file positions at start phase.
So if you restart fluentd, pos_file is updated.

Can this cause performance problems with fluentd?

I'm not sure. I didn't receive any report of pos_file releated performance issue.

fluentd/lib/fluent/plugin/in_tail.rb

Line 746 in 9d0a8dc

class FilePositionEntry

You can see pos_file implementation is here.
I think this IO cost should be ignored on normal file system.

[tagomoris]

The file which grows infinitely sounds terrible in production environment (even if that growing speed is slow, or if there are no reports about performance regression).
I think we should add any features to clean it up to tail plugin.

But currently, we have a plan to switch to use storage plugin from pos file for that purpose.
I think we can add this cleanup feature at the same time with switching storage plugins.

[repeatedly]

I added note for this issue on http://docs.fluentd.org/articles/in_tail.

[piosz]

cc @crassirostris

[mrkstate]

How do we get this feature (defect) fixed? This is creating a situation for us - as the POS file is growing rather quickly and the process has to continue to comb thru this file. We see constant CPU cycles even after no additional data has been written to the tailed log.

[roffe]

@mrkstate Run a cron that kills fluentd every 30 minutes https://github.com/roffe/kube-gelf/blob/master/cron.yaml

[Krishna1408]

hi @roffe If i kiil fluentd won't it create problem for logging ? e.g. fluentd was down for 2 minutes so in that case won't I loose the log data for those 2 minutes ?

[roffe]

@Krishna1408 no, since the position file will let it know where to pickup from last time

[TiagoJMartins]

Is restarting fluentd still the only solution to this? I've just confirmed it's still happening and would be nice to know if a fix is underway or if I can try and tackle the issue.

[ganmacs]

This feature will be released in the next version. #2805

[juliantaylor]

we see position file corruption with the compaction enabled, see #2918