Td-agent.log file size keep on increasing

[harsh288]

Describe the bug

The log file (C:\opt\td-agent\td-agent.log) is increasing continuously, how to put a limit ??

There is some configuration like file rotate and there is a command however we have a fluentd running as windows service, so if there is any configuration could you please suggest either in conf file or while running the fluentd service from powershell.

To Reproduce

Keep fluentd service running and process different messages the file size keeps increasing.

Expected behavior

The old contents in the file should be deleted after a certain limit is reached.

Your Environment

- Fluentd version:
- TD Agent version:
- Operating system:
- Kernel version:

Your Configuration

Normal configuration

Your Error Log

No Errors it's physical size issue.

Additional context

No response

[ashie]

Moved to fluent/fluent-package-builder#342

[ashie]

You can use rotate_age and rotate_size settings in <system> section in fluent.conf.

refs:

• Enable system.log.rotate_... system configuration #3352
• Windows Service logs don't rotate, no way to set rotation for the service fluentd-docs-gitbook#246
• 1.0: Document rotate_age and rotate_size in system config section fluentd-docs-gitbook#316
• MSI: add log rotation setting fluent-package-builder#284

[harsh288]

Hello @ashie

I tested with the below configuration, it's not working, 1048 means 1kb right ?? After happens if that limit is reached, my file size is more than 2Mb and it's still growing.

<system>
   <log>
     rotate_age 1
     rotate_size 1048
   </log>
</system>

I was confused where to the above settings so I have tried in 2 locations:

C:\opt\td-agent\lib\ruby\gems\2.7.0\bundler\gems\fluentd-b6e76c09d60e\Fluent.conf
C:\opt\td-agent\etc\td-agent\td-agent.conf

Below is my configuration.

Fluentd version: 4.1.1
TD Agent version: 4.1.1
Operating system: Windows Server 2019

[kenhys]

TD Agent 4.1.1 bundles fluentd 1.12.3,
But the above parameter was supported since 1.13.0
https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1130---20210529
You should use TD Agent 4.2.0 or upgrade Fluentd.

[harsh288]

@kenhys So, it should work on the version that I'm using, could you please let me where this should be updated ??

C:\opt\td-agent\lib\ruby\gems\2.7.0\bundler\gems\fluentd-b6e76c09d60e\Fluent.conf
C:\opt\td-agent\etc\td-agent\td-agent.conf

And what should be the behaviour after the configuration is done???

[kenhys]

@harsh288

So, it should work on the version that I'm using, could you please let me where this should be updated ??

No, It will not work because you use Fluentd v1.12.3 (maybe).
It needs to execute td-agent-gem install fluentd for upgrading.

C:\opt\td-agent\lib\ruby\gems\2.7.0\bundler\gems\fluentd-b6e76c09d60e\Fluent.conf
C:\opt\td-agent\etc\td-agent\td-agent.conf

If you using Fluentd v1.13.0 or later,
C:\opt\td-agent\etc\td-agent\td-agent.conf should be updated.

[kenhys]

Oh, It should be followed on fluent/fluent-package-builder#342