[BR]: latency limit needs to be configurable per jail

[jikamens]

Environment:

• Fail2Ban version : fail2ban-1.0.1-2.el9.noarch
• OS, including release name/version : CentOS Stream 9
• Fail2Ban installed via OS/distribution mechanisms
• You have not applied any additional foreign patches to the codebase
• Some customizations were done to the configuration (provide details below is so)

The issue:

Currently the time limit past which a log entry is considered to have an "odd" timestamp is hard-coded as 60 seconds in filter.py. This should be configurable per-jail.

Steps to reproduce

This is just one example of why this needs to be configurable: Apache httpd logs the timestamp of when a request started being processed, not when it finished being processed, but it doesn't write the log entry to the log file until the request is finished, so if the web server is running slowly or processing a complicated request that takes more than a minute, fail2ban will think the log entry for it has a bad timestamp.

Expected behavior

I should be able to configure a particular jail to allow timestamps to be older than a minute without fail2ban complaining about them.

Observed behavior

The timestamp is hard-coded at 60 seconds so I can't reconfigure it, and as a result logwatch complains to me every night about a bunch of fail2ban log entries for bad timestamps.

Any additional information

Configuration, dump and another helpful excerpts

Any customizations done to /etc/fail2ban/ configuration

Relevant parts of /var/log/fail2ban.log file:

Relevant lines from monitored log files:

[sebres]

This should be configurable per-jail.
I should be able to configure a particular jail to allow timestamps to be older than a minute without fail2ban complaining about them.

Why? I mean what is the goal of such parameter? To silent warning? Hmm...

This subject was extensively discussed in #2882 (starting somewhere from #2882 (comment)).

It is not clear too me, which pros we'll get implementing option like this.

Apache httpd logs the timestamp of when a request started being processed

This could be surely changed on httpd side (no idea whether it is configurable by apache), e. g. why not log both times, time of response (as event time) and time of request.
But no matter whether it is done or not, such an option just to silent some warning, that by the way occurring only once a day, is a bit heavy, is not it?
At least considering precedents of other tasks/RFE that we have to implement, I'll set the priority of this to very low.

[jikamens]

Monitoring software should not generate ominous-looking warnings about things that are not actually error conditions. When I see alerts about problematic timestamps in logs, I think there's something wrong that I need to fix. In fact there isn't, and I shouldn't have to waste my time investigating to determine that, and neither should other admins.

"Just ignore the warnings, they're not important," is not a good answer. If they're not important then why are you printing them? They sure sound important! They sound like something's not working properly and we need to fix it! In fact, they say, explicitly, that something's not working properly and we need to fix it!

If this were configurable then you could change the Apache httpd configuration that you ship to prevent these warnings for httpd without degrading how timestamps are handled for other applications that generate logs.

This could be surely changed on httpd side

It's not reasonable to suggest that the maintainers of httpd should change how/what they log to prevent fail2ban from generating spurious warnings. They consider what they're logging to be correct. Fail2ban should be able to cope with that.

[sebres]

Monitoring software should not generate ominous-looking warnings about things that are not actually error conditions.

1. It is a warning condition, therefore a warning (not an error) is thrown.
2. if you want to see only errors, set appropriate loglevel to fail2ban, so it'd not bother you till some error conditions.

"Just ignore the warnings, they're not important," is not a good answer. If they're not important then why are you printing them?

Because it is important.
Firstly the behavior gets changed in some version: previously the time in log was leading (so periodically people coming here to ask why the ban doesn't occur), now fail2ban recognize that in operation mode, so it the warning explains what happens with time read from log.
Secondly one could take certain precautions to avoid this situation, or if one's aware to ignore the warning.

"Just ignore the warning" means in that case, do it if you understand the consequences and it is not important for you.

Also note that if such latency option would be implemented, it will be 0 by default.
If you hoped that we would set it to some potential affected jails (so user is not burdened by "evil" warnings) I have bad news to you, we would not do that for specific jails like apache by default.
Simply because some people properly configure its fail2ban correctly, so never see that warning at all.
For instance monitoring of whole access-log is not advisable at all, see Best practice # Reduce parasitic log traffic, so if some jail must only react at unauthorized (e. g. 401) or failed attempts like 404, 403, etc, it is advisable to log such attempts into separate log-file and monitor only that.
In this case, because such 40x-50x requests are very fast usually, the response time will scarcely deviate from request time, so normally such users would never see that warning, and if they do it suddenly - something may be wrong and warning is justified.

It's not reasonable to suggest that the maintainers of httpd should change

They should not.
The user can do that, for example using nginx it is simply to configure own access-log format, and it is even recommended due to Best practice.
But it is even not expected for many/many constellation where response time almost equals request time and it's expected.

[opoplawski]

Couple comments:

• Some jails seem to require scanning the full access logs because you are still wanting to catch 2XX messages (e.g. apache-badbots php-url-fopen)
• Perhaps a threshold on the number of delayed messages would be appropriate? I mean you are pulling log items and each are on time and then just one is off? In fact in the apache case time might even go backwards:

10.10.41.2 - - [25/Apr/2023:10:27:34 -0600] "GET / HTTP/1.1" 
114.119.158.142 - - [25/Apr/2023:10:27:16 -0600] "GET /

• I might allow logwatch to ignore this warning for particular jails