fluentd gives no error on empty cert file during startup

[gfrankliu]

Describe the bug

I have below TLS listener which works fine.

 <source>
      @type forward
          port 20001
          bind 0.0.0.0
          <transport tls>
                ca_path /opt/tls/ca.crt
                cert_path /opt/tls/tls.crt
                private_key_path /opt/tls/tls.key
                client_cert_auth true
          </transport>
  </source>

Recently clients can no longer connect. It turned out the /opt/tls/tls.crt on the server got an empty file. fluentd came up with no errors, but of course won't accept any TLS connections with empty cert file. I would expect fluentd should fail to start, and give errors about the empty cert file tls.crt.

To Reproduce

See above in description

Expected behavior

fluentd should fail during starting up, instead of starting up but fail to work.

Your Environment

• Fluentd or td-agent version: fluentd --version or td-agent --version

fluentd --version
fluentd 1.11.1

• Operating system: cat /etc/os-release

cat /etc/os-release 
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

• Kernel version: uname -r

uname -r
4.18.0-193.6.3.el8_2.x86_64

If you hit the problem with older fluentd version, try latest version first.

Your Configuration

See above in description

Your Error Log

No errors during starting up.

Additional context

[gfrankliu]

Tested to have key file empty, and fluentd was able to detect during booting up, and exit with failure.

[repeatedly]

Tested to have key file empty, and fluentd was able to detect during booting up, and exit with failure.

So private_key_path rejects empty file but cert_path accepts empty file, right?

[repeatedly]

Patch: #3086

[gfrankliu]

So private_key_path rejects empty file but cert_path accepts empty file, right?

That's correct.
Also if I put a random valid cert in thee cert_path (but not matching the key), fluentd comes up without any errors. After fluentd comes up, when clients try to connect, fluentd then throws the non-fatal error:

fluentd-1.11.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-07-27 22:11:55 +0000 [error]: #0 unexpected error error_class=OpenSSL::SSL::SSLError error="SSL_CTX_use_PrivateKey: key values mismatch"
  2020-07-27 22:11:55 +0000 [error]: #0 suppressed same stacktrace

At this time, fluentd just runs, without really working.

[gfrankliu]

Just tried the patch. It does fix the empty cert file, or wrong formatted cert file, but it didn't fix what I mentioned in my last comment:

Also if I put a random valid cert in the cert_path (but not matching the key), fluentd comes up without any errors.

[repeatedly]

I put a random valid cert in the cert_path (but not matching the key)

Could you write how to validate such cert at startup phase?

[gfrankliu]

openssl provides way to check if cert file and key file match: https://www.ssl247.com/kb/ssl-certificates/troubleshooting/certificate-matches-private-key