SSL_accept SYSCALL returned=5 errno=0

[venky999]

Describe the bug

I am seeing this warning continuously and causing logs lost..
[warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"

To Reproduce
Run td-agent with the following config
Expected behavior
The warning should not occur

Your Environment

• Fluentd or td-agent version: fluentd --version or td-agent --version
  td-agent 1.10.0
• Operating system: cat /etc/os-release
  RED HAT 7.7
• Kernel version: uname -r
  3.10.0-1062.12.1.el7.x86_64
• Ruby
  2.0.0p648 (2015-12-16) [x86_64-linux]
• Openssl
  1.0.2k-fips 26 Jan 2017

Your Configuration

<source>
  @type forward
  port 52466
  <security>
      self_hostname XXX
      shared_key yyyyyyyy
  </security>
  <transport tls>
      cert_path /etc/td-agent/certs/test.crt
      private_key_path /etc/td-agent/certs/test.key
  </transport>
</source>

Your Error Log

2020-04-14 17:21:44 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:44 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:46 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:46 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:46 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:46 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"

Additional context

[ganmacs]

2.0.0p648 (2015-12-16) [x86_64-linux]

we don't support ruby2.0.0. please update ruby version.

[kritisingh]

i have same issue
Fluentd or td-agent version: fluentd --version or td-agent --version
td-agent 1.10.0
Operating system: cat /etc/os-release
Debian GNU/Linux 10 (buster)
Kernel version: uname -r
4.19.0-8-cloud-amd64
Ruby
ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-linux-gnu]
Openssl
OpenSSL 1.1.1d 10 Sep 2019

I followed link https://bundler.io/v2.0/guides/rubygems_tls_ssl_troubleshooting_guide.html#troubleshooting-certificate-errors and it says below

ruby -ropen-uri -e 'eval open("https://git.io/vQhWq").read'
--> output
Here's your Ruby and OpenSSL environment:

Ruby: 2.5.5p157 (2019-03-15 revision 67260) [x86_64-linux-gnu]
RubyGems: 2.7.6.2
Compiled with: OpenSSL 1.1.1d 10 Sep 2019
Loaded version: OpenSSL 1.1.1d 10 Sep 2019
SSL_CERT_FILE: /usr/lib/ssl/cert.pem
SSL_CERT_DIR: /usr/lib/ssl/certs

With that out of the way, let's see if you can connect to rubygems.org...

Bundler connection to rubygems.org: failed ❌ (uninitialized constant Bundler)
RubyGems connection to rubygems.org: success ✅
Ruby net/http connection to rubygems.org: success ✅

Although your Ruby installation and RubyGems can both connect to rubygems.org, Bundler is having trouble. The most likely way to fix this is to upgrade Bundler by running gem install bundler. Run this script again after doing that to make sure everything is all set. If you're still having trouble, check out the troubleshooting guide at http://ruby.to/ssl-check-failed 📦
(eval):136: warning: constant OpenSSL::SSL::SSLContext::METHODS is deprecated

/opt/td-agent/embedded/bin/fluent-gem list
--> output
*** LOCAL GEMS ***

addressable (2.7.0)
async (1.24.2)
async-http (0.50.7)
async-io (1.27.6)
async-pool (0.2.0)
aws-eventstream (1.0.3)
aws-partitions (1.288.0)
aws-sdk-core (3.92.0)
aws-sdk-kms (1.30.0)
aws-sdk-s3 (1.61.1)
aws-sdk-sqs (1.24.0)
aws-sigv4 (1.1.1)
bigdecimal (default: 1.3.2)
bundler (1.16.6)
concurrent-ruby (1.1.6)
console (1.8.2)
cool.io (1.6.0)
did_you_mean (1.1.0)
digest-crc (0.5.1)
elasticsearch (6.8.1)
elasticsearch-api (6.8.1)
elasticsearch-transport (6.8.1)
excon (0.73.0)
faraday (0.17.3)
ffi (1.12.2)
fluent-config-regexp-type (1.0.0)
fluent-logger (0.8.2)
fluent-plugin-elasticsearch (4.0.7, 3.0.0)
fluent-plugin-input-gelf (0.3.1)
fluent-plugin-kafka (0.13.0)
fluent-plugin-prometheus (1.7.3)
fluent-plugin-prometheus_pushgateway (0.0.2)
fluent-plugin-record-modifier (2.1.0)
fluent-plugin-rewrite-tag-filter (2.2.0)
fluent-plugin-s3 (1.3.0)
fluent-plugin-systemd (1.0.2)
fluent-plugin-td (1.1.0)
fluent-plugin-td-monitoring (0.2.4)
fluent-plugin-webhdfs (1.2.4)
fluentd (1.10.0)
gelfd2 (0.4.1)
hirb (0.7.3)
http_parser.rb (0.6.0)
httpclient (2.8.2.4)
io-console (default: 0.4.6)
ipaddress (0.8.3)
jmespath (1.4.0)
json (default: 2.0.4)
ltsv (0.1.2)
mini_portile2 (2.4.0)
minitest (5.10.1)
mixlib-cli (1.7.0)
mixlib-config (2.2.4)
mixlib-log (1.7.1)
mixlib-shellout (2.2.7)
msgpack (1.3.3)
multi_json (1.14.1)
multipart-post (2.1.1)
net-telnet (0.1.1)
nio4r (2.5.2)
nokogiri (1.10.9)
ohai (6.20.0)
oj (3.8.1)
openssl (default: 2.0.9)
parallel (1.19.1)
power_assert (0.4.1)
prometheus-client (0.9.0)
protocol-hpack (1.4.2)
protocol-http (0.15.1)
protocol-http1 (0.10.3)
protocol-http2 (0.11.6)
psych (default: 2.2.2)
public_suffix (4.0.3)
quantile (0.2.1)
rake (12.3.3, 12.0.0)
rdkafka (0.7.0)
rdoc (default: 5.0.1)
ruby-kafka (1.0.0)
ruby-progressbar (1.10.1)
rubyzip (1.3.0)
serverengine (2.2.1)
sigdump (0.2.4)
strptime (0.2.3)
systemd-journal (1.3.3)
systemu (2.5.2)
td (0.16.8)
td-client (1.0.7)
td-logger (0.3.27)
test-unit (3.2.3)
timers (4.3.0)
tzinfo (2.0.1)
tzinfo-data (1.2019.3)
webhdfs (0.9.0)
xmlrpc (0.2.1)
yajl-ruby (1.4.1)
zip-zip (0.3)

[kritisingh]

@venky999 did you find a solution yet?

[venky999]

@kritisingh no..seeing same issue

[johnkazmerzak]

This log message was being logged as a trace but is now logged as a warn. #2861

[repeatedly]

Yes. Above change shows this warning logs.
But we can't judge this issue is fluentd bug or certificate/openssl setting mismatch.
We need more information and reproducible step, e.g. how to create certificates, old fluentd works or not and more.

[kritisingh]

I get warning from section, even if there is no <match **> section,

<source>
    @type forward
    <transport tls>
        cert_path /etc/ssl/xx/xx.crt
        private_key_path /etc/ssl/xx/xx.key
        private_key_passphrase xx
    </transport>
</source>

I downgraded to td-agent 1.9.2 version and the ssl warning isn't visible anymore, probably because it is being logged as a trace in this version.

[konstantin-kornienko]

@venky999 , do you use fluentd behind load balancer? we're experiencing same errors on td-agents behind AWS ELB.

[Minipada]

@konstantin-kornienko I am seeing the same behaviour. If i test a tls server using the Load balancer port, it instantly complains:

ERROR
shutting down SSL
CONNECTION CLOSED
SSL_accept:before SSL initialization
SSL_accept:error in before SSL initialization

But if i change the port, everything is fine. Could you find a solution?

[Phlak106]

Same behavior on kubernetes:
fluent/fluentd:v1.11.5-debian-1.0

Fluentd config:

<source>
  @type forward
  <transport tls>
    cert_path /fluentd/etc/fluentd.crt
    private_key_path /fluentd/etc/fluentd.key
    private_key_passphrase *******************
  </transport>
</source>

<match **>
  @type stdout
</match>

K8s spec:

apiVersion: v1
kind: Namespace
metadata:
  name: fluent
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fluent-deployment
  namespace: fluent
  labels:
    app: fluent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fluent
  template:
    metadata:
      labels:
        app: fluent
    spec:
      containers:
      - name: fluentd
        image: "myImageWithCertificatesBasedOnFluent1.15"
        imagePullPolicy: Always
        ports:
        - containerPort: 24224
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
---
kind: Service
apiVersion: v1
metadata:
  name: fluent-svc
  namespace: fluent
spec:
  type: LoadBalancer
  ports:
  - port: 24224
    targetPort: 24224
    name: forward
  selector:
    app: fluent

[robermar23]

I am experiencing the same issue in kubernetes.

Same specs as @Phlak106

I have also tried versions 1.11 and 1.12

[alyssenko-rtr]

We're also seeing this issue in k8s:

fluent/fluentd:v1.11.2-1.0
ruby 2.5.8

<source>
  @type tcp
  bind 0.0.0.0
  ...
  <transport tls>
    cert_path tls/tls.crt
    private_key_path tls/tls.key
  </transport>
  ...
</source>

[cedriclaudrel]

We have the same issue here:

• fluentd 1.11.2
• ruby 2.7.0

<source>
  @type forward
  @id source_forward
  @label processing
  <transport tls>
    cert_path *****
    private_key_path ******
    private_key_passphrase ''
  </transport>
  <security>
    shared_key *****
    self_hostname *****
  </security>
  bind 0.0.0.0
  port 12225
</source>

2021-03-24 18:35:10 +0000 [warn]: #0 [source_forward] unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=before SSL initialization"

Not sure if we lose logs because of that, we don't see any issue on nodes that send logs to it.

Even weirder, we don't have the issue on other fluentd relays with exactly the same config (same distro, same config file, same versions, etc)

We don't use any LB between forwards output and input, all connections are established directly on the same private network.

[krishna295]

anyone has resolution to this issue, I am seeing same error
"unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=before SSL initialization""

[bloodguard]

Im writting this here to say i had the same issue, but i was able to resolve it.

A bit of context. The FluentD is on an EKS cluster. That EKS has a load balancer spawned with AWS Load Balancer Controller. I had the same error:
"unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=before SSL initialization". I recently updated to the latest version of fluentdD (v1.12.3) and was able to see the IP of the host of the connection. The IP was from the Load Balancer. I changed the health check port/path of the Load Balancer to the path of the monitor_agent of the fluentd, and the error just stopped appearing.

[krishna295]

i had non SSL traffic was hitting SSL port(i.e. I inadvertently configured to send non ssl and SSL data to same port), once i segregated everything looks ok

…

On Mon, May 31, 2021 at 2:40 PM SpicyBrown ***@***.***> wrote: Im writting this here to say i had the same issue, but i was able to resolve it. A bit of context. The FluentD is on an EKS cluster. That EKS has a load balancer spawned with AWS Load Balancer Controller. I had the same error: "unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=before SSL initialization". I recently updated to the latest version of fluentdD (v1.12.3) and was able to see the IP of the host of the logs. The IP was from the Load Balancer. I changed the health check port/path of the Load Balancer to the path of the monitor_agent of the fluentd, and the error just stopped appearing. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2952 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASAM3O4J2GMCRORHKRBKSHLTQPJZFANCNFSM4MH5HSUQ> .

[sky-amoncadot]

TL;DR - Ensure that you can configure your load balancer (if you are using one) to route health checks/traffic over the SSL protocol.

I had this issue also.

My setup was:

GCP VM sending SSL encrypted traffic with a self-signed certificate -> GCP Internal Load balancer -> GCP VMs running Fluentd hosting a self-signed cert/key.

@bloodguard proposed solution helped, upgrading Fluentd (we used 1.13.1) allowed us to see the IP address of the GCP Internal LB.

The original health check configured showed the OpenSSL error in Fluent logs:

Protocol: TCP
Port: 9000

Once we changed the protocol to SSL, but kept the port to our designated Fluentd port, the OpenSSL errors went away:

Protocol: SSL
Port: 9000

Hope this helps anyone with this issue.

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[farnsworth]

I found the same problem using an AWS network load balancer (NLB) with an EKS cluster, but NLB has no health check options and solution proposed by @bloodguard can't be used. Has somebody else faced the same issue?

[gabrielnecula]

I am getting this on a barebone virtual machine. Some amount of logs seem to be received while others I'm receiving this error. Anyone managed to reach a root cause? Should I try to change ports?

[gabrielnecula]

I am receiving this and have no clues why
unexpected error before accepting TLS connection by OpenSSL addr="" host="" port=34680 error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=before SSL initialization"

[krishna295]

This is mostly to do with non ssl traffic reaching ssl port

On Wed, Jan 19, 2022 at 5:18 PM gabrielnecula ***@***.***> wrote: I am receiving this and have no clues why unexpected error before accepting TLS connection by OpenSSL addr="" host="" port=34680 error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=before SSL initialization" — Reply to this email directly, view it on GitHub <#2952 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASAM3O3Y4YWCWMDK3556WUTUW42B5ANCNFSM4MH5HSUQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

-- Sent from Gmail Mobile

[jasminehung]

Having this error too, in my case it was due to istio. Fixed by adding AuthorizationPolicy to allow fluentd to send transactions to Elastic.

[andrevcf]

Having this error too.
Modify the LoadBalancer service to externalTrafficPolicy: Local and it fixes that for me on Azure.

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions]

This issue was automatically closed because of stale in 30 days

[pigi-spm]

Hi,

is there a solution to this problem? I get the following error every day at 3:00 a.m:

rage: "unexpected error before accepting TLS connection by OpenSSL addr=\"10.208.x.xx\" host=\"10.208.x.xx\" port=49xxx error_class=OpenSSL::SSL::SSLError error=\"SSL_accept SYSCALL returned=5 errno=0 state=before SSL initialization\"",
    "unexpected error before accepting TLS connection by OpenSSL addr=\"10.208.x.xx\" host=\"10.208.x.xx\" port=39xxx error_class=OpenSSL::SSL::SSLError error=\"SSL_accept SYSCALL returned=5 errno=0 state=before SSL initialization\"",
    "unexpected error before accepting TLS connection by OpenSSL addr=\"185.156.xx.x\" host=\"185.156.xx.x\" port=26xxx error_class=OpenSSL::SSL::SSLError error=\"SSL_accept returned=1 errno=0 state=error: session id context uninitialized\""

I have see on internet, the problem should be on the configuration part of fluentd in the transport tls, but I have not found a solution for that.
My configuration part of fluentd is:

<source>
      @type http
      port xxxxx
      cors_allow_origins ["*"]
      bind 0.0.0.0
      # Certificates must be adjusted
      <transport tls>
        ca_path path/to/chain.pem
        cert_path path/to/cert.pem
        private_key_path path/to/privkey.pem
        # private_key_passphrase PASSPHRASE
        ciphers ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384
        version TLSv1_2
      </transport>
    </source>
...

[dimitarKiryakov]

Hi,
For everyone wondering how to fix the above in the AWS load balancer case:
You have multiple options but the easiest one is to put the TCP metrics port in the first place if you have one. In my case I have Prometheus metrics, so I used the following spec in the load balancer:

kind: Service
metadata:
  name: any-service
  annotations: 
  namespace: {{ .Values.namespace }}
  labels:
    app: any-label
spec:
  type: LoadBalancer
  ports:
    - name: metrics
      port: {{ .Values.metrics.targetPort }}
      targetPort: {{ .Values.metrics.targetPort }}
      protocol: TCP
    - name: syslog
      port: {{ .Values.port }}
      targetPort: {{ .Values.port }}
      protocol: TCP

This is due to the fact that AWS uses only the first port in the list for the health check.