Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hello, I am experiencing a sudden termination here that causes the service to go down haproxy: Thread 14 is about to kill the process. #2566

Closed
Mark11122 opened this issue May 15, 2024 · 5 comments
Closed

Hello, I am experiencing a sudden termination here that causes the service to go down haproxy: Thread 14 is about to kill the process. #2566

Mark11122 opened this issue May 15, 2024 · 5 comments
Labels
status: feedback required The developers are waiting for a reply from the reporter. type: bug This issue describes a bug.

Comments

@Mark11122
Copy link

Detailed Description of the Problem

May 14 23:59:26 haproxy: Thread 14 is about to kill the process.
May 14 23:59:26 haproxy: Thread 1 : id=0x7f65e4a6c2c0 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
May 14 23:59:26 haproxy: 1/1 stuck=0 prof=0 harmless=1 wantrdv=0
May 14 23:59:26 haproxy: cpu_ns: poll=5258378644773 now=5258665955274 diff=287310501
May 14 23:59:26 haproxy: curr_task=0
May 14 23:59:26 haproxy: Thread 2 : id=0x7f65dfeae700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
May 14 23:59:26 haproxy: 1/2 stuck=0 prof=0 harmless=1 wantrdv=0
May 14 23:59:26 haproxy: cpu_ns: poll=5486068836817 now=5486355652298 diff=286815481
May 14 23:59:26 haproxy: curr_task=0
May 14 23:59:26 haproxy: Thread 3 : id=0x7f65de19c700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
May 14 23:59:26 haproxy: 1/3 stuck=0 prof=0 harmless=1 wantrdv=0
May 14 23:59:26 haproxy: cpu_ns: poll=5778666787847 now=5778876699136 diff=209911289
May 14 23:59:26 haproxy: curr_task=0
May 14 23:59:26 haproxy: Thread 4 : id=0x7f65dd814700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
May 14 23:59:26 haproxy: 1/4 stuck=0 prof=0 harmless=1 wantrdv=0
May 14 23:59:26 haproxy: cpu_ns: poll=5454785404853 now=5454980545759 diff=195140906
May 14 23:59:26 haproxy: curr_task=0
May 14 23:59:26 haproxy: Thread 5 : id=0x7f65dd013700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
May 14 23:59:26 haproxy: 1/5 stuck=0 prof=0 harmless=1 wantrdv=0
May 14 23:59:26 haproxy: cpu_ns: poll=6221042503982 now=6221327641201 diff=285137219
May 14 23:59:26 haproxy: curr_task=0
May 14 23:59:26 haproxy: >Thread 6 : id=0x7f65dc812700 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=1 rqsz=4
May 14 23:59:26 haproxy: 1/6 stuck=1 prof=0 harmless=0 wantrdv=0
May 14 23:59:26 haproxy: cpu_ns: poll=6083621974346 now=6085511981745 diff=1890007399
May 14 23:59:26 haproxy: curr_task=0x7f65ac157ab0 (tasklet) calls=2
May 14 23:59:26 haproxy: fct=0x434d20(ssl_sock_io_cb) ctx=0x7f65c05b9bd0
May 14 23:59:26 haproxy: call trace(16):
May 14 23:59:26 haproxy: | 0x58ea9b [85 c0 89 c5 75 17 48 81]: ha_dump_backtrace+0x2b/0x2d9
May 14 23:59:26 haproxy: | 0x58f558 [64 41 8b 75 00 8d 46 01]: debug_handler+0x118/0x252
May 14 23:59:26 haproxy: | 0x7f65e3bf8100 [48 c7 c0 0f 00 00 00 0f]: libpthread:+0xf100
May 14 23:59:26 haproxy: | 0x7f65e3bf772b [48 3d 00 f0 ff ff 48 89]: libpthread:send+0x7b/0xa9
May 14 23:59:26 haproxy: | 0x5ddbc7 [48 83 f8 00 0f 8e 97 00]: main+0x1b14f7
May 14 23:59:26 haproxy: | 0x431efd [85 c0 41 89 c4 75 19 48]: main+0x582d
May 14 23:59:26 haproxy: | 0x7f65e362f9ab [85 c0 7e 07 48 63 d0 48]: libcrypto:BIO_write+0x6b/0x100
May 14 23:59:26 haproxy: | 0x7f65e3632a81 [48 89 ef 41 89 c6 e8 34]: libcrypto:+0x11ea81
May 14 23:59:26 haproxy: | 0x7f65e39968db [85 c0 0f 8e bd 01 00 00]: libssl:ssl3_accept+0x13b/0xe13
May 14 23:59:26 haproxy: | 0x4350cf [83 f8 01 4c 8b 4c 24 08]: ssl_sock_io_cb+0x3af/0xb49
May 14 23:59:26 haproxy: Thread 7 : id=0x7f65d4f27700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
May 14 23:59:26 haproxy: 1/7 stuck=0 prof=0 harmless=1 wantrdv=0
May 14 23:59:26 haproxy: cpu_ns: poll=5827196411899 now=5827382801104 diff=186389205
May 14 23:59:26 haproxy: curr_task=0
May 14 23:59:26 haproxy: Thread 8 : id=0x7f65d4726700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
May 14 23:59:26 haproxy: 1/8 stuck=0 prof=0 harmless=1 wantrdv=0
May 14 23:59:26 haproxy: cpu_ns: poll=5945594787238 now=5945803756400 diff=208969162
May 14 23:59:26 haproxy: curr_task=0
May 14 23:59:26 haproxy: Thread 9 : id=0x7f65d3f25700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
May 14 23:59:26 haproxy: 1/9 stuck=0 prof=0 harmless=1 wantrdv=0
May 14 23:59:26 haproxy: cpu_ns: poll=5465812987950 now=5466001892812 diff=188904862
May 14 23:59:26 haproxy: curr_task=0
May 14 23:59:26 haproxy: Thread 10: id=0x7f65d359d700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
May 14 23:59:26 haproxy: 1/10 stuck=0 prof=0 harmless=1 wantrdv=0
May 14 23:59:26 haproxy: cpu_ns: poll=5609236266996 now=5609502567179 diff=266300183
May 14 23:59:26 haproxy: curr_task=0
May 14 23:59:26 haproxy: Thread 11: id=0x7f65d2d9c700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
May 14 23:59:26 haproxy: 1/11 stuck=0 prof=0 harmless=1 wantrdv=0
May 14 23:59:26 haproxy: cpu_ns: poll=5998406833643 now=5998698758896 diff=291925253
May 14 23:59:26 haproxy: curr_task=0
May 14 23:59:26 haproxy: Thread 12: id=0x7f65d259b700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
May 14 23:59:26 haproxy: 1/12 stuck=0 prof=0 harmless=1 wantrdv=0
May 14 23:59:26 haproxy: cpu_ns: poll=5460791068272 now=5461076825814 diff=285757542
May 14 23:59:26 haproxy: curr_task=0
May 14 23:59:26 haproxy: Thread 13: id=0x7f65d1d9a700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
May 14 23:59:26 haproxy: 1/13 stuck=0 prof=0 harmless=1 wantrdv=0
May 14 23:59:26 haproxy: cpu_ns: poll=5257951167579 now=5258103161843 diff=151994264
May 14 23:59:26 haproxy: curr_task=0
May 14 23:59:26 haproxy: *>Thread 14: id=0x7f65d1599700 act=1 glob=1 wq=1 rq=0 tl=1 tlsz=0 rqsz=3
May 14 23:59:26 haproxy: 1/14 stuck=1 prof=0 harmless=0 wantrdv=0
May 14 23:59:26 haproxy: cpu_ns: poll=5170878374862 now=5174130648758 diff=3252273896
May 14 23:59:26 haproxy: curr_task=0x7f65b836a110 (tasklet) calls=393
May 14 23:59:26 haproxy: fct=0x573780(sc_conn_io_cb) ctx=0x7f65a81709e0
May 14 23:59:26 haproxy: strm=0x7f65a82bb780,10084b src=52.97.27.93 fe=fe_ex2016 be=be_ex2016_eas dst=ex_80_12
May 14 23:59:26 haproxy: txn=0x7f65a820c9b0,40000 txn.req=MSG_DONE,d txn.rsp=MSG_DATA,d
May 14 23:59:26 haproxy: rqf=48c40080 rqa=8000 rpf=80070202 rpa=4000000
May 14 23:59:26 haproxy: scf=0x7f65a814a0b0,EST,0 scb=0x7f65a81709e0,EST,91
May 14 23:59:26 haproxy: af=(nil),0 sab=(nil),0
May 14 23:59:26 haproxy: cof=0x7f65cc686900,80000300:H1(0x7f65b415ebf0)/SSL(0x7f65d8199270)/tcpv4(1319)
May 14 23:59:26 haproxy: cob=0x7f659c462b90,300:H1(0x7f65cc3a6530)/SSL(0x7f65cc58d300)/tcpv4(1458)
May 14 23:59:26 haproxy: call trace(27):
May 14 23:59:26 haproxy: | 0x7f65e3bf8100 [48 c7 c0 0f 00 00 00 0f]: libpthread:+0xf100
May 14 23:59:26 haproxy: | 0x7f65e3bf772b [48 3d 00 f0 ff ff 48 89]: libpthread:send+0x7b/0xa9
May 14 23:59:26 haproxy: | 0x5ddbc7 [48 83 f8 00 0f 8e 97 00]: main+0x1b14f7
May 14 23:59:26 haproxy: | 0x431efd [85 c0 41 89 c4 75 19 48]: main+0x582d
May 14 23:59:26 haproxy: | 0x7f65e362f9ab [85 c0 7e 07 48 63 d0 48]: libcrypto:BIO_write+0x6b/0x100
May 14 23:59:26 haproxy: | 0x7f65e39a1722 [8b 8b 1c 01 00 00 39 c8]: libssl:ssl3_write_pending+0x72/0x14b
May 14 23:59:26 haproxy: | 0x7f65e39a1e16 [85 c0 7e 36 44 39 f8 74]: libssl:ssl3_write_bytes+0xd6/0x71a
May 14 23:59:26 haproxy: | 0x432ed9 [41 8b 56 04 f7 c2 00 00]: main+0x6809
May 14 23:59:26 haproxy: | 0x49c85d [48 85 c0 74 59 44 8b 0d]: main+0x7018d
May 14 23:59:26 haproxy: | 0x4a589b [85 c0 74 2c 49 8b 06 f7]: main+0x791cb
May 14 23:59:26 haproxy: | 0x570b03 [85 c0 0f 8e e7 fe ff ff]: main+0x144433
May 14 23:59:26 haproxy: | 0x5728a0 [48 8b 73 20 0f b6 4b 01]: main+0x1461d0
May 14 23:59:26 haproxy: | 0x571cf1 [49 8b 44 24 30 49 8b 4c]: main+0x145621
May 14 23:59:26 haproxy: | 0x5724cf [48 8b 7b 28 e8 48 b0 f4]: main+0x145dff
May 14 23:59:26 haproxy: | 0x573806 [eb a6 0f 1f 84 00 00 00]: sc_conn_io_cb+0x86/0x88
May 14 23:59:26 haproxy: Thread 15: id=0x7f65d0d98700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
May 14 23:59:26 haproxy: 1/15 stuck=0 prof=0 harmless=1 wantrdv=0
May 14 23:59:26 haproxy: cpu_ns: poll=5753946219460 now=5754215037897 diff=268818437
May 14 23:59:26 haproxy: curr_task=0
May 14 23:59:26 haproxy: Thread 16: id=0x7f65cbfff700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
May 14 23:59:26 haproxy: 1/16 stuck=0 prof=0 harmless=1 wantrdv=0
May 14 23:59:26 haproxy: cpu_ns: poll=5362084364853 now=5362379469875 diff=295105022
May 14 23:59:26 haproxy: curr_task=0
May 14 23:59:27 haproxy: [ALERT] (27217) : Current worker (11379) exited with code 134 (Aborted)
May 14 23:59:27 haproxy: [ALERT] (27217) : exit-on-failure: killing every processes with SIGTERM
May 14 23:59:27 haproxy: [WARNING] (27217) : All workers exited. Exiting... (134)
May 14 23:59:27 Keepalived_vrrp[12876]: Quorum lost for tracked process track_haproxy
May 14 23:59:27 Keepalived_vrrp[12876]: (VI_101) Changing effective priority from 120 to 100
May 14 23:59:27 systemd: haproxy.service: main process exited, code=exited, status=134/n/a
May 14 23:59:27 systemd: Unit haproxy.service entered failed state.
May 14 23:59:27 systemd: haproxy.service failed.
May 14 23:59:27 abrt-server: Executable '/usr/local/haproxy/sbin/haproxy' doesn't belong to any package and ProcessUnpackaged is set to 'no'
May 14 23:59:27 abrt-server: 'post-create' on '/var/spool/abrt/ccpp-2024-05-14-23:59:26-11379' exited with 1
May 14 23:59:27 abrt-server: Deleting problem directory '/var/spool/abrt/ccpp-2024-05-14-23:59:26-11379'

Expected Behavior

OpenSSL 1.0.2k-fips 26 Jan 2017

HAProxy version 2.7.6-4dadaaa 2023/03/28 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2024.
Known bugs: http://www.haproxy.org/bugs/bugs-2.7.6.html
Running on: Linux 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64

Steps to Reproduce the Behavior

  1. This is a sudden anomaly that has been running for more than 1 month

Do you have any idea what may have caused this?

No response

Do you have an idea how to solve the issue?

No response

What is your configuration?

global
    maxconn 200000
    #chroot /user/local/haproxy
    user root
    group root
    daemon
    pidfile /usr/local/haproxy/socket/haproxy.pid
    log 127.0.0.1 local2 info
    stats timeout 2m
    ssl-server-verify none
    tune.ssl.default-dh-param 2048

    stats socket /usr/local/haproxy/socket/stats mode 666 level admin

    #导入lua文件
    lua-prepend-path conf/lua/?.lua


        h1-case-adjust accept Accept
        h1-case-adjust authorization Authorization
        h1-case-adjust authrequired AuthRequired
        h1-case-adjust cache-control Cache-Control
        h1-case-adjust client-request-id Client-Request-Id
        h1-case-adjust connection Connection
        h1-case-adjust content-length Content-Length
        h1-case-adjust content-type Content-Type
        h1-case-adjust cookie Cookie
        h1-case-adjust date Date
        h1-case-adjust host Host
        h1-case-adjust persistent-auth Persistent-Auth
        h1-case-adjust pragma Pragma
        h1-case-adjust request-header Request-Header
        h1-case-adjust response-header Response-Header
        h1-case-adjust server Server
        h1-case-adjust set-cookie Set-Cookie
        h1-case-adjust status-code Status-Code
        h1-case-adjust transfer-encoding Transfer-Encoding
        h1-case-adjust user-agent User-Agent
        h1-case-adjust www-authenticate WWW-Authenticate
        h1-case-adjust x-anchormailbox X-AnchorMailbox
        h1-case-adjust x-clientapplication X-ClientApplication
        h1-case-adjust x-clientInfo X-ClientInfo
        h1-case-adjust x-content-type-options X-Content-Type-Options
        h1-case-adjust x-deviceinfo X-DeviceInfo
        h1-case-adjust x-elapsedtime X-ElapsedTime
        h1-case-adjust x-expirationinfo X-ExpirationInfo
        h1-case-adjust x-feserver X-FEServer
        h1-case-adjust x-mapihttpcapability X-MapiHttpCapability
        h1-case-adjust x-pendingperiod X-PendingPeriod
        h1-case-adjust x-powered-by X-Powered-By
        h1-case-adjust x-requestid X-RequestId
        h1-case-adjust x-requesttype X-RequestType
        h1-case-adjust x-responsecode X-ResponseCode
        h1-case-adjust x-serverapplication X-ServerApplication
        h1-case-adjust x-starttime X-StartTime
        h1-case-adjust x-user-identity X-User-Identity


defaults
    option                  dontlognull
    option                  redispatch
    option http-keep-alive
    retries                 3
    timeout http-request    900s
    timeout queue           1m
    timeout connect         10s
    timeout client          15m # this value should be rather high with Exchange
    timeout server          15m # this value should be rather high with Exchange
    timeout http-keep-alive 15m
    timeout check           10s
    maxconn                 200000
    #bind-process 1 
listen stats 
    mode http
    bind *:8999
    stats enable
    #stats admin if TRUE
    stats refresh 30s
    stats uri /stats
    stats realm Haproxy\ Statistics
    stats auth haadmin:111111
    stats   hide-version

frontend mail_587_tcp
  log 127.0.0.1 local3 info 
  bind *:587
  mode tcp
  option tcplog
  tcp-request content accept if { src -f /usr/local/haproxy/conf/white_list.txt }
  tcp-request content reject
  use_backend ex_tcp_587
  
  

backend ex_tcp_587
  mode tcp
  balance source
hash-type consistent

  server ex_80_12 :587 check weight 10 inter 3s
  server ex_80_30 :587 check weight 10 inter 3s
  server ex_80_31 :587 check weight 10 inter 3s
  

frontend fe_ex2019_http
  #window mapi must config
  option h1-case-adjust-bogus-client
  option                  httplog
  acl blacklist src -f /usr/local/haproxy/conf/blacklist.txt
  http-request deny if blacklist
  http-response set-header X-Frame-Options SAMEORIGIN
  http-response set-header X-Content-Type-Options nosniff
  option forwardfor 
  option http-keep-alive
  #http-request set-header X-Real-IP %[src]
  mode http
  bind *:80

  default_backend be_ex2019_http
  option                  forwardfor   header X-Forwarded-For # header X-Client-IP #转发客户IP到后端服务器
  option http-buffer-request
  log 127.0.0.1 local2 info
  log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %fi"

frontend fe_ex2019
    option h1-case-adjust-bogus-client
  option                  httplog
  option                  forwardfor   header X-Forwarded-For # header X-Client-IP #转发客户IP到后端服务器
  option http-buffer-request
  acl blacklist src -f /usr/local/haproxy/conf/blacklist.txt
  http-request deny if blacklist
  #http-response set-header X-Frame-Options SAMEORIGIN
  #http-response set-header X-Content-Type-Options nosniff
  option forwardfor 
  #http-request set-header X-Real-IP %[src]
  mode http
  option http-keep-alive
  bind *:443 ssl crt /usr/local/haproxy/conf/mail.didiglobal.com.pem 
  #redirect scheme https code 301 if !{ ssl_fc }   # redirect 80 -> 443 (for owa)
  
  acl autodiscover url_beg /Autodiscover
  acl autodiscover url_beg /autodiscover
  acl mapi url_beg /mapi
  acl mapi url_beg /MAPI
  acl rpc url_beg /rpc
  acl owa url_beg /owa
  acl owa url_beg /OWA
  acl eas url_beg /Microsoft-Server-ActiveSync
  acl eas url_beg /Microsoft-Server-activeSync
  acl ecp url_beg /ecp
  acl ews url_beg /EWS
  acl ews url_beg /ews
  acl oab url_beg /OAB

  use_backend be_ex2019_autodiscover if autodiscover
  use_backend be_ex2019_mapi if mapi
  use_backend be_ex2019_rpc if rpc
  use_backend be_ex2019_owa if owa
  use_backend be_ex2019_eas if eas
  use_backend be_ex2019_ecp if ecp
  use_backend be_ex2019_ews if ews
  use_backend be_ex2019_oab if oab


  default_backend be_ex2019
  log 127.0.0.1 local2 info
  log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %fi"


backend be_ex2019_autodiscover
  mode http
    option http-keep-alive
    option prefer-last-server
  balance source
hash-type consistent

    stick on src
    stick-table type ip size 64m expire 15m
  server ex_80_12 :443 check weight 10 ssl inter 3s
  server ex_80_30 :443 check weight 10 ssl inter 3s
  server ex_80_31 :443 check weight 10 ssl inter 3s
  

backend be_ex2019_mapi
  mode http
    option http-keep-alive
    option prefer-last-server
  balance source
hash-type consistent

    stick on src
    stick-table type ip size 64m expire 15m
  server ex_80_12 :443 check weight 10 ssl inter 3s
  server ex_80_30 :443 check weight 10 ssl inter 3s
  server ex_80_31 :443 check weight 10 ssl inter 3s
  

backend be_ex2019_rpc
  mode http
    option http-keep-alive
    option prefer-last-server
  balance source
hash-type consistent

    stick on src
    stick-table type ip size 64m expire 15m
  server ex_80_12 :443 check weight 10 ssl inter 3s
  server ex_80_30 :443 check weight 10 ssl inter 3s
  server ex_80_31 :443 check weight 10 ssl inter 3s
  

backend be_ex2019_owa
  mode http
    option http-keep-alive
    option prefer-last-server
  balance source
hash-type consistent

    stick on src
    stick-table type ip size 64m expire 15m
  server ex_80_12 :443 check weight 10 ssl inter 3s
  server ex_80_30 :443 check weight 10 ssl inter 3s
  server ex_80_31 :443 check weight 10 ssl inter 3s

  

backend be_ex2019_eas
  mode http
    option http-keep-alive
    option prefer-last-server
  balance source
hash-type consistent

    stick on src
    stick-table type ip size 64m expire 15m
  server ex_80_12 :443 check weight 10 ssl inter 3s
  server ex_80_30 :443 check weight 10 ssl inter 3s
  server ex_80_31 :443 check weight 10 ssl inter 3s
  

backend be_ex2019_ecp
  mode http
    option http-keep-alive
    option prefer-last-server
  balance source
hash-type consistent

    stick on src
    stick-table type ip size 64m expire 15m
  server ex_80_12 :443 check weight 10 ssl inter 3s
  server ex_80_30 :443 check weight 10 ssl inter 3s
  server ex_80_31 :443 check weight 10 ssl inter 3s
  

backend be_ex2019_ews
  mode http
    option http-keep-alive
    option prefer-last-server
  balance source
hash-type consistent

    stick on src
    stick-table type ip size 64m expire 15m
  server ex_80_12 :443 check weight 10 ssl inter 3s
  server ex_80_30 :443 check weight 10 ssl inter 3s
  server ex_80_31 :443 check weight 10 ssl inter 3s
  

backend be_ex2019_oab
  mode http
    option http-keep-alive
    option prefer-last-server
  balance source
hash-type consistent

    stick on src
    stick-table type ip size 64m expire 15m
  server ex_80_12 :443 check weight 10 ssl inter 3s
  server ex_80_30 :443 check weight 10 ssl inter 3s
  server ex_80_31 :443 check weight 10 ssl inter 3s

backend be_ex2019
  mode http
    option http-keep-alive
    option prefer-last-server
  balance source
hash-type consistent

    stick on src
    stick-table type ip size 64m expire 15m
  server ex_80_12 :443 check weight 10 ssl inter 3s
  server ex_80_30 :443 check weight 10 ssl inter 3s
  server ex_80_31 :443 check weight 10 ssl inter 3s
  

backend be_ex2019_http
  mode http
    option http-keep-alive
    option prefer-last-server 
  balance source
hash-type consistent

    stick on src
    stick-table type ip size 64m expire 15m
  server ex_80_12 :80 check weight 10 inter 3s
  server ex_80_30 :80 check weight 10 inter 3s
  server ex_80_31 :80 check weight 10 inter 3s

Output of haproxy -vv

HAProxy version 2.7.6-4dadaaa 2023/03/28 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2024.
Known bugs: http://www.haproxy.org/bugs/bugs-2.7.6.html
Running on: Linux 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -m64 -march=x86-64 -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_PCRE=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_CPU_AFFINITY=1 USE_SYSTEMD=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE +LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY +LUA -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT +PCRE -PCRE2 -PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL -PROMEX -PTHREAD_EMULATION -QUIC +RT +SHM_OPEN -SLZ -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL +ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=16).
Built with OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.6
Built with network namespace support.
Support for malloc_trim() is enabled.
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 4.8.5 20150623 (Red Hat 4.8.5-44)

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : none

Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

Last Outputs and Backtraces

No response

Additional Information

No response
@Mark11122 Mark11122 added status: needs-triage This issue needs to be triaged. type: bug This issue describes a bug. labels May 15, 2024
@git001
Copy link
Contributor

git001 commented May 15, 2024

Any chance to use the latest 2.7 version?

@capflam
Copy link
Member

capflam commented May 15, 2024

The 2.7 is EOL now. It is a better choice to switch to 2.8 instead.

@Mark11122
Copy link
Author

We'll try to use version 2.8, but it's a longer process, and now this issue can't be fixed anytime soon

@wtarreau
Copy link
Member

OpenSSL 1.0.2 was not good in terms of threading and here it crashed in lock contention inside OpenSSL. I guess that if you're running on 1.0.2 it's because you have a low load, so the best thing to do with this version of OpenSSL is to significantly reduce the number of threads (e.g. 4) using "nbthread 4" for example in the global section. Of course you need to test carefully because you never know what the low limit will be for your workload without testing. In any case your process is not behaving well, it died because some operations couldn't make any progress for two full seconds, that's an eternity at this level! If you can rebuild with at least 1.1.1 (and an up-to-date haproxy version), you'll get a better experience.

@capflam capflam added status: feedback required The developers are waiting for a reply from the reporter. and removed status: needs-triage This issue needs to be triaged. labels May 21, 2024
@Mark11122
Copy link
Author

Thank you, I'm going to test the 2.8.9 version, and if the test is normal, I'm going to replace the 2.7.6 version

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
status: feedback required The developers are waiting for a reply from the reporter. type: bug This issue describes a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@capflam @git001 @wtarreau @Mark11122