Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

QUIC: segfault in qcc_recv_stop_sending #2563

Open
Tristan971 opened this issue May 7, 2024 · 6 comments
Open

QUIC: segfault in qcc_recv_stop_sending #2563

Tristan971 opened this issue May 7, 2024 · 6 comments
Labels
status: needs-triage This issue needs to be triaged. type: bug This issue describes a bug.

Comments

@Tristan971
Copy link
Member

Tristan971 commented May 7, 2024 •
edited

Detailed Description of the Problem

HAProxy crashes

Expected Behavior

No crash

Steps to Reproduce the Behavior

  1. Use QUIC

Do you have any idea what may have caused this?

No response

Do you have an idea how to solve the issue?

No response

What is your configuration?

as usual

Output of haproxy -vv

HAProxy version 3.0-dev10-fbbc292+mangadex-beeead9 2024-05-07T17:39+00:00 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 5.15.126-1-pve #1 SMP PVE 5.15.126-1 (2023-10-03T17:24Z) x86_64
Build options :
  TARGET  = linux-glibc
  CC      = cc
  CFLAGS  = -O2 -g -ggdb3 -gdwarf-4 -fwrapv -DMAX_SESS_STKCTR=5
  OPTIONS = USE_LIBCRYPT=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_NS=1 USE_SYSTEMD=1 USE_QUIC=1 USE_PROMEX=1 USE_STATIC_PCRE2=1 USE_PCRE2=1 USE_PCRE2_JIT=1
  DEBUG   = -DDEBUG_MEMORY_POOLS -DDEBUG_STRICT

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE +STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=8).
Built with OpenSSL version : OpenSSL 1.1.1w+quic-mangadex-beeead9  7 May 2024
Running on OpenSSL version : OpenSSL 1.1.1w+quic-mangadex-beeead9  7 May 2024
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.4.6
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.42 2022-12-11
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with clang compiler version 17.0.6 (++20231208085813+6009708b4367-1~exp1~20231208085906.81)

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE     mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=

Available services : prometheus-exporter
Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] compression
	[FCGI] fcgi-app
	[SPOE] spoe
	[TRACE] trace

Last Outputs and Backtraces

Core was generated by `/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -dMno-mer'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000055672c77b0f0 in qcc_recv_stop_sending (qcc=0x7f9ac47149c0, id=<optimized out>, err=268) at src/mux_quic.c:1597
1597	src/mux_quic.c: No such file or directory.
[Current thread is 1 (Thread 0x7f9ad8715a80 (LWP 173150))]
(gdb) t a a bt full

Thread 8 (Thread 0x7f9ad7dff640 (LWP 173151)):
#0  0x00007f9ad8a8de2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x000055672c7478e7 in _do_poll (p=<optimized out>, exp=-135, wake=0) at src/ev_epoll.c:232
        timeout = 11
        updt_idx = <optimized out>
        fd = <optimized out>
        old_fd = <optimized out>
        wait_time = 11
        status = <optimized out>
        count = <optimized out>
#2  0x000055672c8f17ef in run_poll_loop () at src/haproxy.c:3145
        _ = {
          func = 0x55672cc6b73b "run_poll_loop",
          file = 0x55672cc6b749 "src/haproxy.c",
          line = 3104,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = -687855104
#3  0x000055672c8f5d10 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3287
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            __wseq = {
              __value64 = 51,
              __value32 = {
                __low = 51,
                __high = 0
              }
            },
            __g1_start = {
              __value64 = 39,
              __value32 = {
                __low = 39,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 24,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = "3\000\000\000\000\000\000\000'", '\000' <repeats 23 times>, "\030", '\000' <repeats 14 times>,
          __align = 51
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#4  0x00007f9ad89fcac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x00007f9ad8a8e850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 7 (Thread 0x7f9ac20c3640 (LWP 173153)):
#0  0x00007f9ad8a8de2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x000055672c7478e7 in _do_poll (p=<optimized out>, exp=-135, wake=0) at src/ev_epoll.c:232
        timeout = 11
        updt_idx = <optimized out>
        fd = <optimized out>
        old_fd = <optimized out>
        wait_time = 11
        status = <optimized out>
        count = <optimized out>
#2  0x000055672c8f17ef in run_poll_loop () at src/haproxy.c:3145
        _ = {
          func = 0x55672cc6b73b "run_poll_loop",
          file = 0x55672cc6b749 "src/haproxy.c",
          line = 3104,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = -1112330240
#3  0x000055672c8f5d10 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3287
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            __wseq = {
              __value64 = 51,
              __value32 = {
                __low = 51,
                __high = 0
              }
            },
            __g1_start = {
              __value64 = 39,
              __value32 = {
                __low = 39,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 24,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = "3\000\000\000\000\000\000\000'", '\000' <repeats 23 times>, "\030", '\000' <repeats 14 times>,
          __align = 51
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#4  0x00007f9ad89fcac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x00007f9ad8a8e850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 6 (Thread 0x7f9ac08c0640 (LWP 173156)):
#0  0x00007f9ad8a8de2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x000055672c7478e7 in _do_poll (p=<optimized out>, exp=-135, wake=0) at src/ev_epoll.c:232
        timeout = 14
        updt_idx = <optimized out>
        fd = <optimized out>
        old_fd = <optimized out>
        wait_time = 14
        status = <optimized out>
        count = <optimized out>
#2  0x000055672c8f17ef in run_poll_loop () at src/haproxy.c:3145
        _ = {
          func = 0x55672cc6b73b "run_poll_loop",
          file = 0x55672cc6b749 "src/haproxy.c",
          line = 3104,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = -1103941632
#3  0x000055672c8f5d10 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3287
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            __wseq = {
              __value64 = 51,
              __value32 = {
                __low = 51,
                __high = 0
              }
            },
            __g1_start = {
              __value64 = 39,
              __value32 = {
                __low = 39,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 24,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = "3\000\000\000\000\000\000\000'", '\000' <repeats 23 times>, "\030", '\000' <repeats 14 times>,
          __align = 51
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#4  0x00007f9ad89fcac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x00007f9ad8a8e850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 5 (Thread 0x7f9ac28c4640 (LWP 173152)):
#0  0x00007f9ad8a8de2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x000055672c7478e7 in _do_poll (p=<optimized out>, exp=-135, wake=0) at src/ev_epoll.c:232
        timeout = 19
        updt_idx = <optimized out>
        fd = <optimized out>
        old_fd = <optimized out>
        wait_time = 19
        status = <optimized out>
        count = <optimized out>
#2  0x000055672c8f17ef in run_poll_loop () at src/haproxy.c:3145
        _ = {
          func = 0x55672cc6b73b "run_poll_loop",
          file = 0x55672cc6b749 "src/haproxy.c",
          line = 3104,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = -1085067264
#3  0x000055672c8f5d10 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3287
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            __wseq = {
              __value64 = 51,
              __value32 = {
                __low = 51,
                __high = 0
              }
            },
            __g1_start = {
              __value64 = 39,
              __value32 = {
                __low = 39,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 24,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = "3\000\000\000\000\000\000\000'", '\000' <repeats 23 times>, "\030", '\000' <repeats 14 times>,
          __align = 51
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#4  0x00007f9ad89fcac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x00007f9ad8a8e850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 4 (Thread 0x7f9ac18c2640 (LWP 173154)):
#0  http_parse_authority (parser=0x7f9ac18b55d8, no_userinfo=1) at src/http.c:660
        start = <optimized out>
        ptr = 0x7f9abb0153d7 "adex.org/manga/dc332d04-d3b0-413c-a767-70f5e451b031/feed?includes%5B%5D=scanlation_group&includes%5B%5D=user&limit=500&offset=0&translatedLanguage%5B%5D=en&order%5Bvolume%5D=desc&order%5Bchapter%5D=desc&includeFuturePublishAt=0&includeEmptyPages=0&contentRating%5B%5D=safe&contentRating%5B%5D=suggestive&contentRating%5B%5D=erotica&contentRating%5B%5D=pornographicHTTP/2.0hostapi.mangadex.orgrefererhttps://mangadex.org/extraAndroid/14 Tachiyomi/0.15.2 MangaDex/1.4.190cache-controlno-cacheaccept-encodingbr,gzip"...
        end = 0x7f9abb015543 "HTTP/2.0hostapi.mangadex.orgrefererhttps://mangadex.org/extraAndroid/14 Tachiyomi/0.15.2 MangaDex/1.4.190cache-controlno-cacheaccept-encodingbr,gzipuser-agentTachiyomi Dalvik/2.1.0 (Linux; U; Android 14; SM-G998N Build/UP1A.231005.007)cookie_ga_N0GP6FDRPH=GS1.1.1712610796.48.1.1712610800.56.0.0; _ga=GA1.1.1011480249.1656623776gx-forwarded-for$redactedx-forwarded-hostapi.mangadex.orgx-forwarded-protohttpsforwardedby=EU-WEST-U2x1;for=$redacted;host=api.mangadex.org;proto=httpsvch3=\":443\"; ma=86400md-progr"...
        not_found = <optimized out>
#1  http_parse_path (parser=parser@entry=0x7f9ac18b55d8) at src/http.c:713
        ptr = <optimized out>
        end = <optimized out>
#2  0x000055672c908f9a in smp_fetch_path (args=<optimized out>, smp=0x7f9ac18b5690, kw=0x55672cc1c68c "path", private=<optimized out>) at src/http_fetch.c:1102
        parser = {
          uri = {
            ptr = 0x7f9abb0153cf "api.mangadex.org/manga/dc332d04-d3b0-413c-a767-70f5e451b031/feed?includes%5B%5D=scanlation_group&includes%5B%5D=user&limit=500&offset=0&translatedLanguage%5B%5D=en&order%5Bvolume%5D=desc&order%5Bchapter%5D=desc&includeFuturePublishAt=0&includeEmptyPages=0&contentRating%5B%5D=safe&contentRating%5B%5D=suggestive&contentRating%5B%5D=erotica&contentRating%5B%5D=pornographicHTTP/2.0hostapi.mangadex.orgrefererhttps://mangadex.org/extraAndroid/14 Tachiyomi/0.15.2 MangaDex/1.4.190cache-controlno-cacheaccept-encodin"...,
            len = 372
          },
          state = URI_PARSER_STATE_SCHEME_DONE,
          format = URI_PARSER_FORMAT_ABSURI_OR_AUTHORITY
        }
        chn = <optimized out>
        htx = <optimized out>
        sl = <optimized out>
        path = <optimized out>
#3  0x000055672c879842 in sample_process (px=px@entry=0x7f9ac47dac00, sess=sess@entry=0x7f9abcb46140, strm=strm@entry=0x7f9abcd7fc00, opt=opt@entry=6, expr=0x7f9ac4730bb0, p=p@entry=0x7f9ac18b5690) at src/sample.c:1370
No locals.
#4  0x000055672c95b77b in acl_exec_cond (cond=<optimized out>, px=px@entry=0x7f9ac47dac00, sess=sess@entry=0x7f9abcb46140, strm=strm@entry=0x7f9abcd7fc00, opt=6, opt@entry=2) at src/acl.c:1071
        smp = {
          flags = 0,
          data = {
            type = 0,
            u = {
              sint = 0,
              ipv4 = {
                s_addr = 0
              },
              ipv6 = {
                __in6_u = {
                  __u6_addr8 = '\000' <repeats 15 times>,
                  __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
                  __u6_addr32 = {0, 0, 0, 0}
                }
              },
              str = {
                size = 0,
                area = 0x0,
                data = 0,
                head = 0
              },
              meth = {
                meth = HTTP_METH_OPTIONS,
                str = {
                  size = 0,
                  area = 0x0,
                  data = 0,
                  head = 0
                }
              }
            }
          },
          ctx = {
            p = 0x0,
            i = 0,
            ll = 0,
            d = 0,
            a = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}
          },
          px = 0x7f9ac47dac00,
          sess = 0x7f9abcb46140,
          strm = 0x7f9abcd7fc00,
          opt = 6
        }
        cond_res = ACL_TEST_FAIL
        suite = 0x7f9ad6f950c0
        suite_res = ACL_TEST_PASS
        term = 0x7f9ad6f95080
        acl_res = ACL_TEST_FAIL
        expr = 0x7f9ad8225fa0
        acl = <optimized out>
#5  0x000055672c8542cc in http_req_get_intercept_rule (px=px@entry=0x7f9ac47dac00, def_rules=0x0, rules=0x7f9ac47dac48, s=s@entry=0x7f9abcd7fc00) at src/http_ana.c:2667
        ret = <optimized out>
        sess = 0x7f9abcb46140
        txn = 0x7f9abcc64e00
        rule_ret = HTTP_RULE_RES_CONT
        act_opts = 2
        rule = 0x7f9ad8222100
#6  0x000055672c85331e in http_process_req_common (s=0x7f9abcd7fc00, req=req@entry=0x7f9abcd7fc28, an_bit=an_bit@entry=16, px=0x7f9ac47dac00) at src/http_ana.c:387
        sess = 0x7f9abcb46140
        txn = 0x7f9abcc64e00
        msg = <optimized out>
        conn = 0x7f9ac359da00
        htx = 0x7f9abb015380
        def_rules = <optimized out>
        rules = 0x16d
        verdict = <optimized out>
        rule = <optimized out>
#7  0x000055672c844a51 in process_stream (t=t@entry=0x7f9abcd45000, context=context@entry=0x7f9abcd7fc00, state=<optimized out>) at src/stream.c:2021
        max_loops = 199
        ana_back = 48
        ana_list = 48
        scf_flags_ana = 1058
        scb_flags_ana = 33
        s = 0x7f9abcd7fc00
        sess = 0x7f9abcb46140
        req = 0x7f9abcd7fc28
        res = 0x7f9abcd7fc70
        scf = 0x7f9ab92b58c0
        scb = 0x7f9ab95fc6c0
        rate = <optimized out>
        rqf_last = <optimized out>
        rpf_last = 0
        scf_flags = 1058
        scb_flags = 33
        rq_prod_last = <optimized out>
        rq_cons_last = 0
        rp_prod_last = 0
        rp_cons_last = 8
        req_ana_back = <optimized out>
        res_ana_back = <optimized out>
        srv = <optimized out>
#8  0x000055672c92b68b in run_tasks_from_lists (budgets=budgets@entry=0x7f9ac18b5a00) at src/task.c:632
        _ = {
          func = 0x55672cc76c79 "run_tasks_from_lists",
          file = 0x55672cc76c8e "src/task.c",
          line = 657,
          what = 6 '\006',
          arg8 = 0 '\000',
          arg32 = 0
        }
        tl_queues = 0x55672d059aa0 <ha_thread_ctx+2208>
        budget_mask = 15 '\017'
        profile_entry = 0x0
        done = 0
        queue = 1
        t = 0x7f9abcd45000
        process = 0x55672c844240 <process_stream>
        ctx = 0x7f9abcd7fc00
        state = <optimized out>
#9  0x000055672c92c1fa in process_runnable_tasks () at src/task.c:876
        max = {0, 89, 0, 0}
        tt = 0x55672d059a00 <ha_thread_ctx+2048>
        default_weights = {64, 48, 16, 1}
        heavy_queued = 1
        max_processed = 90
        max_total = <optimized out>
        queue = 4
        budget = 90
        grq = <optimized out>
        lrq = <optimized out>
        gpicked = <optimized out>
        lpicked = <optimized out>
        t = <optimized out>
        tmp_list = <optimized out>
#10 0x000055672c8f1824 in run_poll_loop () at src/haproxy.c:3073
        _ = {
          func = 0x55672cc6b73b "run_poll_loop",
          file = 0x55672cc6b749 "src/haproxy.c",
          line = 3104,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = <optimized out>
#11 0x000055672c8f5d10 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3287
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            __wseq = {
              __value64 = 51,
              __value32 = {
                __low = 51,
                __high = 0
              }
            },
            __g1_start = {
              __value64 = 39,
              __value32 = {
                __low = 39,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 24,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = "3\000\000\000\000\000\000\000'", '\000' <repeats 23 times>, "\030", '\000' <repeats 14 times>,
          __align = 51
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#12 0x00007f9ad89fcac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#13 0x00007f9ad8a8e850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 3 (Thread 0x7f9ac10c1640 (LWP 173155)):
#0  0x00007f9ad8a8de2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x000055672c7478e7 in _do_poll (p=<optimized out>, exp=-135, wake=0) at src/ev_epoll.c:232
        timeout = 11
        updt_idx = <optimized out>
        fd = <optimized out>
        old_fd = <optimized out>
        wait_time = 11
        status = <optimized out>
        count = <optimized out>
#2  0x000055672c8f17ef in run_poll_loop () at src/haproxy.c:3145
        _ = {
          func = 0x55672cc6b73b "run_poll_loop",
          file = 0x55672cc6b749 "src/haproxy.c",
          line = 3104,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = -1099747328
#3  0x000055672c8f5d10 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3287
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            __wseq = {
              __value64 = 51,
              __value32 = {
                __low = 51,
                __high = 0
              }
            },
            __g1_start = {
              __value64 = 39,
              __value32 = {
                __low = 39,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 24,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = "3\000\000\000\000\000\000\000'", '\000' <repeats 23 times>, "\030", '\000' <repeats 14 times>,
          __align = 51
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#4  0x00007f9ad89fcac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x00007f9ad8a8e850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 2 (Thread 0x7f9ac00bf640 (LWP 173157)):
#0  0x00007f9ad8a8de2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x000055672c7478e7 in _do_poll (p=<optimized out>, exp=-135, wake=0) at src/ev_epoll.c:232
        timeout = 11
        updt_idx = <optimized out>
        fd = <optimized out>
        old_fd = <optimized out>
        wait_time = 11
        status = <optimized out>
        count = <optimized out>
#2  0x000055672c8f17ef in run_poll_loop () at src/haproxy.c:3145
        _ = {
          func = 0x55672cc6b73b "run_poll_loop",
          file = 0x55672cc6b749 "src/haproxy.c",
          line = 3104,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = -1108135936
#3  0x000055672c8f5d10 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3287
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            __wseq = {
              __value64 = 51,
              __value32 = {
                __low = 51,
                __high = 0
              }
            },
            __g1_start = {
              __value64 = 39,
              __value32 = {
                __low = 39,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 24,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = "3\000\000\000\000\000\000\000'", '\000' <repeats 23 times>, "\030", '\000' <repeats 14 times>,
          __align = 51
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#4  0x00007f9ad89fcac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x00007f9ad8a8e850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 1 (Thread 0x7f9ad8715a80 (LWP 173150)):
#0  0x000055672c77b0f0 in qcc_recv_stop_sending (qcc=0x7f9ac47149c0, id=<optimized out>, err=268) at src/mux_quic.c:1597
        qcs = 0x7f9ac2c2b000
#1  0x000055672c7abb48 in qc_parse_pkt_frms (qc=<optimized out>, pkt=<optimized out>, qel=<optimized out>) at src/quic_rx.c:898
        ss_frm = <optimized out>
        frm = <optimized out>
        pos = <optimized out>
        fast_retrans = 0
        ret = <optimized out>
        end = 0x7f9aba9dc07d "\377\305~\212\267\"\362\215T0\357^\025\216\356a@\357\r\016\225pCC\271\017\017\020E1D\352P\344\223T?\251\351\222[&N\236\331\365-\341\316\064\275\067M\327j\313\333\301\064\360&\f\236\177;\362[\226\ae\277]\262\217\234{\207\343e\374\357\355\305)\250\345\327!\370\352\301\346\335\376\355\303\323\025\331\251\361\326\377\204\356\310-\025ngBz\373\062a\277'|h\361\275\221^\331{\277O\306]g\a\233\270\333o\r\351&\325|4\346\237\367mX\270\335\325\247\315w\235>\027\273c\253\v\372\377v\340.3q\307T%\335\217\307\231\367\343\277\373\273\307\356>v\340:9\333\303@wX\375h\375\354\364\336c\221iy\326}\271\334\024\205.a-\204\336\237\301\336\237\307q\227\002\336\372d\216\277>9>\220q\204\372*\033\375\362\265]\033\037\016\302k\341\352\337\315\230yj\325\025>5\302\326\215\221\376;\313\220\265\235e\232\217\036O;r\335\262\365Y\221\253vLQv-\271(\025\377\026\222\350\304nm\037\266\254\r\307\205\363\337\305\370\353\375\341\355\v\217\066m\231s{~\252\331\366\364=\343\277\267e\205>|{\367\233F `k\264\333\370\242\303\356\267\205\334Mf\304\266\016\335\067\376\206\377\017\070\374E{6V\362\034\371\361\330\\\363\264\323>}im4^4yzY\356\360i\337\217\362\277;\211\027*J\377\255\233\356\317\323\303y\357?\327\004\231\352K\232\372\263\225j\226\v\337j\016\324\316\274I?R\325\021|\326k\006\223\036\245\350\071\aD\001q\320\211\307\336}\340"
        __x = <optimized out>
        __x = <optimized out>
        nb_streams = <optimized out>
        fin = <optimized out>
        strm_frm = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        rtt_sample = <optimized out>
        ack_delay = <optimized out>
        _a = <optimized out>
        _b = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        rs_frm = <optimized out>
        ss_frm = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        md_frm = <optimized out>
        msd_frm = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        conn_id = <optimized out>
        tree = <optimized out>
        __lk_r = <optimized out>
        __set_r = <optimized out>
        __msk_r = <optimized out>
        __pl_r = <optimized out>
        __ptr = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        iqel = <optimized out>
        hqel = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
#2  qc_treat_rx_pkts (qc=qc@entry=0x7f9ac35b3000) at src/quic_rx.c:1227
        pkt = 0x7f9ac2c2bc00
        ret = 0
        largest_pn = -1
        largest_pn_time_received = 0
        qel = 0x7f9ac4713840
        node = 0x7f9ac2c2bca0
        qelbak = <optimized out>
#3  0x000055672c76f626 in quic_conn_app_io_cb (t=t@entry=0x7f9ac2d8d1f0, context=context@entry=0x7f9ac35b3000, state=<optimized out>) at src/quic_conn.c:579
        send_list = {
          n = 0x7ffe863f1d20,
          p = 0x7ffe863f1d20
        }
        qc = <optimized out>
        qel = 0x7f9ac4713840
#4  0x000055672c92b852 in run_tasks_from_lists (budgets=budgets@entry=0x7ffe863f1e80) at src/task.c:596
        _ = {
          func = 0x55672cc76c79 "run_tasks_from_lists",
          file = 0x55672cc76c8e "src/task.c",
          line = 657,
          what = 6 '\006',
          arg8 = 0 '\000',
          arg32 = 0
        }
        tl_queues = 0x55672d0592a0 <ha_thread_ctx+160>
        budget_mask = 15 '\017'
        profile_entry = 0x0
        done = 0
        queue = 2
        t = 0x7f9ac2d8d1f0
        process = 0x55672c76f570 <quic_conn_app_io_cb>
        ctx = 0x7f9ac35b3000
        state = 0
#5  0x000055672c92c1fa in process_runnable_tasks () at src/task.c:876
        max = {0, 0, 92, 0}
        tt = 0x55672d059200 <ha_thread_ctx>
        default_weights = {64, 48, 16, 1}
        heavy_queued = 1
        max_processed = 93
        max_total = <optimized out>
        queue = 4
        budget = 0
        grq = <optimized out>
        lrq = <optimized out>
        gpicked = <optimized out>
        lpicked = <optimized out>
        t = <optimized out>
        tmp_list = <optimized out>
#6  0x000055672c8f1824 in run_poll_loop () at src/haproxy.c:3073
        _ = {
          func = 0x55672cc6b73b "run_poll_loop",
          file = 0x55672cc6b749 "src/haproxy.c",
          line = 3104,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = <optimized out>
#7  0x000055672c8f5d10 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3287
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            __wseq = {
              __value64 = 51,
              __value32 = {
                __low = 51,
                __high = 0
              }
            },
            __g1_start = {
              __value64 = 39,
              __value32 = {
                __low = 39,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 24,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = "3\000\000\000\000\000\000\000'", '\000' <repeats 23 times>, "\030", '\000' <repeats 14 times>,
          __align = 51
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#8  0x000055672c8f49ba in main (argc=<optimized out>, argv=0x7ffe863f23b8) at src/haproxy.c:3989
        limit = {
          rlim_cur = 18446744073709551615,
          rlim_max = 18446744073709551615
        }
        pidfd = <optimized out>
        retry = <optimized out>
        err = <optimized out>
        intovf = <optimized out>

Additional Information

No response
@Tristan971 Tristan971 added status: needs-triage This issue needs to be triaged. type: bug This issue describes a bug. labels May 7, 2024
@Tristan971
Copy link
Member Author

Tristan971 commented May 7, 2024 •
edited

I don't know if it's expected, but I note some application-level data now can show up in backtraces (don't think it was ever the case before); specifically in thread 4 here:

#0  http_parse_authority (parser=0x7f9ac18b55d8, no_userinfo=1) at src/http.c:660
        start = <optimized out>
        ptr = 0x7f9abb0153d7 "adex.org/manga/dc332d04-d3b0-413c-a767-70f5e451b031/feed?includes%5B%5D=scanlation_group&includes%5B%5D=user&limit=500&offset=0&translatedLanguage%5B%5D=en&order%5Bvolume%5D=desc&order%5Bchapter%5D=desc&includeFuturePublishAt=0&includeEmptyPages=0&contentRating%5B%5D=safe&contentRating%5B%5D=suggestive&contentRating%5B%5D=erotica&contentRating%5B%5D=pornographicHTTP/2.0hostapi.mangadex.orgrefererhttps://mangadex.org/extraAndroid/14 Tachiyomi/0.15.2 MangaDex/1.4.190cache-controlno-cacheaccept-encodingbr,gzip"...
        end = 0x7f9abb015543 "HTTP/2.0hostapi.mangadex.orgrefererhttps://mangadex.org/extraAndroid/14 Tachiyomi/0.15.2 MangaDex/1.4.190cache-controlno-cacheaccept-encodingbr,gzipuser-agentTachiyomi Dalvik/2.1.0 (Linux; U; Android 14; SM-G998N Build/UP1A.231005.007)cookie_ga_N0GP6FDRPH=GS1.1.1712610796.48.1.1712610800.56.0.0; _ga=GA1.1.1011480249.1656623776gx-forwarded-for$redactedx-forwarded-hostapi.mangadex.orgx-forwarded-protohttpsforwardedby=EU-WEST-U2x1;for=$redacted;host=api.mangadex.org;proto=httpsvch3=\":443\"; ma=86400md-progr"...
        not_found = <optimized out>
#1  http_parse_path (parser=parser@entry=0x7f9ac18b55d8) at src/http.c:713
        ptr = <optimized out>
        end = <optimized out>
#2  0x000055672c908f9a in smp_fetch_path (args=<optimized out>, smp=0x7f9ac18b5690, kw=0x55672cc1c68c "path", private=<optimized out>) at src/http_fetch.c:1102
        parser = {
          uri = {
            ptr = 0x7f9abb0153cf "api.mangadex.org/manga/dc332d04-d3b0-413c-a767-70f5e451b031/feed?includes%5B%5D=scanlation_group&includes%5B%5D=user&limit=500&offset=0&translatedLanguage%5B%5D=en&order%5Bvolume%5D=desc&order%5Bchapter%5D=desc&includeFuturePublishAt=0&includeEmptyPages=0&contentRating%5B%5D=safe&contentRating%5B%5D=suggestive&contentRating%5B%5D=erotica&contentRating%5B%5D=pornographicHTTP/2.0hostapi.mangadex.orgrefererhttps://mangadex.org/extraAndroid/14 Tachiyomi/0.15.2 MangaDex/1.4.190cache-controlno-cacheaccept-encodin"...,
            len = 372
          },
          state = URI_PARSER_STATE_SCHEME_DONE,
          format = URI_PARSER_FORMAT_ABSURI_OR_AUTHORITY
        }
        chn = <optimized out>
        htx = <optimized out>
        sl = <optimized out>
        path = <optimized out>

Which can include PII/secrets if it prints out sensitive headers like Cookie and XFF/Forwarded in this case.

Not that I personally mind that much (I'll just pay closer attention in the future), but I figure it was not by sheer luck that this never was the case in the past.

@Tristan971
Copy link
Member Author

nb: if it's being a big pain to figure out on your end, let me know, I can probably reasonably get traces in this case as it's really common

@a-denoyelle
Copy link
Contributor

Okay this part as changed recently. The fix should be easy enough here, no need for any traces thanks.

haproxy-mirror pushed a commit that referenced this issue May 10, 2024
@a-denoyelle
BUG/MEDIUM: mux-quic: fix crash on STOP_SENDING received without SD
cc9827b 
Abort reason code received on STOP_SENDING is notified to upper layer
since the following commit :
  367ce1e
  MINOR: mux-quic: Set tha SE abort reason when a STOP_SENDING frame is received

However, this causes a crash when a STOP_SENDING is received on a QCS
instance without any stream instantiated. Fix this by checking first if
qcs->sd is not NULL before setting abort code.

This bug can easily be reproduced by emitting a STOP_SENDING as first
frame of a stream.

This should fix github issue #2563.

This does not need to be backported.
@wtarreau
Copy link
Member

Regarding the risk of data in traces, it solely depends on where the code crashes and what gdb will print. Typically if one of the functions in the stack has a pointer argument to such a string or buffer for example and gdb decodes it, it will indeed appear. I think that you should be able to truncate the length of such strings in backtraces using "set print elements 10" (it should limit it to 10 chars). 0 is unlimited.

@Tristan971
Copy link
Member Author

Tristan971 commented May 11, 2024 •
edited

if one of the functions in the stack has a pointer argument to such a string or buffer for example and gdb decodes it, it will indeed appear

I guess so, but I was just surprised as I think it's the first time that happens so clearly in a HAProxy backtrace over the past years and however many backtraces I got :)

So I was (wrongfully) thinking there was some logic preventing it (printing L7 data, that is) at play.

I'll just keep an eye out for it in the future either way, it's fine 👍

@Tristan971
Copy link
Member Author

Btw I can confirm that the crash is fixed 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
status: needs-triage This issue needs to be triaged. type: bug This issue describes a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Tristan971 @wtarreau @a-denoyelle