Build HAProxy 2.6-dev with QUIC Support?

[cmason3]

Is it possible to build the 2.6-dev Docker images with HTTP/3 (QUIC) support as the current 2.6-dev8 doesn't allow you to bind using quic4 or quic6?

[tianon]

Any idea what we need to do in the build in order to enable this? (Missing dependencies, build configuration, etc.)

[TimWolla]

Any idea what we need to do in the build in order to enable this? (Missing dependencies, build configuration, etc.)

https://github.com/haproxy/haproxy/blob/63fc900ba282437b96868103f1eb7db9ee7f482c/INSTALL#L395-L417

[tianon]

Ah 😬

I was just about to open a PR (0dc7b07...cd833ec) and then noticed the "QUICTLS" (fork of OpenSSL) bit there. 🙈

[cmason3]

Yeh, it seems OpenSSL can't agree with the rest of the community on how best to implement QUIC support in OpenSSL (openssl/openssl#8797 (comment)) so the current upstream got forked as QuicTLS (https://github.com/quictls/openssl) which is required to build QUIC support in HAProxy :(

[cmason3]

Apologies as this request doesn't seem to be that simple. It requires the QuicTLS port of OpenSSL to be compiled from source and then used to build HAProxy. I am not sure I really like the idea of building HAProxy Docker images against a port of OpenSSL for a feature that the majority of the users won't be using in the short term.

Anyway, I managed to get a working Dockerfile using the following modifications:

@@ -24,6 +24,8 @@
 ENV HAPROXY_URL https://www.haproxy.org/download/2.6/src/devel/haproxy-2.6-dev9.tar.gz
 ENV HAPROXY_SHA256 8d8a068350819cb74599d14f742340bec99f16fef3fcabc157b1ba12632bc424

+ENV QUICTLS_URL https://codeload.github.com/quictls/openssl/tar.gz/OpenSSL_1_1_1n+quic
+
 # see https://sources.debian.net/src/haproxy/jessie/debian/rules/ for some helpful navigation of the possible "make" arguments
 RUN set -eux; \
        \
@@ -35,11 +37,25 @@
                liblua5.3-dev \
                libpcre2-dev \
                libssl-dev \
+                perl \
                make \
                wget \
        ; \
        rm -rf /var/lib/apt/lists/*; \
        \
+        wget -O quictls.tar.gz "$QUICTLS_URL"; \
+        mkdir -p /usr/src/quictls; \
+        tar -xzf quictls.tar.gz -C /usr/src/quictls --strip-components=1; \
+        rm quictls.tar.gz; \
+        cd /usr/src/quictls; \
+        ./config --libdir=lib --prefix=/opt/quictls; \
+        make; \
+        make install; \
+        cp /opt/quictls/lib/libcrypto.so /usr/lib/; \
+        cp /opt/quictls/lib/libssl.so /usr/lib/; \
+        cd -; \
+        rm -rf /usr/src/quictls; \
+        \
        wget -O haproxy.tar.gz "$HAPROXY_URL"; \
        echo "$HAPROXY_SHA256 *haproxy.tar.gz" | sha256sum -c; \
        mkdir -p /usr/src/haproxy; \
@@ -50,7 +66,10 @@
                TARGET=linux-glibc \
                USE_GETADDRINFO=1 \
                USE_LUA=1 LUA_INC=/usr/include/lua5.3 \
+                USE_QUIC=1 \
                USE_OPENSSL=1 \
+                SSL_INC=/opt/quictls/include SSL_LIB=/opt/quictls/lib \
+                LDFLAGS="-Wl,-rpath,/opt/quictls/lib" \
                USE_PCRE2=1 USE_PCRE2_JIT=1 \
                USE_PROMEX=1 \
                \

I am not sure we are at the point where it is a good idea to impose the QuicTLS port of OpenSSL on everyone who uses HAProxy - maybe introduce a "-quic" label? The future of QUIC support in OpenSSL seems to be a long way off being resolved :(

[steschuser]

Yay for -quic label

[uvjustin]

Just a side note, 2.6.0 was released a few days back and it now officially supports QUIC, so this feature will probably get adopted by more users.
+1 for the -quic label.
Edit: QUIC support is still technically experimental, but it features prominently in the 2.6.0 blog post and is called the "star of this release" on the community edition webpage.

[JuniorJPDJ]

I'm using this:

# Original source:
#   https://github.com/docker-library/haproxy/blob/fc61eb99aa22c59229b212d83a3c9096a442df2c/2.6/alpine/Dockerfile
# Modified using:
#   https://git.alpinelinux.org/aports/tree/main/openssl3/APKBUILD
#   https://github.com/haproxytech/haproxy-qns/blob/master/Dockerfile

# First build quictls (OpenSSL fork with QUIC support)
FROM alpine:3.16 AS builder-ssl

RUN apk add gcc make perl linux-headers libc-dev \
  ; \
  wget https://github.com/quictls/openssl/archive/refs/heads/openssl-3.0.5+quic.tar.gz ; \
  tar -xf openssl-3.0.5+quic.tar.gz ; \
  cd openssl-openssl-3.0.5-quic \
  ; \
  perl ./Configure \
    linux-x86_64 \
    --prefix=/usr/local \
    --libdir=lib \
    --openssldir=/etc/ssl3 \
    shared \
    no-zlib \
    no-async \
    no-comp \
    no-idea \
    no-mdc2 \
    no-rc5 \
    no-ec2m \
    no-sm2 \
    no-sm4 \
    no-ssl3 \
    no-seed \
    no-weak-ssl-ciphers \
    enable-ec_nistp_64_gcc_128 \
    -Wa,--noexecstack \
  ; \
  make -j"$(getconf _NPROCESSORS_ONLN)" \
  ; \
  make install_sw


FROM alpine:3.16

# roughly, https://git.alpinelinux.org/aports/tree/main/haproxy/haproxy.pre-install?h=3.12-stable
RUN set -eux; \
	addgroup --gid 99 --system haproxy; \
	adduser \
		--disabled-password \
		--home /var/lib/haproxy \
		--ingroup haproxy \
		--no-create-home \
		--system \
		--uid 99 \
		haproxy \
	; \
	mkdir /var/lib/haproxy; \
	chown haproxy:haproxy /var/lib/haproxy

ENV HAPROXY_VERSION 2.6.1
ENV HAPROXY_URL https://www.haproxy.org/download/2.6/src/haproxy-2.6.1.tar.gz
ENV HAPROXY_SHA256 915b351e6450d183342c4cdcda7771eac4f0f72bf90582adcd15a01c700d29b1

COPY --from=builder-ssl /usr/local/include/openssl/ /usr/local/include/openssl/
COPY --from=builder-ssl /usr/local/lib/libssl.so* /usr/local/lib/libcrypto.so* /usr/local/lib/

# see https://sources.debian.net/src/haproxy/jessie/debian/rules/ for some helpful navigation of the possible "make" arguments
RUN set -eux; \
	\
	apk add --no-cache --virtual .build-deps \
		gcc \
		libc-dev \
		linux-headers \
		lua5.3-dev \
		make \
		pcre2-dev \
		readline-dev \
		tar \
	; \
	\
	wget -O haproxy.tar.gz "$HAPROXY_URL"; \
	echo "$HAPROXY_SHA256 *haproxy.tar.gz" | sha256sum -c; \
	mkdir -p /usr/src/haproxy; \
	tar -xzf haproxy.tar.gz -C /usr/src/haproxy --strip-components=1; \
	rm haproxy.tar.gz; \
	\
	makeOpts=' \
		TARGET=linux-musl \
		USE_GETADDRINFO=1 \
		USE_LUA=1 LUA_INC=/usr/include/lua5.3 LUA_LIB=/usr/lib/lua5.3 \
		USE_OPENSSL=1 \
    USE_QUIC=1 \
    SSL_INC=/usr/local/include/ \
    SSL_LIB=/usr/local/lib/ \
		USE_PCRE2=1 USE_PCRE2_JIT=1 \
		USE_PROMEX=1 \
		\
		EXTRA_OBJS=" \
		" \
	'; \
	\
	nproc="$(getconf _NPROCESSORS_ONLN)"; \
	eval "make -C /usr/src/haproxy -j '$nproc' all $makeOpts"; \
	eval "make -C /usr/src/haproxy install-bin $makeOpts"; \
	\
	mkdir -p /usr/local/etc/haproxy; \
	cp -R /usr/src/haproxy/examples/errorfiles /usr/local/etc/haproxy/errors; \
	rm -rf /usr/src/haproxy; \
	\
	runDeps="$( \
		scanelf --needed --nobanner --format '%n#p' --recursive /usr/local \
			| tr ',' '\n' \
			| sort -u \
			| awk 'system("[ -e /usr/local/lib/" $1 " ]") == 0 { next } { print "so:" $1 }' \
	)"; \
	apk add --no-network --virtual .haproxy-rundeps $runDeps; \
	apk del --no-network .build-deps; \
	\
# smoke test
	haproxy -v

# https://www.haproxy.org/download/1.8/doc/management.txt
# "4. Stopping and restarting HAProxy"
# "when the SIGTERM signal is sent to the haproxy process, it immediately quits and all established connections are closed"
# "graceful stop is triggered when the SIGUSR1 signal is sent to the haproxy process"
STOPSIGNAL SIGUSR1

COPY docker-entrypoint.sh /usr/local/bin/
ENTRYPOINT ["docker-entrypoint.sh"]

USER haproxy
CMD ["haproxy", "-f", "/usr/local/etc/haproxy/haproxy.cfg"]

Anyway I ended disabling QUIC as it tends to freeze whole listener when some bigger stream appears.

[cmason3]

QUIC support is fairly stable in 2.6 now - is there going to be a Docker image with QUIC support or is this currently too difficult with the OpenSSL mess?

Waiting for a mainstream OpenSSL compatible library within Debian is probably years away knowing how slow they are to adopt new stuff in their stable releases.

[tianon]

I would love to see it in Debian (semi-actively discussed in https://bugs.debian.org/1011391, FWIW), but I'm not as worried about that if this is something that the HAproxy project actively recommends/suggests using. It's not ideal to have to build something like OpenSSL from source, but it's not unprecedented (and certainly not a direct blocker by itself). The part that makes me a lot more nervous/hesitant is the relatively young fork of OpenSSL that's required.

Reading through https://quictls.github.io/ (and https://daniel.haxx.se/blog/2021/10/25/the-quic-api-openssl-will-not-provide/ or other similar posts about the situation) make me feel a bit better about the fork itself, but it's still a very complicated situation. It doesn't look like there's a great place to subscribe to updates on the QuickTLS project, such as when they do new releases or updates on the OpenSSL situation.

Another interesting/complicated point is security vulnerability tracking -- it's not far-fetched to imagine new vulnerabilities that are specific to the QUIC implementation or even vulnerabilities that only exhibit when the QUIC patches are applied, and because it's not a full-blown fork (and is faithfully tracking OpenSSL releases, re-applying the QUIC patches on top) it's thus somewhat more difficult to track/categorize those.

The OpenSSL 3.0.6 (security-related) release this week is honestly a really great example of one way that this complexity manifests, specifically because it was later redacted. As a result, it's now hard to automate updates to our QuickTLS version because they've chosen to do releases purely as GitHub branches/tags, which are somewhat less "acceptable" to redact like the link on "openssl.org" was for the 3.0.6 upstream release (https://github.com/openssl/openssl/tags still rightfully includes tags for 3.0.6, but https://www.openssl.org/source/ links to 3.0.5 and there was an official "OpenSSL 3.0.6 and 1.1.1r are withdrawn." announcement sent to the OpenSSL announcement list / noted on https://www.openssl.org/news/newslog.html, which makes the situation very clear).

[tianon]

An even shorter version of that comment would be that we're not waiting for Debian, but rather that we have a lot of similar concerns as Debian does in adopting something like this (because they're reasonable concerns to have in light of the very complicated situation the OpenSSL project has helped create).

[uvjustin]

Just an update for anyone looking - haproxytech has their own QUIC images built with quictls: haproxytech/haproxy-docker-alpine#6
https://hub.docker.com/r/haproxytech/haproxy-alpine-quic

[cmason3]

Just an update for anyone looking - haproxytech has their own QUIC images built with quictls: haproxytech/haproxy-docker-alpine#6
https://hub.docker.com/r/haproxytech/haproxy-alpine-quic

I have moved to using this permanently as it just works.

[tianon]

https://github.com/haproxy/wiki/wiki/SSL-Libraries-Support-Status seems to be the most accurate "current state" and it's still pretty bleak 😞

Given WolfSSL is actually in Debian and the positive way it's spoken about on that page, it's probably our best hope, but that long list of downsides we'd be enforcing on users doesn't seem ideal. 😩

(I wasn't really considering QuicTLS because it would still have the performance downsides of OpenSSL 3 and QUIC isn't the only feature I'm solving for 😅)

[linuxd3v]

@tianon I've been using this image that includes QUIC out of the box for like 6-7 months already and it's great!
https://hub.docker.com/r/haproxytech/haproxy-alpine-quic