docs: Feedback on listener_wrappers (mostly around proxy_protocol)

[polarathene]

Feedback on the docs on Global Caddyfile listener_wrappers setting, hope it's somewhat helpful!

Sorry about the verbosity, low on time 😭

Summary:

• auto_https implicit redirects seem broken with PROXY protocol support (see Traefik logs shared below)
• http_redirect may have a bug or would benefit from clearer communication of caveat.
• http_redirect should document placement order after proxy_protocol, similar to need for before tls.
• proxy_protocol could better clarify allow expectations (suggestion provided below), possibly improve implementation.
• JSON / module docs have some overlap of first vs third party proxy_protocol, could be better clarified in JSON docs.
• Link to PROXY protocol spec might be better suited to link to HAProxy Github master branch for the file instead? (1.8 was a 2016 release, despite the docs being updated in 2022)

http_redirect

http_redirect describes itself as important for redirecting HTTP to HTTPS when the request is arriving on an HTTPS port to avoid responding with "Client sent an HTTP request to an HTTPS server.".

• Seems that it needs to come after proxy_protocol to work correctly (at least for those connections matching allow).
• If there are explicit http:// site addresses: Following the docs example (instead of configuring separate servers per port) will make these routes inaccessible as the redirection is forced (like implicit redirects from autohttps).
  • That caveat isn't communicated however, with docs implying the feature is specifically for the HTTPS port but will still apply to a connection with http:// scheme and HTTP port. Should be no-op on the server using HTTP port?

---

proxy_protocol

https://caddyserver.com/docs/json/apps/http/servers/listener_wrappers/proxy_protocol/#allow

Allow is an optional list of CIDR ranges to allow/require PROXY headers from.

• When not provided, all connections appear to require PROXY protocol to be used?
• When explicit IPs (with /32 CIDR notation) or IP ranges are configured, anything else is not trusted for PROXY protocol connections, while the allowed IPs are required to provide the PROXY protocol header?

This is fine, just comparing to other software it might be helpful to clarify expectations?:

"Allow is an optional list of trusted CIDR ranges expected to provide PROXY headers. Untrusted IPs using PROXY protocol will be rejected, but may otherwise connect normally without PROXY protocol."

Reference - Comparison to other implementations

Here's what I've noticed from least flexible to most flexible:

• Postfix port will only accept connections with PROXY protocol, no support for trusted proxies.
• Dovecot port will only accept connections with PROXY protocol, but also mandates the proxy is added to the related trust setting.
• Caddy is the same as Dovecot, except flips the trust model to any host by default. When trusted IPs are configured, untrusted IPs can however still connect without PROXY protocol (but trusted IPs still cannot).
• Traefik entrypoints are flexible on connections (The port accepts connections regardless of PROXY header being present), but must configure either:
  • Trusted proxies / IPs can be set to restrict when to trust the PROXY header to avoid request forgery.
  • An insecure boolean to true for trusting any host using PROXY protocol.

I think Traefik has it right there, by enforcing communicating trust by default (a list of trusted IPs or toggle insecure) while allowing regular connections through that aren't using PROXY protocol? (other software is a bit inconvenient by requiring separate ports for internal clients)

Caddy can manage to handle this distinction well enough when the specific trusted IPs are known, but allow couldn't use a wider private range subnet like trusted_proxies due to trusted IPs enforcing the requirement for PROXY header (I'm having trouble understanding why that needs to be enforced, other than preventing misconfiguration from hosts that are expected to always send the PROXY header?).

---

Observation - Bugs?

When connecting to an HTTPS site address via curl and the following config:

{
  log default {
    level DEBUG
  }
  servers {
    listener_wrappers {
      proxy_protocol {
        allow 172.16.42.10/32
      }
      #http_redirect
      #tls
    }
  }
}

From the client IP 172.16.42.42 and Traefik (172.16.42.10):

• Direct connection (no Traefik) from curl to Caddy successfully responds.
• Indirect connection via Traefik to Caddy fails to passthrough the TLS connection, unless I uncomment tls.

The Caddy docs are rather clear about proxy_protocol needing to come before tls, so I assume the error is from the PROXY header being present via Traefik and the tls being implicitly before it 👍

That all makes sense, but if tls is uncommented, while HTTPS works correctly, HTTP fails with 400 Bad Request, Traefik wasn't able to connect to Caddy on port 80:

# Traefik log entries related to the connection, Caddy is `172.16.42.3`:
Handling TCP connection address=172.16.42.3:80 remoteAddr=172.16.42.42:45376
Error while terminating TCP connection error="close tcp 172.16.42.10:44528->172.16.42.3:80: use of closed network connection"
# Compared to error when `auto_https disable_redirects`:
Error while dialing backend error="dial tcp 172.16.42.3:80: connect: connection refused"

# NOTE: No related Caddy logs were output from these events, despite the debug log level.

• This does becomes successful when the Traefik configured TCP router avoids using PROXY protocol, or likewise using the equivalent HTTP router (which doesn't have PROXY protocol support; I was confused to find Caddy supporting it via Caddyfile HTTP server - but that's clarified early in "Server Options" section 👍 ).
• It's also successful if I instead add a separate http:// site address block. TCP router with PROXY protocol can be used then.

I know that by default the auto_https feature would have enabled the implicit redirects, and that http:// site block explicitly opts-out of that feature. I guess something there is not compatible the PROXY protocol support? 🤷‍♂️

I can also add http_redirect after proxy_protocol to restore that redirect behaviour for PROXY protocol connections to Caddy, it must be after proxy_protocol though to work. Alternatively in this scenario, Traefik could also handle HTTP => HTTPS, but Traefik isn't relevant here, only used to test support introduced in Caddy 2.7.

I don't have any need for PROXY protocol with HTTP/HTTPS myself, I have used it for TCP traffic for mail servers but was curious about the Caddy 2.7 support and how it compared to the equivalent support in Traefik. Not sure how it compares to the Caddy L4 app, I haven't tried that yet due to current JSON only support.

---

Misc

Not sure why, but proxy_protocol has two entries listed (JSON docs for listener_wrappers) with one marked as non-standard:

Seems to be better communicated here:

---

The PROXY protocol links point to docs location from HAProxy 1.8 release (that seems to have been updated since) but should probably reference a more direct source?: https://github.com/haproxy/haproxy/blob/master/doc/proxy-protocol.txt

[francislavoie]

Can you share your full config? You only showed global options, but the actual site blocks affect the result of servers. Just so we're on the same page.

It sounds to me like a logic error that you use PROXY protocol and are sending both HTTP and HTTPS to the same port. That implies you have a proxy in front of Caddy, so that proxy should be configured to never do that.

The point of http_redirect is mainly for users who want to serve TLS on non-standard ports, because then when you type that port in your browser address bar like example.com:8443 the browser will try HTTP first because you didn't specify a scheme. So the listener detects that case and serves a redirect to https://example.com:8443 to "fix" it.

Not sure why, but proxy_protocol has two entries listed (JSON docs for listener_wrappers) with one marked as non-standard:

That's because it used to only be available as a plugin before 2.7.0, but we've since bundled it with Caddy because we wanted to add PROXY protocol support to reverse_proxy's HTTP transport. We just forgot to unlist the old plugin I guess. We can do that (FYI @mholt).

The PROXY protocol links point to docs location from HAProxy 1.8 release (that seems to have been updated since) but should probably reference a more direct source?: haproxy/haproxy@master/doc/proxy-protocol.txt

Good point, updated. But anyway, the documents aren't materially different, looks like there was just a tiny fix regarding the byte format but that's not relevant to users, only relevant to library authors.

proxy_protocol could better clarify allow expectations (suggestion provided below), possibly improve implementation.

I assume you're using v2.7.6, right? We're using an older PROXY protocol implementation in that version, and we've merged a change in caddyserver/caddy#5915 to use a different library, which will release with v2.8.0

Do you mind trying a build from master to see if it works as you'd expect?

[polarathene]

Can you share your full config? You only showed global options, but the actual site blocks affect the result of servers. Just so we're on the same page.

The Caddyfile was rather simple, beyond the global config (and using local_certs):

# Technically when http:// was active it was a separate site block with different respond text to avoid any confusion
#http://example.test {
example.test {
  log
  respond "hello world!"
}

• I added the http:// as described above while troubleshooting.
• I was testing with curl only.

Curl commands:

# With Traefik
# --resolve adds entries to reroute FQDN:port through Traefik IP instead of direct connection to Caddy
# -k / --insecure because of container boundaries adding friction to the local cert trust.
# This has both ports configured to test:
# - HTTP => HTTPS working (handled by Caddy)
# - PROXY protocol (handled between Traefik => Caddy)
# Additionally without redirect, that Traefik successfully routes and connects HTTP / HTTPS to equivalent site blocks in Caddy
$ curl -vk --resolve example.test:80:172.16.42.10 --resolve example.test:443:172.16.42.10 http://example.test

# Direct to Caddy:
$ curl http://example.test
$ curl https://example.test

# Only for investigating concerns related to http_redirect feature:
# Not really relevant to Traefik
$ curl http://example.test:443
Client sent an HTTP request to an HTTPS server.

I can provide a proper compose.yaml reproduction with Traefik config if interested. It'll have to wait until tomorrow as it's late here.

---

It sounds to me like a logic error that you use PROXY protocol and are sending both HTTP and HTTPS to the same port. That implies you have a proxy in front of Caddy, so that proxy should be configured to never do that.

• There was no proxying of HTTP and HTTPS to the same port. There was a reverse proxy (Traefik) routing to Caddy though,.
• Traefik was configured to route port 80 to port 80 on Caddy (Docker image), and likewise for port 443 to port 443 (but with TLS passthrough, delegating Caddy to terminate TLS).
• Traefik was used for convenience / interoperability since I had just been using that for the TCP router PROXY protocol support for non-web traffic.

With the bug described above (no separate http:// block), for clarity:

• Traefik has two types of routers HTTP (Layer 7) and TCP (Layer 4, akin to Caddy L4 app), but PROXY protocol is only supported at Layer 4 so Traefik must use TCP routers for that.
• With my findings shared above, if the TCP router handling HTTP traffic disabled PROXY protocol, or I swapped it for an HTTP router (exact equivalent settings without PROXY protocol enabled), it'd work fine (a redirect was issued just like a direct Caddy connection).
• This is presumably because the redirect was being issued implicitly before proxy_protocol would be handled? Whereas if the PROXY header information was present Caddy shows nothing in logs and Traefik responds with 400 Bad Request.
• As noted this was resolved by opt-out of the implicit redirect by adding a separate http:// block, but to get back the redirect to HTTPS... Caddy needed listener_wrappers to have http_redirect added explicitly after proxy_protocol. If http_redirect was placed prior, it didn't do anything and the regular HTTP respond directive was returned.

---

The point of http_redirect is mainly for users who want to serve TLS on non-standard ports, because then when you type that port in your browser address bar like example.com:8443 the browser will try HTTP first because you didn't specify a scheme. So the listener detects that case and serves a redirect to https://example.com:8443 to "fix" it.

I understand, and the docs could probably convey that intent better.

Presently they imply the feature redirects in a manner that's contradicting if it's http:// to the HTTP port (as defined by Caddy with default port 80). That should technically be a no-op?

I understand that I would need to specify two separate servers config here otherwise. I'm just documenting the concern as a workaround to the actual bug since http://example.test:80 was not receiving an implicit redirect like it should have by default (with no http:// block in Caddyfile), this was only breaking when the PROXY protocol header was present.

Whereas a TLS / HTTPS connection worked fine with the PROXY protocol header. Plain HTTP works too when the implicit redirect feature isn't there, so that seems like a bug?

---

The PROXY protocol links point to docs location from HAProxy 1.8 release (that seems to have been updated since) but should probably reference a more direct source?: haproxy/haproxy@master/doc/proxy-protocol.txt

Good point, updated. But anyway, the documents aren't materially different, looks like there was just a tiny fix regarding the byte format but that's not relevant to users, only relevant to library authors.

I understand. It may change if there is an update sometime in the future and the 1.8 docs aren't refreshed with that update though 🤷‍♂️ I figure the github source is more appropriate 👍

I'm not a library author but I didn't know anything about it until a week ago, there had been various issues from users with setting up PROXY protocol correctly with Traefik and other maintainers not knowing what the support status was with software we needed to use it with (neither documented v2 support IIRC, while our own prior community contributed docs had mixed versioning). I still found the spec useful :)

---

proxy_protocol could better clarify allow expectations (suggestion provided below), possibly improve implementation.

I assume you're using v2.7.6, right? We're using an older PROXY protocol implementation in that version, and we've merged a change in caddyserver/caddy#5915 to use a different library, which will release with v2.8.0

Do you mind trying a build from master to see if it works as you'd expect?

Yes 2.7.6.

I can do a build tomorrow and get back to you 👍

[francislavoie]

I added the http:// as described above while troubleshooting.

So the thing is, the servers global option with no arguments applies to all listeners produced by site blocks in your Caddyfile. If you have just a site with no scheme, then it's a :443 server. But if you add an http:// site block, then it will always apply to the :80 server. Use caddy adapt -p to see what it produces.

Basically if you're using the tls listener wrapper, you should explicitly configure servers :443 and not just servers so it doesn't accidentally apply to your :80 server as well.

Presently they imply the feature redirects in a manner that's contradicting if it's http:// to the HTTP port (as defined by Caddy with default port 80). That should technically be a no-op?

I'm not sure I follow, but it's definitely wrong to enable the http_redirect listener wrapper on an :80 server, because then it would prevent things like the ACME HTTP challenge from working (because it would redirect before the HTTP handlers get a chance to run).

Similarly, it doesn't really make sense to enable PROXY protocol on the :80 server because there's no real value in knowing the real client IP when the result is redirecting to HTTPS and not reaching the app.

[polarathene]

Basically if you're using the tls listener wrapper, you should explicitly configure servers :443 and not just servers so it doesn't accidentally apply to your :80 server as well.

I'm not sure I follow, but it's definitely wrong to enable the http_redirect listener wrapper on an :80 server, because then it would prevent things like the ACME HTTP challenge from working (because it would redirect before the HTTP handlers get a chance to run).

I don't actually want the http://,

• I was just documenting the observations with auto_https implicit redirect behaviour breaking when proxy_protocol is in use.
• No http_redirect is needed, But docs should probably clarify that it should come after proxy_protocol when both are present, as per the summary bullet points.
• http_redirect + http:// was only to confirm that the functionality was working (HTTP => HTTPS) with proxy_protocol when configured explicitly. So the incompatibility was with implicit logic.

We'll see if the 2.8 build tomorrow is any different there.

Regarding the ACME example and the quote, http_redirect should logically not apply if the scheme is http:// and the port is 80? (or rather http_port) but it does.

---

Similarly, it doesn't really make sense to enable PROXY protocol on the :80 server because there's no real value in knowing the real client IP when the result is redirecting to HTTPS and not reaching the app.

It wasn't about if it made sense. Just identifying inconsistent in behaviour with connections when PROXY protocol is enabled or not. It should not break the default implicit redirect functionality.

example.test {
  respond "test"
}

That should redirect http to https when PROXY protocol is enabled, it doesn't.

[francislavoie]

No http_redirect is needed, But docs should probably clarify that it should come after proxy_protocol when both are present, as per the summary bullet points.

I think I'd rather just document that they shouldn't be used together. There's not really any situation where it makes sense to have both.

Regarding the ACME example and the quote, http_redirect should logically not apply if the scheme is http:// and the port is 80? (or rather http_port) but it does.

Moreso, it shouldn't be configured on :80. It's not designed to be "smart", it's designed to be a "dumb" redirect.

That should redirect http to https when PROXY protocol is enabled, it doesn't.

So you're saying that curl -v http://example.com with that config + PROXY protocol listener wrapped enabled does not show Location: https://example.com/ ? Yeah that's probably a bug if that's the case, but I have no idea why that would happen. Debug level logs (or just the curl output) might show something interesting. If proven, an issue should be open on the main Caddy repo.

[polarathene]

Built Caddy 2.8 as shown in config below.

• Observed bug still occurs. Caddy must have an http:// or similar declaration in Caddyfile:
  • Otherwise the implicit redirect on port 80 is presently incompatible with PROXY protocol.
  • Presumably because it behaves similar to http_redirect before proxy_protocol.
• The switch to go-proxyproto is an improvement (Same package as what Traefik uses). Now a port can accept traffic with/without PROXY protocol from the same host 👍

I'll trim the below example to bare minimal for demonstrating the bug in Caddy 2.8 and open an issue with that if you'd like?

Full reproduction config

# compose.yaml
# Usage:
# - Bring up services: docker compose up -d --force-recreate
# - Run curl tests: docker compose run --rm client

services:
  reverse-proxy:
    image: docker.io/traefik:3.0 #2.10
    hostname: traefik.internal.test
    networks:
      default:
        ipv4_address: 172.16.42.10
    command:
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --providers.docker.network=dms-test-network
      - --entrypoints.web.address=:80
      # Equivalent to Caddy trusted_proxies (insecure trusts everyone):
      # An entrypoint is roughly equivalent to a caddy server config.
      # NOTE: TCP routers will not discard any HTTP headers added by curl
      #       Nor will any `X-Forwarded-*` headers be added by Traefik.
      #- --entryPoints.web.forwardedHeaders.insecure=true
      # Equivalent to Caddy implicit redirects (HTTP/80 => HTTPS/443) functionality:
      #- --entrypoints.web.http.redirections.entryPoint.to=websecure
      #- --entrypoints.web.http.redirections.entryPoint.scheme=https
      - --entryPoints.websecure.address=:443
      #- --log.level=debug
    # CAUTION: Production usage should configure socket access better (see Traefik docs)
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

  web:
    # Test Caddy 2.7 via image by uncommenting, should have priority over the build field.
    # image: caddy:2.7
    hostname: caddy.internal.test
    networks:
      default:
        ipv4_address: 172.16.42.20
    # Caddy 2.8 makes PROXY protocol header optional on
    # the receiving port, just like Traefik.
    # Build master branch for unreleased Caddy 2.8:
    build:
     dockerfile_inline: |
       FROM caddy:2.7-builder AS builder
       RUN xcaddy build master

       FROM caddy:2.7
       COPY --from=builder /usr/bin/caddy /usr/bin/caddy
    labels:
      - traefik.enable=true

      # Only TCP services can enable PROXY Protocol:
      - traefik.tcp.routers.web.rule=HostSNI(`*`)
      - traefik.tcp.routers.web.entrypoints=web
      - traefik.tcp.routers.web.service=web
      - traefik.tcp.services.web.loadbalancer.server.port=80
      # Comment the below line to properly toggle off PROXY protocol for HTTP traffic:
      - traefik.tcp.services.web.loadbalancer.proxyProtocol.version=2

      # TLS passthrough delegates TLS termination to Caddy instead:
      - traefik.tcp.routers.websecure.rule=HostSNI(`*`)
      - traefik.tcp.routers.websecure.tls.passthrough=true
      - traefik.tcp.routers.websecure.entrypoints=websecure
      - traefik.tcp.routers.websecure.service=websecure
      - traefik.tcp.services.websecure.loadbalancer.server.port=443
      - traefik.tcp.services.websecure.loadbalancer.proxyProtocol.version=2

      # As TCP routers can only match host by SNI (which requires TLS), the best they can do is
      # use a wildcard catch-all via HOSTSNI(`*`).
      # - However since TCP routers rules have precedence over HTTP routers,
      #   this prevents a more specific HTTP router rule from being matched:
      #   https://doc.traefik.io/traefik/routing/routers/#general_1
      # - For HTTP routers with TLS (HTTPS) the precedence is flipped:
      #   https://github.com/traefik/traefik/issues/10186#issuecomment-1781200577
      #   https://github.com/traefik/traefik/pull/9024
      # - Thus while the service below will work for https://http-only.test,
      #   the router itself will never match when the TCP non-TLS router is present above.
      # - All non-TLS traffic incoming to Traefik is via the TCP router,
      #   only https://no-proxy-header.test tests without PROXY protocol as a result.
      # NOTE: A rule like "PathPrefix(`/`)" should match any host if needed instead.
      - traefik.http.routers.web.rule=Host(`http-only.test`) || Host(`no-proxy-header.test`)
      - traefik.http.routers.web.entrypoints=web
      - traefik.http.routers.web.service=web
      - traefik.http.services.web.loadbalancer.server.port=80
      # Uncomment this and traffic routed through this service to Caddy (eg: https://http-only.test)
      # would always result in a HTTP redirect response as the host header is stripped away:
      # https://doc.traefik.io/traefik/routing/services/#pass-host-header
      #- traefik.http.services.web.loadbalancer.passHostHeader=false

      # While this could have been a HTTP router with TLS, it instead demonstrates TCP TLS router
      # behaviour when PROXY protocol is not enabled:
      # Only matches requests to the explicit servername (SNI):
      - traefik.tcp.routers.no-proxy-header.rule=HostSNI(`no-proxy-header.test`)
      - traefik.tcp.routers.no-proxy-header.tls.passthrough=true
      - traefik.tcp.routers.no-proxy-header.entrypoints=websecure
      - traefik.tcp.routers.no-proxy-header.service=no-proxy-header
      - traefik.tcp.services.no-proxy-header.loadbalancer.server.port=443

      # Handle https://http-only.test differently than other HTTPS requests.
      # Traefik will terminate TLS instead of delegating to Caddy (no passthrough).
      # Thus the Traefik cert is presented instead, and TLS HTTP router uses the same
      # HTTP/80 service defined above (`web`) to send decrypted traffic to Caddy.
      - traefik.http.routers.http-only.rule=Host(`http-only.test`)
      - traefik.http.routers.http-only.tls=true
      - traefik.http.routers.http-only.entrypoints=websecure
      - traefik.http.routers.http-only.service=web

    configs:
      - source: caddyfile
        target: /etc/caddy/Caddyfile

  # Only intended for running via `docker compose run --rm -it client`
  client:
    hostname: client.internal.test
    # An easy to recognize IP in the logs:
    networks:
      default:
        ipv4_address: 172.16.42.42
    # Prevent starting this service by default with `docker compose up`:
    profiles:
      - testing
    build:
      dockerfile_inline: |
        FROM alpine
        RUN apk add curl
    command: ash /tmp/run.sh
    configs:
      - source: curl-cmds
        target: /tmp/run.sh

networks:
  default:
    name: dms-test-network
    ipam:
      config:
        - subnet: "172.16.42.0/24"

configs:
  curl-cmds:
    content: |
      #!/bin/sh

      # Curl options:
      # - `--location` (`-L`) to follow redirect for HTTP => HTTPS (used for direct Caddy example.test)
      # - `-w '\n'` for ensuring a final LF which is not part of the Caddy respond directive
      # - `--connect-to` routes traffic to the Traefik reverse proxy:
      #   https://curl.se/docs/manpage.html#--connect-to


      echo 'HTTP should respond without redirect (when http:// site address is enabled):'

      # This fails with `404 Bad Request` by default since the site address is disabled in Caddyfile
      # When enabled, the next two tests will no longer fail and instead respond with a redirect.
      echo -e '\n- Client => Traefik (PROXY protocol) => Caddy (http://http-only.test)'
      curl --connect-to ::traefik.internal.test http://http-only.test -w '\n'
      # HTTP without redirect
      # Client IP: 172.16.42.42, Remote Host IP: 172.16.42.42


      echo -e '\n------\nHTTP should redirect:'

      # PROXY protocol fails to get redirect response unless Caddyfile has `http:// {}` or similar:
      echo -e '\n- Client => Traefik (PROXY protocol) => Caddy (http://example.test)'
      curl --verbose --connect-to ::traefik.internal.test http://example.test 2>&1 \
        | grep -E '^< HTTP/1.1 (308 Permanent Redirect|400 Bad Request)'
      # < HTTP/1.1 400 Bad Request
      # < HTTP/1.1 308 Permanent Redirect

      # This is inaccurate, Traefik router precedence prevents HTTP non-TLS being accessible
      # when TCP non-TLS router is active.. Only the https://no-proxy-header.test is relevant.
      echo -e '\n- Client => Traefik (no PROXY protocol) => Caddy (http://no-proxy-header.test) | **Unreliable**'
      curl --verbose --connect-to ::traefik.internal.test http://no-proxy-header.test 2>&1 \
        | grep -E '^< HTTP/1.1 (308 Permanent Redirect|400 Bad Request)'
      # < HTTP/1.1 400 Bad Request
      # < HTTP/1.1 308 Permanent Redirect

      # Direct connections, always a redirect response:
      echo -e '\n- Client => Caddy (http://example.test)'
      curl --verbose --connect-to ::caddy.internal.test http://example.test 2>&1 | grep '308 Permanent Redirect'
      # < HTTP/1.1 308 Permanent Redirect

      echo -e '\n- Client => Caddy (http://no-proxy-header.test)'
      curl --verbose --connect-to ::caddy.internal.test http://no-proxy-header.test 2>&1 | grep '308 Permanent Redirect'
      # < HTTP/1.1 308 Permanent Redirect


      echo -e '\n------\nHTTPS should respond successfully:'

      echo -e '\n- Client => Traefik (PROXY protocol) => Caddy (https://example.test)'
      curl --insecure --connect-to ::traefik.internal.test https://example.test -w '\n'
      # Hello HTTP => HTTPS
      # Client IP: 172.16.42.42, Remote Host IP: 172.16.42.42

      echo -e '\n- Client => Traefik (no PROXY protocol) => Caddy 443 (https://no-proxy-header.test)'
      curl --insecure --connect-to ::traefik.internal.test https://no-proxy-header.test -w '\n'
      # Caddy 2.7 will reject connections from allowed PROXY protocol hosts when the PROXY header is missing. Caddy 2.8 is flexible!
      # Client IP: 172.16.42.10, Remote Host IP: 172.16.42.10
      #
      # Curl output with Caddy 2.7:
      # curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to no-proxy-header.test:443

      echo -e '\n- Client => Traefik (no PROXY protocol, TLS termination) => Caddy 80 (https://http-only.test)'
      curl --insecure --connect-to ::traefik.internal.test https://http-only.test -w '\n'
      # HTTP without redirect
      # Client IP: 172.16.42.42, Remote Host IP: 172.16.42.42
      # X-Forwarded-* headers are present
      #
      # Curl output with Caddy 2.7:
      # 400 Bad Request

      echo -e '\n- Client => Caddy (https://example.test)'
      curl --location --insecure --connect-to ::caddy.internal.test http://example.test -w '\n'
      # Hello HTTP => HTTPS
      # Client IP: 172.16.42.42, Remote Host IP: 172.16.42.42

      # NOTE: This check is redundant but included for consistency:
      echo -e '\n- Client => Caddy (https://no-proxy-header.test)'
      curl --insecure --connect-to ::caddy.internal.test https://no-proxy-header.test -w '\n'
      # Caddy 2.7 will reject connections from allowed PROXY protocol hosts when the PROXY header is missing. Caddy 2.8 is flexible!
      # Client IP: 172.16.42.42, Remote Host IP: 172.16.42.42

  caddyfile:
    content: |
      # Global options:
      {
        local_certs

        #log default {
        #  level DEBUG
        #}

        # Enable ProxyProtocol for incoming Traefik connections
        # https://caddyserver.com/docs/caddyfile/options#listener-wrappers
        servers {
          log_credentials
          trusted_proxies static 172.16.42.0/24

          listener_wrappers {
            # Only trust PROXY protocol connections from Traefik:
            proxy_protocol {
              allow 172.16.42.10/32
            }

            # Technically neither of the below rules should be used for
            # servers.listener_wrappers on :80, if needing PROXY protocol
            # on HTTP as well, you should have separate server configs per port.

            # Must come after proxy_protocol but before tls:
            #http_redirect
            tls
          }
        }
      }

      # https://caddyserver.com/docs/caddyfile/options#client-ip-headers
      # > Pairing with trusted_proxies, allows configuring which headers to use to determine the client's IP address.
      # > By default, only X-Forwarded-For is considered.
      # https://caddyserver.com/docs/caddyfile/matchers#client-ip
      # > Only requests from trusted proxies will have their client IP parsed at the start of the request;
      # > untrusted requests will use the remote IP address of the immediate peer.
      # https://caddyserver.com/docs/caddyfile/matchers#remote-ip
      # > the first IP in the X-Forwarded-For request header, if present, will be preferred as the reference IP,
      # > rather than the immediate peer's IP, which is the default.
      (respond_with) {
        log

        respond <<EOF
          {args[0]}
          Caddy Received:
            Client IP (ideally the real client): {client_ip}
            Remote Host IP (connected to Caddy): {remote_host} (requests.remote_ip)
            Request To (Host header): {host}
            ---
            NOTE: Traefik only provides these headers for connections using HTTP router/services not TCP:
            X-Forwarded-Server (aka Remote Host): {header.X-Forwarded-Server}
            X-Forwarded-For (aka Client IP): {header.X-Forwarded-For}
            X-Forwarded-Host (aka Host header): {header.X-Forwarded-Host}
            X-Real-Ip (aka Client IP): {header.X-Real-Ip}
        EOF
      }

      # Test default implicit redirect for HTTP/80 => HTTPS/443
      # Fails when PROXY protocol header is present!
      example.test {
        import respond_with "Hello HTTP => HTTPS"
      }

      # This works from Traefik with Caddy 2.8, but not 2.7:
      no-proxy-header.test {
        import respond_with "Caddy 2.7 will reject connections from allowed PROXY protocol hosts when the PROXY header is missing. Caddy 2.8 is flexible!"
      }

      # Confirms bug with implicit HTTP -> HTTPS redirect on above site addresses:
      # - Even with a PROXY header received, http:// connection responds without failure.
      # - An `http_redirect` added after `proxy_protocol` in `servers.listener_wrappers` will also
      #   trigger this to handle HTTP => HTTPS, which demonstrates that an explicit redirect works.
      #http://http-only.test {
      #  import respond_with "HTTP without redirect"
      #}

      # The http:// site block enables HTTP => HTTPS redirect for any inbound address on port 80 (no matching site address required)
      # `http:// {}` is sufficient to trigger that.

---

Observations from above reproduction

Caddy 2.7 only, with http:// site address block enabled (neither http:// or https:// work as Caddy only expects PROXY protocol from Traefik):

HTTPS should respond successfully:

- Client => Traefik (no PROXY protocol) => Caddy 443 (https://no-proxy-header.test)
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to no-proxy-header.test:443

- Client => Traefik (no PROXY protocol, TLS termination) => Caddy 80 (https://http-only.test)
400 Bad Request

---

Caddy 2.7 and 2.8, when an http:// site address is not present, fails to provide redirect when PROXY header is present:

HTTP should redirect:

- Client => Traefik (PROXY protocol) => Caddy (http://example.test)
< HTTP/1.1 400 Bad Request

- Client => Caddy (http://example.test)
< HTTP/1.1 308 Permanent Redirect

---

Third line is present when http:// site address is used (perhaps has some relevance to redirect behaviour being fixed for other site addresses?):

{"level":"info","logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}

[polarathene]

So you're saying that curl -v http://example.com with that config + PROXY protocol listener wrapped enabled does not show Location: https://example.com/ ?

Yes.

# Start the Traefik and Caddy containers:
$ docker compose up -d --force-recreate
# Get a shell into the client container for manual commands:
$ docker compose run --rm client ash

# Route to Caddy through Traefik TCP router with PROXY protocol:
$ curl --verbose --connect-to ::traefik.internal.test http://example.test

* Connecting to hostname: traefik.internal.test
* Host traefik.internal.test:80 was resolved.
* IPv6: (none)
* IPv4: 172.16.42.10
*   Trying 172.16.42.10:80...
* Connected to traefik.internal.test (172.16.42.10) port 80
> GET / HTTP/1.1
> Host: example.test
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Content-Type: text/plain; charset=utf-8
< Connection: close
<
* Closing connection
400 Bad Request

# Direct to Caddy:
$ curl --verbose --connect-to ::caddy.internal.test http://example.test
* Connecting to hostname: caddy.internal.test
* Host caddy.internal.test:80 was resolved.
* IPv6: (none)
* IPv4: 172.16.42.20
*   Trying 172.16.42.20:80...
* Connected to caddy.internal.test (172.16.42.20) port 80
> GET / HTTP/1.1
> Host: example.test
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://example.test/
< Server: Caddy
< Date: Tue, 13 Feb 2024 08:22:02 GMT
< Content-Length: 0
<
* Closing connection

In the previous comment, comment out this TCP non-TLS service to disable PROXY protocol usage:

- traefik.tcp.services.web.loadbalancer.proxyProtocol.version=2

Run the same curl command again:

$ curl --verbose --connect-to ::traefik.internal.test http://example.test

* Connecting to hostname: traefik.internal.test
* Host traefik.internal.test:80 was resolved.
* IPv6: (none)
* IPv4: 172.16.42.10
*   Trying 172.16.42.10:80...
* Connected to traefik.internal.test (172.16.42.10) port 80
> GET / HTTP/1.1
> Host: example.test
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://example.test/
< Server: Caddy
< Date: Tue, 13 Feb 2024 08:23:30 GMT
< Content-Length: 0
<
* Closing connection

Issue is related to PROXY protocol header.

---

Alternatively, enable PROXY protocol again for the TCP non-TLS service; Then uncomment the Caddyfile entry for http://http-only.test:

http://http-only.test {
  import respond_with "HTTP without redirect"
}

Ignore that site address, just try connect to http://example.test again:

$ curl --verbose --connect-to ::traefik.internal.test http://example.test

* Connecting to hostname: traefik.internal.test
* Host traefik.internal.test:80 was resolved.
* IPv6: (none)
* IPv4: 172.16.42.10
*   Trying 172.16.42.10:80...
* Connected to traefik.internal.test (172.16.42.10) port 80
> GET / HTTP/1.1
> Host: example.test
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://example.test/
< Server: Caddy
< Date: Tue, 13 Feb 2024 08:29:19 GMT
< Content-Length: 0
<
* Closing connection

You can see that it works too.

---

Yeah that's probably a bug if that's the case, but I have no idea why that would happen.

Something to do with that change and/or PROXY protocol header interaction with the default implicit HTTP => HTTPS redirect?

[francislavoie]

Observed bug still occurs. Caddy must have an http:// or similar declaration in Caddyfile:
Otherwise the implicit redirect on port 80 is presently incompatible with PROXY protocol.

Okay so you're saying that you're sending PROXY protocol to port 80, while having only configured an HTTPS site? Yes, it's expected that this would fail as-is.

As noted in the docs (e.g. https://caddyserver.com/docs/caddyfile/options#name), the HTTP server is only created after the fact by Automatic HTTPS. There's no way for it to guess that it should have enabled PROXY protocol. A user could just as easily only be using PROXY protocol for port 443 and not port 80, so assuming that and copying over the listener wrapper automatically or whatever, would be a flaky assumption I think.

So I think there's no bug here, seems to be working as intended.

[polarathene]

Okay so you're saying that you're sending PROXY protocol to port 80, while having only configured an HTTPS site? Yes, it's expected that this would fail as-is.

Then why doesn't it fail when I connect to port 80 without PROXY protocol? 😕

As noted in the docs (e.g. https://caddyserver.com/docs/caddyfile/options#name), the HTTP server is only created after the fact by Automatic HTTPS. There's no way for it to guess that it should have enabled PROXY protocol.

Okay, I think that docs could be explained better. I read through it several times and still wasn't quite sure how to grok the admonition as it uses name for both 443 and 80, while only stating you need to add a http:// / :80 site block too:

Thought process to making sense of it

I was under the assumption that server block I defined covered both :80 and :443, and as I mentioned without an http:// site block:

• Direct connections to Caddy at http:// redirect to HTTPS
• Connections to http:// routed through Traefik fail with 400 Bad Request, but only when PROXY protocol is enabled.

When I add an http:// block myself (regardless of direct to Caddy or via Traefik with PROXY protocol):

• I can access that address directly and get the HTTP respond as response, no redirect.
• I can access another site block like http://example.test and get a redirect to https://example.test

Note that:

• I did not add a name http to the servers config as per those docs.
• Automatic redirects were clearly working by default without an http:// block, but only when no PROXY header is involved.
• Just adding the http:// block fixes the incompatibility with PROXY header, which is unclear.

The above image from the servers.name docs doesn't really explain this.

• Ok Automatic HTTPS:
  • Creates a servers :80 implicitly to handle the redirects
  • It apparently cannot know to enable proxy_protocol (despite the modified servers config that applies to :80?)
  • Adding http:// / :80 fixes it, presumably because servers ports are only adapted by the site block ports which previously were only implicitly https:// / :443 so that's the only servers config that was adapted to JSON?

So rather the docs linked are trying to communicate that the feature is unaware of the Caddyfile with generic servers as a default, since it only has JSON adapted servers for explicit ports? This seems to not exactly be relevant to name (which yes it clarifies that information is lost), it also has explicit servers :80 but states that http:// is required, implying that the site block is the only way to persist a configured server through to adapted JSON, discarding any configured servers (implicit or explicit ports) when no site block to use it exists.

Those docs could better explain:

• That the issue is related to servers config in Caddyfile only adapts to JSON by port (implicit or explicit in servers) when a port is explicitly used as a site address block (eg: http:// / :80).
• Automatic HTTPS will generate implicitly an :80 server if one was not adapted to JSON from an explicit http:// / :80 site block. Despite the docs focus on losing the name attribute, it's rather than entire servers block is ignored without that explicit site block to adapt it to JSON (servers doesn't need to define an explicit :80, that was only to demonstrate name working).

I didn't care about the server name, nor was that immediately apparent in logic for me to grok that I'd probably have missed what it was explaining. That information should probably be pulled out to the main servers description, or at the very least for proxy_protocol section to raise awareness on where it's going to have the most likely amount of friction/confusion similar to name probably did.

---

A user could just as easily only be using PROXY protocol for port 443 and not port 80, so assuming that and copying over the listener wrapper automatically or whatever, would be a flaky assumption I think.

No, my understanding was that if I define servers without an explicit port, it configures proxy_protocol for all ports/servers.

If I only wanted it on one port, I would be explicit with servers (which as discovered, doesn't matter unless an explicit site block uses that port).

At least now I know that the implicit :80 listener wasn't getting proxy_protocol from servers, and that I had to add an http:// {} hint to provide an explicit :80 listener. That's where the UX issue was.

---

So I think there's no bug here, seems to be working as intended.

Agreed not a bug, but still a docs issue.

Not exactly specific to proxy_protocol, and I don't know how often someone would encounter that but not a fun one to troubleshoot 😆

I had figured it was related to the servers config and Automatic HTTPS, hence the exploration with http_redirect. Maybe the issue is relevant elsewhere, or might be in future settings/features...

That needs a disclaimer not hidden under name regarding the expectation of a site block using servers config, an explicit port in servers alone is not sufficient. Excluding a port to rely on implicit defaults is probably only relevant to :80 with Automatic HTTPS, but should probably draw attention to that too.

name and proxy_protocol sections could then link a reference to that for added awareness/troubleshooting.

[francislavoie]

Then why doesn't it fail when I connect to port 80 without PROXY protocol? 😕

Why would it? You're just making a simple HTTP request to a simple HTTP server.

If you send PROXY protocol to a simple HTTP server, then it won't be happy because it didn't see HTTP, it saw the PROXY preamble.

Okay, I think that docs could be explained better.

It's a really difficult concept to explain. It requires a grasp of all these things:

• that the Caddyfile produces JSON config, and that's what actually runs
• that sites in the Caddyfile don't map 1:1 to JSON servers (multiple sites may be one server, or one site may be multiple servers) depending on the site addresses
• that listener addresses are the key for servers, and since this is "hidden" by site addresses, it's not obvious
• how Automatic HTTPS works, and that it augments the JSON config at runtime, and can produce an :80 server that didn't exist at Caddyfile adapt time
• that servers are applied at Caddyfile adapt time, after parsing sites, but before adapting to JSON
• and finally, that because of all of the above, servers doesn't apply to :80 unless http:// { } is added.

You might say "but if servers is used and auto-https is configured, just always produce a dummy :80 in the Caddyfile adapter" but that's not a valid assumption because listener wrappers for :443 cannot work for :80 (the tls listener wrapper should not be used on :80 for example). It really has to be explicitly configured by the user, more implicit behaviour won't be better.

But anyway, yes this note shouldn't be only in the name part. It was just that the first person who ran into this confusing quirk had it happen when they were trying to use name, but it applies in general to the whole concept of servers due to auto-https etc. So yes that explanation should be hoisted up.

But still... it's so much to explain, and we don't want the docs to exhaust readers, so it needs to be presented in a sensible way, and I'm having trouble resolving those competing requirements.

To be clear - the docs will change, I'm working on it. But it'll take time to find the right approach.

[polarathene]

Why would it? You're just making a simple HTTP request to a simple HTTP server.

Sorry, I forgot to go backwards and edit that away.

I was asking that on the assumption that I had :80 implicitly configured for proxy_protocol, and that adding http:// site block resolved the incompatibility in some way that wasn't apparent to me beyond how Automatic HTTPS was working behind the scenes.

That's now understood that my servers config was ignored and only applying to https:// / :443 since that's all I had implicitly defined in site blocks until I added an explicit http:// 👍

---

It's a really difficult concept to explain.

Warning

• servers configuration only applies to site addresses explicitly defined
• Automatic HTTPS provides an implicit servers :80 configuration unless your Caddyfile explicitly has a http:// or :80 site address defined with a matching servers configuration (:80 may be explicit or implicit`)

Could do with some revision but that already conveys the issue more clearly to the reader, without being hidden under servers.name?

Doesn't need to explain the technical details regarding adapting the Caddyfile to JSON, just that expectation/requirement of servers only being valid/retained when an explicit site address would use it.

---

You might say "but if servers is used and auto-https is configured, just always produce a dummy :80 in the Caddyfile adapter" but that's not a valid assumption because listener wrappers for :443 cannot work for :80 (the tls listener wrapper should not be used on :80 for example). It really has to be explicitly configured by the user, more implicit behaviour won't be better.

While I'm sure you're correct about that, it does work regardless with tls listener wrapper on :80. I guess the scheme detection is the only differentiator for why HTTP vs HTTPS still works with curl.

Someone interested in proxy_protocol may very well from the current docs, just setup servers like I did and either already have an http:// / :80 site block, or adds one to resolve the observed "bug". As you've mentioned, there's quite a lot going on and that confusion is only going to impact a user more who lacks the working knowledge you have.

---

But still... it's so much to explain, and we don't want the docs to exhaust readers, so it needs to be presented in a sensible way, and I'm having trouble resolving those competing requirements.

While it's not ideal, when I have this issue for the docs I maintain, niche concerns with a lot of technical details usually get placed in collapsed admonitions, or I link to a Github issue comment/thread for more info/insights.

Here is an example for IPv6 docker config:

That's already quite a bit on it's own, but neatly organizes alternative config via tabs and collapses more niche info.

Above that part is an example of where I link to a Github comment for more info:

For reference, recently revised PROXY protocol docs page (unreleased, so this link may eventually become broken for future readers). Quite a lot there is hidden away with the primary focus on the Traefik and related config to use PROXY protocol to preserve the client IP where needed.

---

To be clear - the docs will change, I'm working on it. But it'll take time to find the right approach.

No worries, I understand 👍 Writing/improving docs is a time consuming effort for me too 😓

I just wanted to raise awareness and clear up some confusion, which you've been extremely helpful with! ❤️

[francislavoie]

Could do with some revision but that already conveys the issue more clearly to the reader, without being hidden under servers.name?

Doesn't need to explain the technical details regarding adapting the Caddyfile to JSON, just that expectation/requirement of servers only being valid/retained when an explicit site address would use it.

Hmm yeah, I guess so. I'll play with that, thanks.

While I'm sure you're correct about that, it does work regardless with tls listener wrapper on :80. I guess the scheme detection is the only differentiator for why HTTP vs HTTPS still works with curl.

Oh right 🤦‍♂️ I keep forgetting that the tls LW is only a placeholder (aka ordering/positioning marker) and it doesn't actually do anything on its own. It only causes TLS to happen on a TLS-enabled server, and on a non-TLS server, it's a no-op. So yeah I guess it's fine if it's set on an HTTP server. I'll add a note of that in the docs.

While it's not ideal, when I have this issue for the docs I maintain, niche concerns with a lot of technical details usually get placed in collapsed admonitions

The issue is our docs right now are pure markdown (with some sprinkled in jquery for targeted augmentations), so we don't have the infra for collapsibles right now. I did set up a tab box with AlpineJS for this one section https://caddyserver.com/docs/running#local-https-with-docker but I haven't gotten around to using this in more places yet (we will though). I'll need to set up the stuff for collapsibles with AlpineJS too eventually.

No worries, I understand 👍 Writing/improving docs is a time consuming effort for me too 😓

I just wanted to raise awareness and clear up some confusion, which you've been extremely helpful with! ❤️

🫶

[polarathene]

The issue is our docs right now are pure markdown (with some sprinkled in jquery for targeted augmentations)

The docs I reference are generated from markdown as well (mkdocs-material). There is just some additional syntax their parser understands for features like admonitions and tabs. It's common to see in quite a few docs platforms with markdown as the source format.

You should be able to use HTML usually, and can add <details> + <summary> like I have done in earlier comments above for collapsed sections. Just for actual docs you may wants some styling added via CSS :)

I really love the feature your docs have though with config snippets that are interactable (clicking a setting name goes to that part of the docs, or hover reveals some extra information).

[francislavoie]

FYI #374, I'm working on a big docs update, this feedback is incorporated.