Broken when Reverse Proxy - CSRF_TRUSTED_ORIGINS Missing

[anymuster2]

Seems like Django 4.x added mandatory origin verification - https://docs.djangoproject.com/en/4.2/ref/settings/#csrf-trusted-origins

Healthchecks broke for me with a reverse proxy with CSRF validation errors - I was unabble to register or login as a result. Adding and additional env option and adding the line CSRF_TRUSTED_ORIGINS = os.getenv("CSRF_TRUSTED_ORIGINS").split(",") to settings.py fixed the issue locally.

[cuu508]

Thanks for the report. A few questions to help me understand what is going wrong:

• What reverse proxy are you using?
• What request headers is the reverse proxy passing to the app?
• What request headers is the reverse proxy adding?
• What WSGI server are you using?
• What do you have in the CSRF_TRUSTED_ORIGINS environment variable?
• Are you using a Docker image? If yes, which image?

[JKJameson]

Nice work anymuster2, I can confirm this fixed the problem for me with Nginx.

[adelton]

For the record, adding

proxy_set_header Host $http_host;

to my Nginx TLS proxy configured with

proxy_pass http://localhost:8000/;

resolved the issue even without setting CSRF_TRUSTED_ORIGINS -- the https://docs.djangoproject.com/en/4.2/ref/settings/#csrf-trusted-origins says

For requests that include the Origin header, Django’s CSRF protection requires that header match the origin present in the Host header.

so I just made the proxy play by those rules.

[justinmerrell]

For whatever reason, this was still giving me trouble, ended up adding these:

location / {
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Forwarded-Port $server_port;
                proxy_redirect off;
                proxy_pass http://localhost:8000;
        }

Found here: https://forum.djangoproject.com/t/how-to-silence-invalid-http-host-header-error/9826/2

I am unsure if all are required or the actual issue, but I got it working.

[JKJameson]

My NGINX block already had proxy_set_header, but it was set to $host instead of $http_host. Changing it to $http_host and removing the code changes also works.

[theko2fi]

Can confirm this issue. The error message below led me to here.

I'm using HAProxy as reverse proxy. It took me hours to get it working. So I think it could be nice to share my solution with everybody.

There's my HAProxy config file haproxy.cfg. The most important parts to focus on, are the frontend and backend sections.

global
  stats socket /var/run/api.sock user haproxy group haproxy mode 660 level admin expose-fd listeners
  log stdout format raw local0 info

defaults
  mode http
  timeout client 10s
  timeout connect 5s
  timeout server 10s
  timeout http-request 10s
  default-server init-addr none
  log global

resolvers docker_resolver
  nameserver dns 127.0.0.11:53

frontend stats
  bind *:8404
  stats enable
  stats uri /
  stats refresh 10s

frontend myfrontend
  bind :80
  bind :443 ssl crt /usr/local/etc/haproxy/certs/
  http-request redirect scheme https code 301 unless { ssl_fc }
  use_backend %[req.hdr(Host),lower]

backend healthchecks.traefik.me
  http-request del-header Origin
  server healthchecks healthchecks:8000 check inter 10s resolvers docker_resolver

This is my healthchecks docker-compose.yml file:

version: '3.9'

networks:
  healthchecks_net:
    name: healthchecks_net
    driver: bridge
  haproxy_net:
    external: true

# services
services:
  healthchecks:
    container_name: healthchecks
    image: healthchecks/healthchecks:v2.8.1
    restart: unless-stopped
    environment:
      - DEBUG=True
      - DB=postgres
      - DB_HOST=${DB_HOST}
      - DB_NAME=${DB_NAME}
      - DB_USER=${DB_USER}
      - DB_PASSWORD=${DB_PASSWORD}
      - REGISTRATION_OPEN=False
      - ALLOWED_HOSTS=healthchecks.traefik.me
      - SITE_ROOT=https://healthchecks.traefik.me
    depends_on:
      - healthchecks_database
    networks:
      - haproxy_net
      - healthchecks_net
  healthchecks_database:
    container_name: healthchecks_database
    image: postgres:15.0
    restart: always
    environment:
      - PGDATA=/var/lib/postgresql/data
      - POSTGRES_DB=${DB_NAME}
      - POSTGRES_PASSWORD=${DB_PASSWORD}
      - POSTGRES_USER=${DB_USER}
    networks:
      - healthchecks_net
    volumes:
    - ./data:/var/lib/postgresql/data

In the example above, my healthchecks instance is reachable at http(s)://healthchecks.traefik.me. Initially, I didn't run into this issue when using HTTP only. The problem appeared after enforcing TLS and redirecting all HTTP requests to HTTPS with the line http-request redirect scheme https code 301 unless { ssl_fc } in the frontend section haproxy.cfg file.

What exactly fixed the problem?

It's the line http-request del-header Origin in the backend section of HAProxy config.

What it actually does?

It deletes the Origin header before sending the request to the healthchecks container.

Why?

As shown in the screenshot, the reason given for the failure is my domain name not being present in the list of trusted domains. Yes, the Origin header value, is the domain name requested by users. So I needed to add my domain name to that list. Unfortunately, healthchecks docker container doesn't support yet setting the CSRF_TRUSTED_ORIGINS environment variable.

My workaround is to delete the origin header before forwarding the requests to the backend.

Improvement

It could be nice in next releases if @cuu508 could add support for CSRF_TRUSTED_ORIGINS environment variable. It could be even better if the default value matches the SITE_ROOT variable value.

In the end, thanks for this amazing project @cuu508. I just heard about this a couple of days ago and already like it. I also read one of your blog posts about your motivation and your journey running healthchecks.io. It's inspiring, keep it up !

[cuu508]

As shown in the screenshot, the reason given for the failure is my domain name not being present in the list of trusted domains.

The Django docs say:

For requests that include the Origin header, Django’s CSRF protection requires that header match the origin present in the Host header.

@theko2fi would you be able to go down the rabbit hole and figure out why the Host and Origin headers didn't match in your case?

Here's my hunch.

From Carlton Gibson's blog:

The CSRF check compares the Host header, which looks like 'your-site.com' with the Origin, which looks like 'https://your-site.com', by using the value of request.is_secure() to determine the scheme.

If you’re doing SSL termination at your web server, maybe your problem is that request.is_secure() is returning False, even though you’re serving with HTTPS, and so your Origin header is https://... whilst the CSRF middleware check is looking for http://....

I think what might be happening, is:

• Your HAProxy instance is not passing on information whether the request was using HTTPS or not to the Docker container
• So Django assumes http://, and compares http://healthchecks.traefik.me (http:// + Host header) with https://healthchecks.traefik.me (the Origin header)
• The two values don't match and you get the CSRF error

If that's the case, the fix is to tell Haproxy to add a X-Forwarded-Proto header like so:

    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-request set-header X-Forwarded-Proto http unless { ssl_fc }

The container image uses uwsgi, which by default checks the X-Forwarded-Proto header, and passes the value on to Django. Then Django knows if the request was secure or not, and the values of scheme+Host and Origin headers match.

My current understanding is CSRF_TRUSTED_ORIGINS should only be needed when doing cross-domain requests, which Healthchecks does not do.

[theko2fi]

@theko2fi would you be able to go down the rabbit hole and figure out why the Host and Origin headers didn't match in your case?

Sure, I reversed the thing:

• I stopped the HTTP to HTTPS redirection
• I forced HAProxy to always send https as scheme to the healthchecks container

There's the configuration I used for that:

frontend myfrontend
  bind :80
  bind :443 ssl crt /usr/local/etc/haproxy/certs/
  #http-request redirect scheme https code 301 unless { ssl_fc }
  use_backend %[req.hdr(Host),lower]

backend healthchecks.traefik.me
  http-request set-header X-Forwarded-Proto https
  server healthchecks healthchecks:8000 check inter 10s resolvers docker_resolver

Then with that configuration, requests to https://healthchecks.traefik.me succeed, while requests to http://healthchecks.traefik.me give the error below. (compare with my first screenshot)

So I can confirm, @cuu508 you're spot on. The problem is a mismatch between the scheme+Host and Origin headers. The solution is to tell HAProxy to pass the correct scheme to the healthchecks container.

For people who come in future looking for a solution, this is the way to go:

backend healthchecks.traefik.me
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http unless { ssl_fc }
  server healthchecks healthchecks:8000 check inter 10s resolvers docker_resolver

Thanks !

[praveenbm5]

For anyone running healthchecks behnd Apache, adding "RequestHeader unset Origin" to the vhost solves the problem.

[cuu508]

A better fix would be to make sure Apache passes X-Forwarded-Proto header to the app.