Esi cache broken ? (started session / private response) #26769

EmmanuelVella · 2018-04-03T12:52:28Z

Q	A
Bug report?	yes
Symfony version	3.4.6

Hello,

I just finished upgrading from Symfony 2.8 to 3.4.6, and my http cache is broken when using AppCache (without debug).

It seems that since this PR, when the session is started, the response cache is set to private, which is fine.

But now when I go to a page where session is started and where I do render_esi(controller('MyBundle:Controller:action')), http cache is not working as before, because subrequests responses are made private even if I don't start / use the session.

The text was updated successfully, but these errors were encountered:

nicolas-grekas · 2018-04-04T12:22:27Z

can you provide a reproducer please?

EmmanuelVella · 2018-04-04T15:14:20Z

@nicolas-grekas Sure, I created a symfony 3.4.6 project here : https://github.com/EmmanuelVella/symfony

You can use docker to bootstrap it :
docker build . -t symfony
docker run --rm -v "$PWD":/var/www/html -p 8000:8000 -d --name symfony symfony
docker exec symfony composer install

Then go to http://localhost:8000/app.php

Interesting parts are the controller and the template.

It seems that the firewall causes the session to start, and the response is then marked as private. Does it means I need to exclude all my esi actions from the firewall ?

With firewall : Cache-Control: max-age=0, must-revalidate, private, s-maxage=60
Without firewall : Cache-Control: public, s-maxage=60

stixx · 2018-05-03T22:12:08Z

Correct me if i'm wrong but it seems the problem relies in the AbstractSessionListener:: onKernelResponse. My first thought was why are we setting default cache-control headers and overwriting ones set on every master request? The thing that confuses me the most is that it seems obvious to me that every master request is already started here and available.

EmmanuelVella · 2018-05-04T08:08:19Z

@stokjes Yes, before symfony 4 this code was in SaveSessionListener class, then it was moved to AbstractSessionListener (see symfony/http-kernel@6622fe2#diff-db3c5d9b0dc8cbb67671806266bef598)

I understand why the response is not cached if the session is started (which is the case if you use the firewall) ; however I don't know how to disable the firewall on the esi actions, as they have not route (they are called like this : render_esi(controller('MyBundle:Controller:action')))

stixx · 2018-05-04T20:09:36Z

What if you use the firewall on the whole domain like: ^/, but not every response is user specific. Our varnish caching stopped working after the upgrade to Symfony 3.4. Even when explicitly
setting caching headers on the response, those are overwritten by the listener.

For you shouldn't the controller() specific call in the Twig template initiate a sub-request which the listener should ignore because it isn't a master request?

EmmanuelVella · 2018-05-17T14:48:25Z

For you shouldn't the controller() specific call in the Twig template initiate a sub-request which the listener should ignore because it isn't a master request?

@stokjes I don't know if this behavior is intended or not :/

nicolas-grekas · 2018-06-19T14:51:03Z

Fixed by #27467 (please report back if not.)

EmmanuelVella · 2018-06-25T09:25:19Z

@nicolas-grekas your PR certainly fixed a part of the problem, but I still have an issue.

I have updated my repository to symfony 3.4 branch (see #26769 (comment)). Please let me give you all the steps to reproduce :

Please note that the app is behind the default firewall.

When going to http://localhost:8000/app.php for the first time, the page esi fragment is correctly cached for 30s. As the session is used in the main request, a session cookie is created.

Once the fragment cache is stale (30s+), when refreshing the page the fragment won't be cached anymore (because of the session cookie ?).

What I understand here is that the firewall uses the session, so the fragment can't be cached (why is it working the first time, then ?). If that's the intented behavior, how can I exclude my esi controller from the firewall, if it has not route ?

Thank you so much for your patience / help.

nicolas-grekas · 2018-06-25T09:31:34Z

Can you provide a simple reproducer, ie a repo we could clone to experience the issue?

EmmanuelVella · 2018-06-25T09:33:54Z

I did it in a previous comment ;) Let me repeat it here :

https://github.com/EmmanuelVella/symfony

You can use docker to bootstrap it :
docker build . -t symfony
docker run --rm -v "$PWD":/var/www/html -p 8000:8000 -d --name symfony symfony
docker exec symfony composer install

Then go to http://localhost:8000/app.php

EmmanuelVella · 2018-06-29T13:29:02Z

@nicolas-grekas FYI, still not working with latest sf 3.4 update !

nicolas-grekas · 2018-06-29T16:18:25Z

It seems that the firewall causes the session to start, and the response is then marked as private. Does it means I need to exclude all my esi actions from the firewall ?

I checked out your app to reproduce and indeed it cannot cache, because as you spotted, the firewall is used on fragments, which uses the session, which makes the response uncacheable.

Would there be any downside to disabling the security on /_fragment URLs by default in https://github.com/symfony/recipes/blob/master/symfony/security-bundle/3.3/config/packages/security.yaml?

EmmanuelVella · 2018-06-29T16:53:37Z

Thank you so much for taking time to test it. Yeah I already tried this solution, and it works, but it felt a bit hacky.. so I wanted to be sure that this behavior was normal.

So currently there is no other way to disable security on a controller (with no route) called like this render_esi(controller('MyBundle:Controller:action'))) ?

nicolas-grekas · 2018-06-29T16:55:19Z

I don't know security well myself, I'd need someone else to answer this...

EmmanuelVella · 2018-06-29T16:56:33Z

Ok ;) Do you know someone who could help ?

nicolas-grekas · 2018-06-29T17:07:04Z

Let's ask on symfony/recipes#424 :)

EmmanuelVella · 2018-06-29T17:28:33Z

👍

EmmanuelVella · 2018-07-03T13:12:17Z

@nicolas-grekas I continue this discussion here to not pollute your PR.

I'm sorry but I still don't get how to disable the firewall on controller called like this (with no route) : render_esi(controller('MyBundle:Controller:action'))), without the new unauthenticated mode.

Could you please be more explicit ?

nicolas-grekas · 2018-07-03T13:18:20Z

Exactly like done here: https://github.com/symfony/recipes/pull/424/files
This disabled security on ESI fragments, meaning non anonymous token will be created for you.

EmmanuelVella · 2018-07-03T13:21:54Z

Ok, so I will do this. Thank you very much for your help ! 👍

…ate only when needed (nicolas-grekas) This PR was merged into the 4.4 branch. Discussion ---------- [Security] Make stateful firewalls turn responses private only when needed | Q | A | ------------- | --- | Branch? | 4.4 | Bug fix? | no | New feature? | yes | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | #26769 *et al.* | License | MIT | Doc PR | - Replaces #28089 By taking over session usage tracking and replacing it with token usage tracking, we can prevent responses that don't actually use the token from turning responses private without changing anything to the lifecycle of security listeners. This makes the behavior much more seamless, allowing to still log the user with the monolog processor, and display it in the profiler toolbar. This works by using two separate token storage services: - `security.token_storage` now tracks access to the token and increments the session usage tracker when needed. This is the service that is injected in userland. - `security.untracked_token_storage` is a raw token storage that just stores the token and is disconnected from the session. This service is injected in places where reading the session doesn't impact the generated output in any way (as e.g. in Monolog processors, etc.) Commits ------- 20df3a1 [Security] Make stateful firewalls turn responses private only when needed

…colas-grekas) This PR was merged into the 4.4 branch. Discussion ---------- [Security] add "anonymous: lazy" mode to firewalls | Q | A | ------------- | --- | Branch? | 4.4 | Bug fix? | no | New feature? | yes | Deprecations? | no | Tickets | Fixes #26769 et al. | License | MIT | Doc PR | - Contains #33663 until it is merged. This PR allows defining a firewall as such: ```yaml security: firewalls: main: anonymous: lazy ``` This means that the corresponding area should not start the session / load the user unless the application actively gets access to it. On pages that don't fetch the user at all, this means the session is not started, which means the corresponding token neither is. Lazily, when the user is accessed, e.g. via a call to `is_granted()`, the user is loaded, starting the session if needed. See #27817 for previous explanations on the topic also. Note that thanks to the logic in #33633, this PR doesn't have the drawback spotted in #27817: here, the profiler works as expected. Recipe update pending at symfony/recipes#649 Commits ------- 5cd1d7b [Security] add "anonymous: lazy" mode to firewalls

xabbuh added Bug HttpKernel Status: Waiting feedback labels Apr 4, 2018

carsonbot added the Status: Needs Review label Apr 4, 2018

xabbuh removed the Status: Waiting feedback label Apr 26, 2018

stixx mentioned this issue May 16, 2018

[HttpKernel] Fix response always private and overwritten if session is started #27282

Closed

EmmanuelVella changed the title ~~AppCache broken ? (started session / private response)~~ Esi cache broken ? (started session / private response) May 17, 2018

nicolas-grekas closed this as completed Jun 19, 2018

nicolas-grekas mentioned this issue Jul 3, 2018

[Security] add "lazy_authentication" mode to firewalls #27817

Closed

2 tasks

nicolas-grekas mentioned this issue Jul 30, 2018

[Security] Make statefull pages NOT turn responses private when the token is not used #28089

Closed

This was referenced Sep 22, 2019

[Security] Make stateful firewalls turn responses private only when needed #33663

Merged

[Security] add "anonymous: lazy" mode to firewalls #33676

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Esi cache broken ? (started session / private response) #26769

Esi cache broken ? (started session / private response) #26769

EmmanuelVella commented Apr 3, 2018 •

edited

nicolas-grekas commented Apr 4, 2018

EmmanuelVella commented Apr 4, 2018 •

edited

stixx commented May 3, 2018 •

edited

EmmanuelVella commented May 4, 2018

stixx commented May 4, 2018

EmmanuelVella commented May 17, 2018 •

edited

nicolas-grekas commented Jun 19, 2018

EmmanuelVella commented Jun 25, 2018 •

edited

nicolas-grekas commented Jun 25, 2018

EmmanuelVella commented Jun 25, 2018

EmmanuelVella commented Jun 29, 2018

nicolas-grekas commented Jun 29, 2018

EmmanuelVella commented Jun 29, 2018

nicolas-grekas commented Jun 29, 2018

EmmanuelVella commented Jun 29, 2018

nicolas-grekas commented Jun 29, 2018

EmmanuelVella commented Jun 29, 2018

EmmanuelVella commented Jul 3, 2018

nicolas-grekas commented Jul 3, 2018

EmmanuelVella commented Jul 3, 2018

Esi cache broken ? (started session / private response) #26769

Esi cache broken ? (started session / private response) #26769

Comments

EmmanuelVella commented Apr 3, 2018 • edited

nicolas-grekas commented Apr 4, 2018

EmmanuelVella commented Apr 4, 2018 • edited

stixx commented May 3, 2018 • edited

EmmanuelVella commented May 4, 2018

stixx commented May 4, 2018

EmmanuelVella commented May 17, 2018 • edited

nicolas-grekas commented Jun 19, 2018

EmmanuelVella commented Jun 25, 2018 • edited

nicolas-grekas commented Jun 25, 2018

EmmanuelVella commented Jun 25, 2018

EmmanuelVella commented Jun 29, 2018

nicolas-grekas commented Jun 29, 2018

EmmanuelVella commented Jun 29, 2018

nicolas-grekas commented Jun 29, 2018

EmmanuelVella commented Jun 29, 2018

nicolas-grekas commented Jun 29, 2018

EmmanuelVella commented Jun 29, 2018

EmmanuelVella commented Jul 3, 2018

nicolas-grekas commented Jul 3, 2018

EmmanuelVella commented Jul 3, 2018

EmmanuelVella commented Apr 3, 2018 •

edited

EmmanuelVella commented Apr 4, 2018 •

edited

stixx commented May 3, 2018 •

edited

EmmanuelVella commented May 17, 2018 •

edited

EmmanuelVella commented Jun 25, 2018 •

edited