Multiple Set-Cookie headers only set 1 cookie

[flotwig]

Originally posted by @pete-om in #5432 (comment)

#5432 is not fixed for me in 3.6.1.

Without a workaround, my authentication script still fails in 3.6.1. Cypress still appears to only process the first cookie in the set-cookie array. Everything works as expected in 3.4.1

The auth flow goes like this:

• cy.request (get) to /account/login
  • this sets token1 in a cookie
  • the body contains token2 as a hidden parameter in a form
• cy.request (post) to /account/login with form:true, and a body of token2, username, password
  • this results in a 302 redirect to /, with two cookies in set-cookie:
    • the first expires an sso cookie, which usually doesn't exist (but doesn't make a difference to this bug if it does exist)
    • the second sets the session auth token -- this part is failing in 3.5.0+. It exists in set-cookie, otherwise my workaround wouldn't work (manually doing a cy.setcookie on everything in the set-cookie bit), it's just not being set during the cy.request

So what we should be left with after all the above is two cookies: token1 and auth token. What I get as of 3.5.0+ is just token1 (from the original get request) and no auth token.

And we're not going outside our subdomain at any point so the cookies all have the same domain.

Note: if I cy.visit('/account/login'), type the username and password in and submit the form, the redirect sets all the cookies correctly and lets cypress log in, this method has never stopped working. It's only doing it via cy.request that's failing.

Hope this helps, let me know if there's anything else I can do :)

[codybarr]

Commenting here as it appears #5432 has moved to this issue now...

[flotwig]

Commenting here as it appears #5432 has moved to this issue now...

You can also click "Subscribe" on the right-hand sidebar to get notifications, without needing to comment. :)

[codybarr]

Commenting here as it appears #5432 has moved to this issue now...

You can also click "Subscribe" on the right-hand sidebar to get notifications, without needing to comment. :)

Yep, I'm aware. I provided logs in the last issue, so wanted you to know I'm still following all of this. Let me know if you need anything else.

[emijmker]

@flotwig Since the thread continued here, here's the addition to my post in 5432 as you requested #5432 (comment). Please note that in our case we don't miss cookies, but we seem to be getting extra cookies that are messing things up.

The way I test our flow in Cypress:

1. Send a POST with cy.request to https://login.dev.domainAAA.dev.com/login-service

2. From the response parse csrf cookie(part of set-cookie array)

3. Send a POST with cy.request to https://hostname.dev.domainAAA.com/sessionRequest. In the request header add the parsed csrf cookie

   _Tried it in Electron but there this POST won't even get exectuted:

   We attempted to make an http request to this URL but the request failed without a 		response. We received this error at the network level:  > Error: Parse Error_
   

4. In that response a redirect url is provided > https://example.dev.domainBBB.com/eyJhbetcetcSomeLongJwtToken

   this is where the FE comes in aka visit the given url either with cypress or manual in browser:

5. cy.visit('https://example.dev.domainBBB.com/eyJhbetcetcSomeLongJwtToken');

6. This url will redirect to https://example.dev.domainBBB.com/actualPageToTest
   This url gets a 504 since it's not a valid url.

Please see the screenshot I added with the difference we see with the set-cookie[Array], that doesn't happen in 3.4.1. It feels like these extra added cookies are messing up the original token, creating an invalid url.

Also I don't use a cy.setCookies command, everything is handled through that redirect url.

Via manual steps in browser

To reproduce your requested example, not via BE calls:

1. login at domainAAA.com
   token for domainAAA.com is set
2. create a session on domainBBB.com
   token for domainBBB is set
   So on both domains 'token' is set, they have different values. It looks like either one overwrites the other or that the token doesn't get properly build.

I'm using

Cypress 3.6.1
Chrome 78.0.3904.87
MacOs Mojave 10.14.6

[flotwig]

I'm working on a PR to reproduce & fix this issue here: #5702

[emijmker]

@flotwig one addition i forgot to mention, maybe it can help, maybe it won't. When putting the jwt url in jwt.io in the right situation (3.4.1) it will give the left result, in the wrong situation (3.6.1) it will give the right result. So typ:"JWT" is missing:

That's why we had the idea that somehow it gets build incorrectly.

[flotwig]

@pete-om @codybarr @emijmker Question, are you attempting to send multiple cookies in ONE Set-Cookie line?

Like this:

Set-Cookie: foo=bar, baz=quux

Or this:

Set-Cookie: foo=bar; baz=quux

As opposed to this:

Set-Cookie: foo=bar
Set-Cookie: baz=quux

[flotwig]

@pete-om I attempted to reproduce your specific setup by doing the following:

I set up an Express server with these handlers:

app.get("/", function(req, res) {
  res.type('html').end(req.cookies);
}

app.get("/account/login", function(req, res) {
  res.cookie('token1', 'foo').type('html').end('hi');
});

app.post("/account/login", function(req, res) {
  res.clearCookie('shouldclear')
  .cookie('token2', 'bar')
  .redirect(302, '/');
});

Then, I wrote this test:

it("sends cookies in redirects", function() {
  cy.setCookie('shouldclear', 'something');

  cy.request('/account/login');

  cy.request({
    method: 'POST',
    url: '/account/login'
  }).its('body').should('deep.include', {
    token1: 'foo',
    token2: 'bar'
  });
});

...and it passes in 3.6.1. Can you help me spot the difference between this and your setup so I can reproduce the problem?

[flotwig]

@emijmker You might be experiencing an issue with URL encoding, is your URL really all alphanumeric like this?

https://example.dev.domainBBB.com/eyJhbetcetcSomeLongJwtToken

Or are there special characters in there? If so, can you share a URL that contains the special characters?

[pete-om]

@flotwig It looks kinda right

Possible things to change:

• The setCookie line isn't needed
• The POST request has form:true, with account details in the request body
• Also possibly of note, the first cookie is cleared by attempting to set a cookie with no value 1 minute into the past. eg: if the time is 02:08:36 GMT, set-cookie will show:
  "ssocompany=; expires=Tue, 19-Nov-2019 02:07:36 GMT; path=/; secure; HttpOnly"

I dumped the POST cy.request xhr to console via cy.request(stuff).then(xhr => console.log(xhr)), here's a side-by-side, which might help?:

The response header details from the initial request, which shows them to be basically identical, with some minor ordering changes:

The request headers of the redirect request, showing differences that are causing the problems:

Interestingly, an ssocompany cookie gets passed in the redirect header in 3.4.1 for some reason, perhaps the browser set the cookie with a time in the past and it hasn't run any expiry cleanup code before it sent the followup request? BUT this is not present in 3.6.1!
So I re-ran my test by adding a cy.setCookie('ssocompany') before the login script, and the login request (in either 3.4.1 or 3.6.1) doesn't clear the ssocompany cookie! BUT if I breakpoint before the login script and manually add a ssocompany cookie with the exact same details, it does get cleared during the request...??? Does using cy.setCookie somehow override any attempts by the site to expire a matching cookie? I'm very very very confused.

[emijmker]

@flotwig
I don't use cy.setCookies at all. In my tests I do in this order:

1. in the beforeEach a Cypress.Cookies.preserveOnce to preserve specific cookies that are automatically set after visiting that JWT url.
2. in the before a cy.clearCookies (otherwise sessions will get mixed).
3. as described before with the login and creating a session, do a cy.visit('jwtUrl')

The JWT token after the hostname is alfanumeric and can contain [ . or - or -- or _ ] and ends with ?ci=somealfanumericagain. Hope this screenshot will help to give you an idea, it's a fragment of the JSON response we get after creating a session by calling a graphql endpoint and that's what I use to cy.visit(). The blocked parts are all alfanumeric.

[tozes]

We are also experiencing issues with authentication on 3.5.0+, however, we observe slightly different behaviour than reported here.

We perform login with a cy.request() to a login endpoint and the response contains only one cookie in the set-cookie header but that cookie is not being set so the app fails to redirect even though the request to the login endpoint was successful.

We're also staying always inside the same domain/subdomain.

[flotwig]

@tozes Does your cookie have a Domain set?

[hornta]

@tozes we have the same problem. Only one ser-cookie header is received but it wont set it

[flotwig]

I believe the problem is most likely #5656, if you're setting a custom Domain property that will most likely fail because of this bug.

[tozes]

@flotwig #5656 is exactly what we're experiencing, we authenticate against the auth endpoint of something.ourdomain.com but the cookie contains domain=.ourdomain.com;
I'll follow that issue instead then, thanks a lot! 🙏

[flotwig]

Hey everyone, version 3.7.0 of Cypress has been released with some fixes for cookie behavior. Please try it out and see if it fixes your issue.

[pete-om]

@flotwig no change here :(

[celeryclub]

Can confirm that this fixes the issue for me. Thank you for all your work on this!

[irfancharania]

Can confirm 3.7 solved my issues also! Thanks! :)

[pete-om]

Hey again @flotwig - I've grabbed our product code this morning and started hacking around in it to try to isolate and reproduce the issue better locally, and it seems like while the first cookie does get processed and the second one doesn't, the existence of the multiple cookies was not the real issue for us :\

• I removed that first cookie, and only sending the auth token cookie through by itself still failed ☹!!
• So now that multiple cookies was ruled out, I looked closer at the auth token and noticed that while our website runs on for eg https://website.app.localhost/ (or website.domain.io for prod, etc) the auth token cookie domain is set to .app.localhost (to cover other subdomains) -- now when I change the code so the cookie domain is website.app.localhost the authentication through cy.request works!
• I reintroduced the first cookie (to double check) and the test still passed with two cookies in the array, when the auth token cookie had the modified domain
• I tried changing the cookie domain to just .localhost and that failed as well
• The first cookie didn't have a domain set so I guess that's why that one was processing

So it appears this is another cookie domain issue and unfortunately has nothing to do with the number of cookies sent through. I'm guessing it's just if the cookie domain doesn't match the full hostname, it doesn't get processed?

Apologies for the bum steer, hopefully this new info helps with finding the root cause! Happy to help further if you need more info or want me to try something else.

[polga]

We're still seeing the issue in 3.7.0, and have a setup the same as @pete-om. Changing the cookie domain to exactly match the URL fixes the problem. The (maybe) interesting thing is that we have an auto-retry plugin for some occasional timing fails, and the first retry actually succeeds. I'd also be happy to help further if necessary. Thanks!

[corwinstephen]

@flotwig I'll see if I can put something together, but in the mean time, it seems to happen when cookies are set using the leading dot (.domain.com), and if it helps, the issue is not present in 3.4.1, so whatever is causing it was introduced after that.

[flotwig]

@corwinstephen Sounds good, thank you for the information. To clarify, you're having issues with Set-Cookie headers that use Domain=.something.com not working as expected in 3.8.0?

[corwinstephen]

@flotwig yep that's correct, and also confirmed the issue is present in 3.7, 3.6, and 3.5

[flotwig]

@corwinstephen There are tests that cover setting a cookie with Domain=.something.com, so it is probably not related to that, it is most likely something else.

Unfortunately I am pretty much out of guesses as to what it could be, so this issue is blocked until someone can share a reproducible example repo.

[polga]

@flotwig the strange thing is that if I hardcode my cookie domain to have a subdomain, like app.something.com, it works.

I'd love to put something together, but for my situation it would take a lot of time to do. I'll see if I can chip away over the holidays, if no one does it sooner. Alternatively, I'd be happy to set up a screen share so you can see it occurring on my machine.

[corwinstephen]

@polga same here, explicitly setting the cookie with a subdomain works

[corwinstephen]

@flotwig Okay, some more detail on what's happening:

In 3.4.1, I make a cy.request to my login endpoint, and the result is a single cookie set with domain=.mydomain.com
In 3.8.0 the same request sets two cookies with the same name. One has domain .mydomain.com and the other mydomain.com. Subsequently, making a request to cy.getCookie('mycookiename') yields only the first one, which is not valid for my subdomain.

In my particular instance (Rails), I believe the invalid session on the subdomain is invalidating the entire session, causing both domains to be logged out.

[borecz]

No updates on this? I see no cookies related fixes in 3.8.2. I am still stuck with 3.4.1

[flotwig]

@borecz Still want to fix this issue, but I have not been able to reproduce it despite several attempts. I am hoping someone can supply some code that reproduces the issue.

[borecz]

May I reach you out in private? I could share the code there.

…

On 13. Jan 2020, at 15:18, Zach Bloomquist ***@***.***> wrote: @borecz <https://github.com/borecz> Still want to fix this issue, but I have not been able to reproduce it despite several attempts. I am hoping someone can supply some code that reproduces the issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5688?email_source=notifications&email_token=ANS5YMVVUCAOW42XLSA5D33Q5RZ3PA5CNFSM4JM6OMT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIY3VCQ#issuecomment-573684362>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANS5YMQBQM2PQG5T4NZ7WDDQ5RZ3PANCNFSM4JM6OMTQ>.

[flotwig]

@borecz Sure, my email is on my GitHub profile.

[corwinstephen]

@borecz were you guys able to connect on this?

[borecz]

Yes, we did. Zac has access to the working example.

…

On 17. Jan 2020, at 18:43, Stephen Elizabeth ***@***.***> wrote: @borecz <https://github.com/borecz> were you guys able to connect on this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5688?email_source=notifications&email_token=ANS5YMQ6PMLLQUZQH6Z6QCLQ6HU3PA5CNFSM4JM6OMT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJIOA7A#issuecomment-575725692>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANS5YMRB3DZ5262TMT2RP5DQ6HU3PANCNFSM4JM6OMTQ>.

[corwinstephen]

@borecz awesome. Thanks for biting the bullet on that one

[johannmayer]

Any news on this?

[corwinstephen]

@flotwig is there still activity on this?

[flotwig]

@corwinstephen I have been unable to devote a lot of time to this lately, working on getting other things out the door. But this is my priority as soon as 4.0 is out.

[corwinstephen]

An update here. I managed to get this working by grabbing the cookies set via cy.request, storing the value, then using cy.clearCookie to remove both cookies (the proper one and the duplicate), and then resetting just the proper cookie using cy.setCookie with the stored value.

[pete-om]

Any update on progress for this, @flotwig ? Cheers

[flotwig]

@pete-om The repo @borecz shared did not seem to reproduce this issue, so I am still unable to make any progress as I do not have an example of the bug to work with.

If nobody can provide a reproduction still, here is a list of locations to look at to try and debug this issue, if anyone wants to try to fix this themselves and open a PR:

• Functions responsible for setting/getting cookies during cy.request and cy.visit:

  cypress/packages/server/lib/request.js

  Lines 511 to 599 in e0b31cb

  	setRequestCookieHeader (req, reqUrl, automationFn, existingHeader) {
  	return automationFn('get:cookies', { url: reqUrl })
  	.then((cookies) => {
  	debug('got cookies from browser %o', { reqUrl, cookies })
  	let header = cookies.map((cookie) => {
  	return `${cookie.name}=${cookie.value}`
  	}).join('; ') || undefined
  	if (header) {
  	if (existingHeader) {
  	// existingHeader = whatever Cookie header the user is already trying to set
  	debug('there is an existing cookie header, merging %o', { header, existingHeader })
  	// order does not not matter here
  	// @see https://tools.ietf.org/html/rfc6265#section-4.2.2
  	header = [existingHeader, header].join(';')
  	}
  	return caseInsensitiveSet(req.headers, 'Cookie', header)
  	}
  	})
  	},
  	setCookiesOnBrowser (res, resUrl, automationFn) {
  	let cookies = res.headers['set-cookie']
  	if (!cookies) {
  	return Promise.resolve()
  	}
  	if (!(cookies instanceof Array)) {
  	cookies = [cookies]
  	}
  	const parsedUrl = url.parse(resUrl)
  	const defaultDomain = parsedUrl.hostname
  	debug('setting cookies on browser %o', { url: parsedUrl.href, defaultDomain, cookies })
  	return Promise.map(cookies, (cyCookie) => {
  	let cookie = tough.Cookie.parse(cyCookie, { loose: true })
  	debug('parsing cookie %o', { cyCookie, toughCookie: cookie })
  	if (!cookie) {
  	// ignore invalid cookies (same as browser behavior)
  	// https://github.com/cypress-io/cypress/issues/6890
  	debug('tough-cookie failed to parse, ignoring')
  	return
  	}
  	cookie.name = cookie.key
  	if (!cookie.domain) {
  	// take the domain from the URL
  	cookie.domain = defaultDomain
  	cookie.hostOnly = true
  	}
  	if (!tough.domainMatch(defaultDomain, cookie.domain)) {
  	debug('domain match failed:', { defaultDomain })
  	return
  	}
  	const expiry = cookie.expiryTime()
  	if (isFinite(expiry)) {
  	cookie.expiry = expiry / 1000
  	}
  	cookie.sameSite = convertSameSiteToughToExtension(cookie.sameSite, cyCookie)
  	cookie = _.pick(cookie, SERIALIZABLE_COOKIE_PROPS)
  	let automationCmd = 'set:cookie'
  	if (expiry <= 0) {
  	automationCmd = 'clear:cookie'
  	}
  	return automationFn(automationCmd, cookie)
  	.catch((err) => {
  	return debug('automation threw an error during cookie change %o', { automationCmd, cyCookie, cookie, err })
  	})
  	})
  	},

• Central cookie automation code: https://github.com/cypress-io/cypress/blob/e0b31cb2b12bbae3165ee01f22edb768c737529f/packages/server/lib/automation/cookies.js#L1-L178
• Electron + Chrome code for massaging cookies for the Chrome DevTools Protocol: https://github.com/cypress-io/cypress/blob/e0b31cb2b12bbae3165ee01f22edb768c737529f/packages/server/lib/browsers/cdp_automation.ts#L1-L208
• Firefox code for massaging cookies for WebExtensions: https://github.com/cypress-io/cypress/blob/e0b31cb2b12bbae3165ee01f22edb768c737529f/packages/extension/app/background.js#L1-L231
• Most of the cookies e2e tests (run by running the 2_cookies_spec e2e test): https://github.com/cypress-io/cypress/blob/e0b31cb2b12bbae3165ee01f22edb768c737529f/packages/server/test/support/fixtures/projects/e2e/cypress/integration/cookies_spec_baseurl.coffee#L1-L326

[corwinstephen]

@flotwig @pete-om @borecz fwiw my issues here were resolved in 4.3.0

[pete-om]

🤦‍♂️ I'd been scanning release notes every update but managed to miss #6628 in the list for 4.3.0, and that sounds exactly like my problem as described in this thread above. Have updated to latest and my issues are now resolved too. Thanks @flotwig @corwinstephen

[flotwig]

Nice!

Closing this issue since @pete-om was the OP. Anyone who is still experiencing cookie issues in the latest version of Cypress, please search the repo for the issue or open a new issue if one does not exist.