Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Set METAREFRESH_IGNORE_TAGS to ["noscript"] by default #6342

Closed
Gallaecio opened this issue May 7, 2024 · 3 comments · Fixed by #6347
Closed

Set METAREFRESH_IGNORE_TAGS to ["noscript"] by default #6342

Gallaecio opened this issue May 7, 2024 · 3 comments · Fixed by #6347
Labels
backward-incompatible bug good first issue

Comments

@Gallaecio
Copy link
Member

Gallaecio commented May 7, 2024 •
edited

I was wrong in #3844. The default value should be ["noscript"], to deal with antibot behaviors.

Found by @mukthy.
@kumar-sanchay
Copy link
Contributor

on it

aisha-partha added a commit to aisha-partha/scrapy that referenced this issue May 8, 2024
@aisha-partha
Closes scrapy#6342. Setting METAREFRESH_IGNORE_TAGS to [noscript] by …
2adfc20 
…default
aisha-partha added a commit to aisha-partha/scrapy that referenced this issue May 8, 2024
@aisha-partha
Closes scrapy#6342. Setting METAREFRESH_IGNORE_TAGS to ["noscript"] b…
17b2046 
…y default
aisha-partha added a commit to aisha-partha/scrapy that referenced this issue May 8, 2024
@aisha-partha
Closes scrapy#6342. Setting METAREFRESH_IGNORE_TAGS to [‘noscript’] b…
180bc9b 
…y default
aisha-partha added a commit to aisha-partha/scrapy that referenced this issue May 8, 2024
@aisha-partha
Closes scrapy#6342. Update documentation on METAREFRESH_IGNORE_TAGS d…
3590a1f 
…efault value change
@Gallaecio Gallaecio mentioned this issue May 8, 2024
Merged
wRAR added a commit that referenced this issue May 10, 2024
@wRAR
Merge pull request #6347 from aisha-partha/master
10a843a 
Closes #6342. Setting METAREFRESH_IGNORE_TAGS to [‘noscript’] by default
@wRAR
Copy link
Member

wRAR commented May 10, 2024

Should we cherry-pick this to 2.11?

@Gallaecio
Copy link
Member Author

+0.5 from me. It is technically backward-incompatible, but it should be so for the better in all scenarios I can think of. And it is trivial to override if that’s not the case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
backward-incompatible bug good first issue
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Closes #6342. Setting METAREFRESH_IGNORE_TAGS to [‘noscript’] by default aisha-partha/scrapy
3 participants
@wRAR @Gallaecio @kumar-sanchay