Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(sitemap): filter all routes with robots meta containing noindex #7964

Merged
merged 2 commits into from Aug 17, 2022
Merged

fix(sitemap): filter all routes with robots meta containing noindex #7964

merged 2 commits into from Aug 17, 2022

Conversation

slorber
Copy link
Collaborator

@slorber slorber commented Aug 17, 2022 •
edited

Motivation

Bug:

This page meta leads to the page being filtered correctly from sitemap.xml:

<head>
  <meta name="robots" content="noindex" />
</head>

But with this page meta, the page stays in sitemap.xml

<head>
  <meta name="robots" content="noindex, nofollow" />
</head>

Both pages should be filtered consistently.

Also Robots meta directives are case insensitive (https://developers.google.com/search/docs/advanced/robots/robots_meta_tag#directives)

Adding an example and improving our SEO doc a little

Test Plan

unit tests

Compared preview sitemap with a previous preview sitemap and didn't notice anything wrong

Test links

Deploy preview:

Related issues/PRs

#7963 (review)

@slorber slorber added the pr: bug fix This PR fixes a bug in a past release. label Aug 17, 2022
@facebook-github-bot facebook-github-bot added the CLA Signed Signed Facebook CLA label Aug 17, 2022
@netlify
Copy link

netlify bot commented Aug 17, 2022

✅ [V2]

Name Link
🔨 Latest commit d9e41c6
🔍 Latest deploy log https://app.netlify.com/sites/docusaurus-2/deploys/62fce8493671ff0008c86f9f
😎 Deploy Preview https://deploy-preview-7964--docusaurus-2.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@github-actions
Copy link

⚡️ Lighthouse report for the deploy preview of this PR

URL Performance Accessibility Best Practices SEO PWA Report
/ 🟢 93 🟢 98 🟢 100 🟢 100 🟠 80 Report
/docs/installation 🟠 82 🟢 100 🟢 100 🟢 100 🟢 90 Report

@github-actions
Copy link

Size Change: +270 B (0%)

Total Size: 813 kB

Filename Size Change
website/build/assets/js/main.********.js 609 kB +270 B (0%)
ℹ️ View Unchanged
Filename Size
website/.docusaurus/globalData.json 52.5 kB
website/build/assets/css/styles.********.css 111 kB
website/build/index.html 40.8 kB

compressed-size-action

@Josh-Cena Josh-Cena changed the title fix(sitemap-plugin): should filter routes with robots meta "noiNdeX, noFFoLow" from sitemap.xml fix(sitemap): filter all routes with robots meta containing noindex from sitemap Aug 17, 2022
@Josh-Cena Josh-Cena changed the title fix(sitemap): filter all routes with robots meta containing noindex from sitemap fix(sitemap): filter all routes with robots meta containing noindex Aug 17, 2022
Josh-Cena
Josh-Cena approved these changes Aug 17, 2022
View reviewed changes
Copy link
Collaborator

@Josh-Cena Josh-Cena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@slorber slorber added the to backport This PR is planned to be backported to a stable version of Docusaurus label Aug 17, 2022
@slorber slorber merged commit a335a69 into main Aug 17, 2022
@slorber slorber deleted the slorber/no-Index-sitemap-bug branch August 17, 2022 14:41
slorber added a commit that referenced this pull request Sep 1, 2022
@slorber
fix(sitemap): filter all routes with robots meta containing noindex (#…
c458f28 
…7964)
slorber added a commit that referenced this pull request Sep 1, 2022
@slorber
fix(sitemap): filter all routes with robots meta containing noindex (#…
fdf0d80 
…7964)
@slorber slorber added backported This PR has been backported to a stable version of Docusaurus and removed to backport This PR is planned to be backported to a stable version of Docusaurus labels Sep 28, 2022
@slorber slorber mentioned this pull request Oct 19, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Josh-Cena Josh-Cena Josh-Cena approved these changes

@lex111 lex111 Awaiting requested review from lex111

Assignees
No one assigned
Labels
backported This PR has been backported to a stable version of Docusaurus CLA Signed Signed Facebook CLA pr: bug fix This PR fixes a bug in a past release.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@slorber @Josh-Cena @facebook-github-bot