Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(docs): allow to configure noIndex per doc version #7963

Merged
merged 4 commits into from Aug 17, 2022

Conversation

slorber
Copy link
Collaborator

@slorber slorber commented Aug 17, 2022

Motivation

You might want to prevent crawlers and google to index older versions of your docs

See also #7960

Test Plan

unit tests + local tests + dogfood on test docs instance (as it's not really useful to index it)

Test links

Deploy preview:


Limitation: the versioned docs tag pages won't be noIndexed. But a further refactor will fix it soon. (fixed in #7966)

@slorber slorber added the pr: new feature This PR adds a new API or behavior. label Aug 17, 2022
@facebook-github-bot facebook-github-bot added the CLA Signed Signed Facebook CLA label Aug 17, 2022
@netlify
Copy link

netlify bot commented Aug 17, 2022

[V2]

Name Link
🔨 Latest commit 0c297a2
🔍 Latest deploy log https://app.netlify.com/sites/docusaurus-2/deploys/62fccf9168e3bf000ad57000
😎 Deploy Preview https://deploy-preview-7963--docusaurus-2.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@github-actions
Copy link

github-actions bot commented Aug 17, 2022

⚡️ Lighthouse report for the deploy preview of this PR

URL Performance Accessibility Best Practices SEO PWA Report
/ 🟢 93 🟢 98 🟢 100 🟢 100 🟠 80 Report
/docs/installation 🟠 86 🟢 100 🟢 100 🟢 100 🟢 90 Report

Copy link
Collaborator

@Josh-Cena Josh-Cena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: how does this play with the sitemap? Over there, we only test tag.props.name === 'robots' && tag.props.content === 'noindex'. Maybe it should be tag.props.content.includes("noindex") instead?

@netlify
Copy link

netlify bot commented Aug 17, 2022

[V2]

Name Link
🔨 Latest commit 290e926
🔍 Latest deploy log https://app.netlify.com/sites/docusaurus-2/deploys/62fcff5fd2aa190008702848
😎 Deploy Preview https://deploy-preview-7963--docusaurus-2.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@github-actions
Copy link

github-actions bot commented Aug 17, 2022

Size Change: +45 B (0%)

Total Size: 813 kB

Filename Size Change
website/build/assets/js/main.********.js 609 kB +45 B (0%)
ℹ️ View Unchanged
Filename Size
website/.docusaurus/globalData.json 52.5 kB
website/build/assets/css/styles.********.css 111 kB
website/build/index.html 40.8 kB

compressed-size-action

@slorber
Copy link
Collaborator Author

slorber commented Aug 17, 2022

Question: how does this play with the sitemap? Over there, we only test tag.props.name === 'robots' && tag.props.content === 'noindex'. Maybe it should be tag.props.content.includes("noindex") instead?

Yes definitively! I thought it was already the case actually 😅 will send another PR and edited our sitemap plugin so that we can see the problem. Will merge a sitemap plugin fix before this PR, ensuring we can see the problem fixed here.

@slorber
Copy link
Collaborator Author

slorber commented Aug 17, 2022

BTW I guess having a noIndex: true flag for plugin instances in a generic way could be useful 🤷‍♂️ we also want to have a label, wonder what else makes sense as a general option for a plugin instance.

Not going to implement this here, but let's keep this in mind

@slorber slorber added the to backport This PR is planned to be backported to a stable version of Docusaurus label Aug 17, 2022
@slorber
Copy link
Collaborator Author

slorber commented Aug 17, 2022

working fine now

those tag urls are still there in the sitemap and indexed, will do another refactor to fix that

<url>
<loc>https://docusaurus.io/tests/docs/tags/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url>
<url>
<loc>https://docusaurus.io/tests/docs/tags/a/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url>
<url>
<loc>https://docusaurus.io/tests/docs/tags/b/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url>
<url>
<loc>https://docusaurus.io/tests/docs/tags/c/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url>
<url>
<loc>https://docusaurus.io/tests/docs/tags/d-custom-permalink/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url>
<url>
<loc>https://docusaurus.io/tests/docs/tags/e/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url>
<url>
<loc>https://docusaurus.io/tests/docs/tags/some-tag/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url>

@slorber slorber merged commit eac3239 into main Aug 17, 2022
@slorber slorber deleted the slorber/noIndex-docs-version branch August 17, 2022 14:55
@slorber slorber added backported This PR has been backported to a stable version of Docusaurus and removed to backport This PR is planned to be backported to a stable version of Docusaurus labels Sep 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backported This PR has been backported to a stable version of Docusaurus CLA Signed Signed Facebook CLA pr: new feature This PR adds a new API or behavior.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants