Skip to content

Commit

Permalink
Added comments count endpoint to robots.txt disallow list
Browse files Browse the repository at this point in the history
fix https://linear.app/tryghost/issue/ENG-771/add-comments-count-endpoint-to-robotstxt-ignorelist

- we've seen web scrapers hitting this endpoint a lot, but the value to
  be taken from it is minimal for SEO purposes
- adding it to robots.txt should encourage web scrapers to ignore it,
  and we should see less traffic as a result
  • Loading branch information
daniellockyer committed Mar 20, 2024
1 parent 7cc65c1 commit 27cc32e
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 0 deletions.
1 change: 1 addition & 0 deletions ghost/core/core/frontend/public/robots.txt
Expand Up @@ -2,5 +2,6 @@ User-agent: *
Sitemap: {{blog-url}}/sitemap.xml
Disallow: /ghost/
Disallow: /email/
Disallow: /members/api/comments/counts/
Disallow: /r/
Disallow: /webmentions/receive/
1 change: 1 addition & 0 deletions ghost/core/test/e2e-frontend/default_routes.test.js
Expand Up @@ -323,6 +323,7 @@ describe('Default Frontend routing', function () {
'User-agent: *\n' +
'Sitemap: http://127.0.0.1:2369/sitemap.xml\nDisallow: /ghost/\n' +
'Disallow: /email/\n' +
'Disallow: /members/api/comments/counts/\n' +
'Disallow: /r/\n' +
'Disallow: /webmentions/receive/\n'
);
Expand Down

0 comments on commit 27cc32e

Please sign in to comment.