fix robots.txt so netlify deployment won't be indexed

[Jolg42]

Current robots.txt

User-agent: *
Allow: /
Sitemap: https://www.prisma.io/sitemap.xml
Host: https://www.prisma.io

https://www.prisma.io/docs/robots.txt
https://prisma2.netlify.app/robots.txt

This will change it so https://prisma2.netlify.app won't appear in the search results anymore.

https://www.prisma.io/docs/ will still be indexed.

Related to prisma/prisma#1949

[Jolg42]

Looks like it worked ;-)

https://deploy-preview-288--prisma2.netlify.app/robots.txt

[janpio]

Will this not publish the robots.txt as prisma.io/docs/robots.txt via the rewrite on the homepage-v8 repo?

Yep it will: https://www.prisma.io/docs/robots.txt
Although no bot will look at this, so it should be safe enough.

[Jolg42]

@janpio yes but it's ok, the robots.txt is only read by engines if it's at the root level.

[janpio]

The robots.txt will actually not be in prisma.io/docs any more if this repo is repaired so that the docs content actually works on the preview URLs of Netlify etc.

[Jolg42]

#288 (comment)
@janpio Not sure I understand 🤔 Also if this repo is repaired?

[janpio]

https://deploy-preview-288--prisma2.netlify.app/ is currently fully broken, you can not follow any of the links.

To fix this, this repo will probably have to be changed to serve https://deploy-preview-288--prisma2.netlify.app/docs/understand-prisma/introduction instead of https://deploy-preview-288--prisma2.netlify.app/understand-prisma/introduction/ - which would put its robots.txt also in a spot where it can not be accessed via prisma.io