Make sure Netlify deployments are not listed with their .netlify.com domain in Google and Co

[janpio]

Currently this can happen (note the .netlify.com domain instead of prisma.io):

Unfortunately this is not just one domain, but this is happening for many of them:
https://www.google.com/search?q=site%3Anetlify.com+prisma

---

We need to implement something that tells search Engines like Google not to list these .netlify.com domains, while still not forbidding indexing of prisma.io, v1.prisma.io etc.

For many subpage deployments we can probably just add a robots.txt that forbids indexing (old docs, old homepages etc). For the main deployment of the website we will need to think of something smarter somehow.

After this is in place, we will need to tell Google to remove these pages and track the removal.

---

Internal discussion with possible workarounds or solutions:
https://prisma-company.slack.com/archives/C5Z9TH6N9/p1585147685000200

[Jolg42]

@janpio I suggest adding a robots.txt like this (deny all)

User-agent: *
Disallow: /

On all netlify deployments that are deployed as subdirectories.

And for the rest / in general use canonical urls with the production url.

[janpio]

Ok, that will require collecting and classifying all the hosts/deployments and then careful applying of these to not mess it up accidentally (especially with the Netlify redirect fallbacks in use sometimes).

[Jolg42]

It looks like we have about 3000 urls that should not be indexed, based on this Google search I found that we need to remove:

• https://homepage-v7-photonjs.netlify.com
• https://prisma-docs.netlify.com
• https://prisma2.netlify.com
• https://prisma-docs-till-1-33.netlify.com
• https://howtographql.netlify.com
• https://graphcool-v3.netlify.com
• https://master--howtographql.netlify.com

I also found these public netlify pages, I didn't these could be public 🙃

• https://app.netlify.com/sites/learnapollo/deploys
• https://app.netlify.com/sites/yoga2/deploys
• https://app.netlify.com/sites/graphql-playground/deploys
• https://app.netlify.com/sites/opencrud/deploys
• https://app.netlify.com/sites/oss-prisma-io/deploys
• https://app.netlify.com/sites/graphql-yoga/deploys
• https://app.netlify.com/sites/howtographql/deploys
• https://app.netlify.com/sites/graphql-europe-2018/deploys
• https://app.netlify.com/sites/learnrelay/deploys

[Jolg42]

For Netlify it looks like turning the logs to private makes the pages private, so I did it for most of them except https://app.netlify.com/sites/graphql-playground/deploys as there is some activity there.

[janpio]

based on this Google search I found that we need to remove:

We should do this more fundamentally and also apply this to other Netlify properties that are not listed yet - or they might be one day.

[Jolg42]

@janpio We can probably delete:
https://app.netlify.com/sites/homepage-v7-photonjs/overview
https://app.netlify.com/sites/homepage-v7-lift/overview

Because lift.prisma.io and photonjs.prisma.io are redirecting to:
https://github.com/prisma/prisma/releases/tag/2.0.0-preview020

[Jolg42]

@janpio We can probably redirect https://ru.howtographql.com/ to https://www.howtographql.com/

[Jolg42]

@janpio Do we still use it or can we remove it? (https://app.netlify.com/sites/prisma-docs-till-1-33/)
https://prisma-docs-till-1-33.netlify.app/
I we still use it I suggest adding a robots.txt but as it's a manual deploy not linked to a git, I have no idea who should do it.

[Jolg42]

@janpio https://prisma-docs.netlify.app is linked to https://github.com/graphcool/prisma-docs but 404, do you know where it lives now?

[janpio]

@janpio We can probably delete:
app.netlify.com/sites/homepage-v7-photonjs/overview
app.netlify.com/sites/homepage-v7-lift/overview

Because lift.prisma.io and photonjs.prisma.io are redirecting to:
2.0.0-preview020 (release)

#1949 (comment)
I set a password (WiFi password) for these: https://homepage-v7-photonjs.netlify.app/ instead of deleting, because we might want to look at these ourselves.
Not sure how Google will deal with this though.
Alternatively of course we again can add a robots.txt

@janpio We can probably redirect ru.howtographql.com to howtographql.com

Yep go ahead and fix this, maybe ask why this exists in the first place though before changing anything.

@janpio Do we still use it or can we remove it? (app.netlify.com/sites/prisma-docs-till-1-33)
prisma-docs-till-1-33.netlify.app
I we still use it I suggest adding a robots.txt but as it's a manual deploy not linked to a git, I have no idea who should do it.

Yes still in use at https://github.com/prisma/v1.prisma.io/blob/master/_redirects - @2color just deployed all these so can do that for you if needed.

@janpio prisma-docs.netlify.app is linked to graphcool/prisma-docs but 404, do you know where it lives now?

You can ask Dom for access to https://github.com/prisma/prisma-docs-v1 (which is the name of the private repo that backs this it seems) - again @2color can deploy stuff there if needed.

[2color]

I can take this over as I've recently deployed the v1 docs. Shouldn't be a problem to add the robots.txt file

[Jolg42]

I successfully added canonical on https://www.howtographql.com/ 🎊
Fun session with netlify deploys 😅

[2color]

Added robots.txt to https://prisma-docs.netlify.app/robots.txt

[Jolg42]

Google currently returns "About 1.400 results" I will check again in a few days.

[janpio]

https://www.google.com/webmasters/tools/removals?pli=1 might help if the correct robots.txt is in place.

[Jolg42]

I just requested the removals through your link @janpio 😃

[Jolg42]

It looks like some pages disappeared and for the rest it will take more time or it's not from us.

[janpio]

Problem still exists, now with a new TLD as well:
https://www.google.com/search?q=site%3Anetlify.app+prisma - About 1.290 results
https://www.google.com/search?q=site%3Anetlify.com+prisma - About 1.270 results

[Jolg42]

I requested removals from Google with both http and https urls for all of them including .com & .app we'll see how it ends up next week 🤷‍♂️

I checked https://prisma-docs-till-1-33.netlify.app/ (https://app.netlify.com/sites/prisma-docs-till-1-33/overview) and I can't find a robots.txt nor a canonical so this one need to be fixed first or maybe just redeployed if the fix was already done 🤔

Also for the number of results it's really not accurate the next pages are full of content that we don't own.

[Jolg42]

@2color just added https://prisma-docs-till-1-33.netlify.app/robots.txt 🎊
I'm requesting deletion from Google for it and closing for good as the other results have a canonical and requesting a Google update doesn't work, it will just disappear one day 🤷‍♂️

[janpio]

Some examples from my page 1:
https://prisma-docs-v2.netlify.app/robots.txt
https://howtographql.netlify.app/robots.txt
https://master--howtographql.netlify.app/robots.txt

[Jolg42]

@janpio We can't put a robots.txt on these pages because it will also apply to the production urls and get all the content down.

For these we added a canonical url but it seems that Google is kind of slow at picking that up and the removal tool is not cooperative on that either 🤷‍♂️