Support deployment-specific robots.txt files

[ciyer]

Motivation

Deployments should be able to control how the content gets indexed.

Proposal

It seems natural to support configuring the robots.txt file via helm/k8s

[lorenzo-cavazzi]

Could you please expand a bit on this with more context and an example?
Currently, we create a robots.txtfile that contains just a pointer to the local sitemaps. Is the proposal to have a section in the values files listing all the content that should go there?

E.G.

client:
    robots-txt:
        enabled: true
        data:
            - "Sitemap: {{localUrl}}/sitemap.xml"

Or is it intended to work in a different way?

[lorenzo-cavazzi]

@ciyer please let me know if what I asked here makes sense ☝️
It's not totally clear to me what should be implemented to close this issue