Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

301 Redirect #208

Open
ahmadkhalaf1 opened this issue Nov 10, 2020 · 37 comments
Open

301 Redirect #208

ahmadkhalaf1 opened this issue Nov 10, 2020 · 37 comments
Assignees
Labels
bug Something isn't working

Comments

@ahmadkhalaf1
Copy link

ahmadkhalaf1 commented Nov 10, 2020

Hello :)

I am trying to apply 301 redirect with my Gatsby and S3 + Cloudfront , and so far i have no success ,
i have about 204 redirect url , i will list below my configuration .

fromPath: `/old-url`,
toPath: `/new-url`,
isPermanent: true,
redirectInBrowser: true,
});
`


 {
      resolve: 'gatsby-plugin-s3',
      options: {
        bucketName: process.env.S3_BUCKET,
        region: 'eu-west-1',
        protocol: 'https',
        hostname: 'stage-mydomain.de',
        bucketPrefix: 'de',
        generateRoutingRules: true,
        generateRedirectObjectsForPermanentRedirects: true,
      }
    }

note that i have no plugin related to redirect inside my config , 
what i would like to accomplish is pure 301 server side redirect , i dont want JS redirect and Meta redirect because 
Js will not work with JS disabled and Meta is not good for SEO . 

can someone help please? 
Thanks

> 
@YoshiWalsh
Copy link
Collaborator

Your config looks fine to me. Could you please look in .cache/s3.redirectObjects.json after running gatsby build and see if the expected redirects are in there?

@ahmadkhalaf1
Copy link
Author

ahmadkhalaf1 commented Nov 10, 2020

Your config looks fine to me. Could you please look in .cache/s3.redirectObjects.json after running gatsby build and see if the expected redirects are in there?

Thank you for your Reply :)

i can see the file you mentioned with 203 object as the following

	"fromPath": "/de/old-url",
	"isPermanent": true,
	"redirectInBrowser": false,
	"toPath": "/de/new-url",
	"force": true
}

which should all correct and good , but still not working :( 

@ahmadkhalaf1
Copy link
Author

please note that my S3 Bucket contain 1 main folder
image
/de and my build files are inside this folder , in case it matter

@ahmadkhalaf1
Copy link
Author

ahmadkhalaf1 commented Nov 10, 2020

also i am doing Gzip to my build like this , not sure if it has any side effect .
inside gatsby-node.js

ps: i tried without this code as well and still not working (return 404 when i try to open an old url that must redirect to new url )

exports.onPostBuild = () =>
  new Promise((resolve, reject) => {
    try {
      const options = { nodir: true };
      const publicPath = path.join(__dirname, 'public');
      const gzippable = glob.sync(`${publicPath}/**/?(*.html|*.js|*.css|*.svg|*.gif)`, options);
      gzippable.forEach(file => {
        const content = fs.readFileSync(file);
        const zipped = zlib.gzipSync(content);
        fs.writeFileSync(`${file}.gz`, zipped);
        const brotlied = iltorb.compressSync(content);
        fs.writeFileSync(`${file}.br`, brotlied);
      });
    } catch (e) {
      reject(new Error('onPostBuild: Could not compress the files'));
    }
    resolve();
  });

@YoshiWalsh
Copy link
Collaborator

What CDN are you using? How do you make the client load the .gz/.br files? Your gzipping code should not break anything, but it won't include the redirect objects (and even if it did, they wouldn't be uploaded with the correct metadata). If you are using something like Lambda@Edge to rewrite requests to include .gz or .br, these requests will not hit the redirect object and would result in a 404.

If you are using CloudFront, note that Gzip and Brotli are supported natively. You don't have to do the compression yourself, you can just enable the compression in your cache policy.

@ahmadkhalaf1
Copy link
Author

What CDN are you using? How do you make the client load the .gz/.br files? Your gzipping code should not break anything, but it won't include the redirect objects (and even if it did, they wouldn't be uploaded with the correct metadata). If you are using something like Lambda@Edge to rewrite requests to include .gz or .br, these requests will not hit the redirect object and would result in a 404.

If you are using CloudFront, note that Gzip and Brotli are supported natively. You don't have to do the compression yourself, you can just enable the compression in your cache policy.

yes i am using CloudFront , so its enough to set this Compress Objects Automatically to Yes inside my cloudfront Behavior right?

i will remove it then :) thanks for the info , but what should i do to fix the redirect ?

thanks

@YoshiWalsh
Copy link
Collaborator

YoshiWalsh commented Nov 10, 2020

Please try disabling your custom gzip/brotli code along with any related Lambda@Edge script and enable the Compress Objects Automatically setting and see if that fixes it. It's my hunch that the redirect issue was related to the Lambda@Edge you were using to load the .gz and .br files. (Although at this point I don't have enough information to be sure)

@ahmadkhalaf1
Copy link
Author

Please try disabling your custom gzip/brotli code along with any related Lambda@Edge script and enable the Compress Objects Automatically setting and see if that fixes it. It's my hunch that the redirect issue was related to the Lambda@Edge you were using to load the .gz and .br files. (Although at this point I don't have enough information to be sure)

Thanks :) i only use 2 Lambda@Edge , one for cache and one for loading index.html.
sadly i already removed the code of Gzip from Gatsby and tried and still return 404 :(

thanks for your time

@ahmadkhalaf1
Copy link
Author

ahmadkhalaf1 commented Nov 10, 2020

Origin Request :

  const request = event.Records[0].cf.request;
  const uri = request.uri;

  if (uri.endsWith('/')) {
    request.uri += 'index.html';
  } else if (!uri.includes('.')) {
    request.uri += '/index.html';
  }

  callback(null, request);
};

Origin Response

exports.handler = (event, context, callback) => {
  const request = event.Records[0].cf.request;
  const response = event.Records[0].cf.response;
  const headers = response.headers;

  if (request.uri.startsWith('/static/')) {
    headers['cache-control'] = [
      {
        key: 'Cache-Control',
        value: 'public, max-age=31536000, immutable'
      }
    ];
  } else {
    headers['cache-control'] = [
      {
        key: 'Cache-Control',
        value: 'public, max-age=0, must-revalidate'
      }
    ];
  }
  
  headers['vary'] = [
    {
      key: 'Vary',
      value: 'Accept-Encoding'
    }
  ];

  callback(null, response);
};

@YoshiWalsh
Copy link
Collaborator

Can you please check if an object has been uploaded to /de/old-url or /de/old-url/index.html? If so, can you please check if the object has x-amz-website-redirect-location in its metadata?

Your remaining Lamda's are also unnecessary, assuming that you're using S3 Static Website Hosting. S3 Static Website Hosting automatically adds trailing slashes and index documents:

If you create a folder structure in your bucket, you must have an index document at each level. In each folder, the index document must have the same name, for example, index.html. When a user specifies a URL that resembles a folder lookup, the presence or absence of a trailing slash determines the behavior of the website. For example, the following URL, with a trailing slash, returns the photos/index.html index document.

http://bucket-name.s3-website.Region.amazonaws.com/photos/

However, if you exclude the trailing slash from the preceding URL, Amazon S3 first looks for an object photos in the bucket. If the photos object is not found, it searches for an index document, photos/index.html. If that document is found, Amazon S3 returns a 302 Found message and points to the photos/ key. For subsequent requests to photos/, Amazon S3 returns photos/index.html. If the index document is not found, Amazon S3 returns an error.

gatsby-plugin-s3 takes care of Cache-Control for you, we follow Gatsby's recommendations. If you want to customise this you can use our params config setting. CloudFront will automatically take care of Accept-Encoding for you.

Did you follow a tutorial that recommended them? I suggest you follow our own documentation which includes everything you need in order to get a Gatsby site deployed to S3+CloudFront with caching and serverside redirects. I'm happy to keep troubleshooting with you, but the easiest/fastest way to get your site working might be to start again and follow our recipe.

@ahmadkhalaf1
Copy link
Author

Thanks a lot ! i will check the documentation u mentioned and update you with the result tomorrow ^_^ thanks again for your time , good night

@ahmadkhalaf1
Copy link
Author

@JoshuaWalsh ok i guess i find out why its not working , and i think ( not so sure ) that there is a bug that does not handle pathPrefix with createRedirect in Gatsby plugin s3 .

so in short , here what i could find out
in my config i have

pathPrefix: '/de',

and 
bucketPrefix: 'de', in gatsby plugin s3 option .

so my main domain is , https:/www.mydomain.de/de/index-page-title/slug-title for example .

so after my build is done , my s3 bucket contain 1 main folder de
and all my build files are inside this folder.

create redirect

 createRedirect({
            fromPath: `/${indexPageTitle}/${oldSlug}`,
            toPath: `/${indexPageTitle}/${newSlug}`,
            isPermanent: true,
            force: true,
          });
checking my ```s3.redirectObject.json ```
i can see it contain 

{
	"fromPath": "/de/index-page-title/old-slug",
	"isPermanent": true,
	"redirectInBrowser": false,
	"toPath": "/de/index-page-title/new-slug",
	"force": true
}

now at the final step , when i do
npx -n \"-r dotenv/config\" gatsby-plugin-s3 deploy --yes

all my build files are copied under de folder in my s3 bucket , but what i find out that all redirect objects are copied under another /de folder inside the main /de folder which is wrong and i am kind of sure that this is the problem .

this is one of the output i see while syncing to s3 
⠦ Syncing...
  Created Redirect de/de/index-title/old-url => https://stage-mydomain.de/de/index-title/new-url

as u see the problem is here ,de/de , this is why i assume its an issue with this step of Gatsby Plugin S3.

and this is a picture how it look like inside my S3 Bucket

this is the main DE folder
image

and this is what we see inside the de folder
image

now inside the second de folder i can find folders with (index-title) name , and inside it there is an object with meta data
x-amz-website-redirect-location pointing to the right redirect url .
but its an object not a folder with index.html inside , is this how it should be?

Thanks and sorry for the long post :)
i hope we can solve this before my boss get more angry :D

@YoshiWalsh
Copy link
Collaborator

Good find. Prefixes are the cause of quite a bit of confusion, as seen in #24.

Could you try including Gatsby's pathPrefix and not including our bucketPrefix?

@ahmadkhalaf1
Copy link
Author

ahmadkhalaf1 commented Nov 11, 2020

Good find. Prefixes are the cause of quite a bit of confusion, as seen in #24.

Could you try including Gatsby's pathPrefix and not including our bucketPrefix?

now this will definitely upload my build in the root of the s3 bucket out side of the main de folder and routing with /de prefix will not work , but i will check at least the redirect thingy , testing now , i will update you here soon :)

thanks

@ahmadkhalaf1
Copy link
Author

ahmadkhalaf1 commented Nov 11, 2020

@JoshuaWalsh ok now i see

⠦ Syncing...
  Created Redirect de/index-title/old-url => https://stage-mydomain.de/de/index-title/new-url

with one de , which is good i guess .

but in the root of my s3 bucket , i can see de/ folder and all redirect content objects (not folders with index.html) is inside it with the right metadata.

@YoshiWalsh
Copy link
Collaborator

What are you using to create the redirects? Are you calling createRedirect directly or using a plugin such as gatsby-redirect-from?

At this point I am confident that there is a bug in gatsby-plugin-s3. When generateRedirectObjectsForPermanentRedirects is false we do not prefix the fromPath with bucketPrefix, but when generateRedirectObjectsForPermanentRedirects is true we do add the prefix.

What's less obvious to me is which behaviour is correct. As I expressed on #24 I think Gatsby's pathPrefix should be used to specify the client-facing path prefix (the one in the URL) and our own bucketPrefix should be used to specify the prefix for the objects in S3. In your case these are both the same, but in general this might not always be true. It seems to me that bucketPrefix should be added to the S3 redirects and that the issue here is that pathPrefix is also being added. So I want to understand if pathPrefix is being added by Gatsby itself, or if it's related to the way you are creating redirects.

@ahmadkhalaf1
Copy link
Author

ahmadkhalaf1 commented Nov 11, 2020

What are you using to create the redirects? Are you calling createRedirect directly or using a plugin such as gatsby-redirect-from?

At this point I am confident that there is a bug in gatsby-plugin-s3. When generateRedirectObjectsForPermanentRedirects is false we do not prefix the fromPath with bucketPrefix, but when generateRedirectObjectsForPermanentRedirects is true we do add the prefix.

What's less obvious to me is which behaviour is correct. As I expressed on #24 I think Gatsby's pathPrefix should be used to specify the client-facing path prefix (the one in the URL) and our own bucketPrefix should be used to specify the prefix for the objects in S3. In your case these are both the same, but in general this might not always be true. It seems to me that bucketPrefix should be added to the S3 redirects and that the issue here is that pathPrefix is also being added. So I want to understand if pathPrefix is being added by Gatsby itself, or if it's related to the way you are creating redirects.

make sense , i am using createRedirect inside exports.createPages
using graphQL i fetch an array of objects , i loop inside it and i createPage and check if there is a redirect needed and call

 createRedirect({
            fromPath: `/${indexPageTitle}/${oldSlug}`,
            toPath: `/${indexPageTitle}/${newSlug}`,
            isPermanent: true,
            force: true,
          });

inside a loop, so no i am not using the plugin you mentioned.

@YoshiWalsh
Copy link
Collaborator

YoshiWalsh commented Nov 11, 2020

It seems from here and here that maybe Gatsby itself is prefixing the paths. I'll try to raise this with the Gatsby Core team and see what they advise. This is unlikely to be a quick fix sorry, but I want to make sure we do this right.

In the meantime, your best option might be to create a private fork and remove these lines.

@ahmadkhalaf1
Copy link
Author

@JoshuaWalsh Thank you so much for your effort ,
i am kind of a noob in the fork thingy , i will check how i can do this and try it out :)

please let me know once you hear from Gatsby or if you could fix this somehow in s3 plugin.

Thanks

@YoshiWalsh
Copy link
Collaborator

YoshiWalsh commented Nov 11, 2020

Creating a fork duplicates the project into a repository that you own. You can change that freely. If we make updates to our version you can pull those into your repository, and if you make changes that you think we'd like you can submit a Pull Request to us.

In your website's package.json you have a line that looks something like this:

"gatsby-plugin-s3": "^0.3.8"

If your fork is hosted on GitHub, you can alter this to point to your fork by doing something like:

"gatsby-plugin-s3": "ahmadkhalaf1/gatsby-plugin-s3"

You can find more details here:

Best of luck :)

@YoshiWalsh YoshiWalsh self-assigned this Nov 11, 2020
@YoshiWalsh YoshiWalsh added the bug Something isn't working label Nov 11, 2020
@YoshiWalsh
Copy link
Collaborator

gatsbyjs/gatsby#27988

@ahmadkhalaf1
Copy link
Author

Creating a fork duplicates the project into a repository that you own. You can change that freely. If we make updates to our version you can pull those into your repository, and if you make changes that you think we'd like you can submit a Pull Request to us.

In your website's package.json you have a line that looks something like this:

"gatsby-plugin-s3": "^0.3.8"

If your fork is hosted on GitHub, you can alter this to point to your fork by doing something like:

"gatsby-plugin-s3": "ahmadkhalaf1/gatsby-plugin-s3"

You can find more details here:

Best of luck :)

sorry for headache , i tried many ways to install the fork to my Gatsby , every time i get this error

npm ERR! code ENOENT
npm ERR! syscall chmod
npm ERR! path /home/ahmad/Desktop/Projects/project-gatsby/node_modules/gatsby-plugin-s3/bin.js
npm ERR! errno -2
npm ERR! enoent ENOENT: no such file or directory, chmod '/home/ahmad/Desktop/Projects/project-gatsby/node_modules/gatsby-plugin-s3/bin.js'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent 

@YoshiWalsh
Copy link
Collaborator

In your forked gatsby-plugin-s3 run npm ci and then npm run-script build.

@ahmadkhalaf1
Copy link
Author

In your forked gatsby-plugin-s3 run npm ci and then npm run-script build.

but this should work locally only?
how can i do the following commands u mentioned when i deploy my repo to my stage env ?

@YoshiWalsh
Copy link
Collaborator

Good question, it was late at night when I gave my last reply and I didn't think it through properly. 3 options:

  1. Publish your fork to npm. Then in your website, change package.json to say "ahmadkhalaf1/gatsby-plugin-s3": "^0.3.8"

  2. Make a branch in your fork called "build" or "javascript". Remove *.js and *.d.ts from .gitignore on this branch. On your website, change package.json to say "gatsby-plugin-s3": "ahmadkhalaf1/gatsby-plugin-s3#build"

  3. On your CI, after you've run npm ci for your project, run:

cd ./node_modules/gatsby-plugin-s3
npm ci
npm run-script build

(That last one might or might not work, I've had issues before with npm in nested projects)

@ahmadkhalaf1
Copy link
Author

Good question, it was late at night when I gave my last reply and I didn't think it through properly. 3 options:

  1. Publish your fork to npm. Then in your website, change package.json to say "ahmadkhalaf1/gatsby-plugin-s3": "^0.3.8"
  2. Make a branch in your fork called "build" or "javascript". Remove *.js and *.d.ts from .gitignore on this branch. On your website, change package.json to say "gatsby-plugin-s3": "ahmadkhalaf1/gatsby-plugin-s3#build"
  3. On your CI, after you've run npm ci for your project, run:
cd ./node_modules/gatsby-plugin-s3
npm ci
npm run-script build

(That last one might or might not work, I've had issues before with npm in nested projects)

so i used first option and i published the fork to NPM
https://www.npmjs.com/package/ahmad-gatsby-plugin-s3
then i changed the name inside my config ,
sadly the lines u told me to remove did not really do anything :D , redirect is still created inside another de/ folder inside the main de/ and still return 404 :(

{
resolve: 'ahmad-gatsby-plugin-s3',
options: {
bucketName: process.env.S3_BUCKET,
region: 'eu-west-1',
protocol: 'https',
hostname: 'stage-gatsby.mymoria.de',
bucketPrefix: 'de',
generateRoutingRules: true,
generateRedirectObjectsForPermanentRedirects: true,
},
},

@YoshiWalsh
Copy link
Collaborator

YoshiWalsh commented Nov 15, 2020

What command are you using to do the deployment? Is it possible your deploy command is still using gatsby-plugin-s3 and not ahmad-gatsby-plugin-s3?

@ahmadkhalaf1
Copy link
Author

ahmadkhalaf1 commented Nov 15, 2020

What command are you using to do the deployment? Is it possible your deploy command is still using gatsby-plugin-s3 and not ahmad-gatsby-plugin-s3?

yes !!! you are right , i forgot to change it sorry .

so now things looks much much better and i am sure we are about to fix this , so now after i deploy this is how redirect looks like now .

lets say i have redirect from
de/index-page/old-url
to
de/index-page/new-url

now in my s3 i have 1 main de/ folder which is great
inside it i have index-page/ folder
it contain 2 items

1- new-url/ as a folder and inside it index.html

2- an object old-url with meta data


System defined | x-amz-website-redirect-location | https://stage-mydomain.de/de/index-page/new-url

with all that said , sadly redirect is not working and still return 404 . 

thanks

@YoshiWalsh
Copy link
Collaborator

9/10 times when redirects aren't working it's because CloudFront is configured with an "S3 Origin" instead of using the S3 Static Website Hosting endpoint. Please check your CloudFront Origin settings and verify that it is set to "Custom Origin" with a URL that looks like this: http://stage-mydomain.s3-website-eu-west-1.amazonaws.com/

@ahmadkhalaf1
Copy link
Author

9/10 times when redirects aren't working it's because CloudFront is configured with an "S3 Origin" instead of using the S3 Static Website Hosting endpoint. Please check your CloudFront Origin settings and verify that it is set to "Custom Origin" with a URL that looks like this: http://stage-mydomain.s3-website-eu-west-1.amazonaws.com/

you are right again ^^
image

i am trying to find out how to change it to Custom Origin , because when i click edit , i see that i have 3 values
1-Origin Domain Name (stage-mydomain.de.s3.amazonaws.com)
2-Origin ID (stage-mydomain.de)
3-Origin Path (empty)
but no other option to change origin type , i will search around and see how to do that :D

will keep you updated

@YoshiWalsh
Copy link
Collaborator

I think you need to delete the origin and add a new one.

@ahmadkhalaf1
Copy link
Author

but when i click create origin
image

i get the same fields

@YoshiWalsh
Copy link
Collaborator

Check out the "Make a CloudFront Distribution" sectin of this guide. They added the origin while creating the distribution, but you can do the same thing when adding an origin to an existing distribution. Basically you just need to put the S3 Static Website Hosting endpoint in the Origin Domain Name field.

@ahmadkhalaf1
Copy link
Author

it working !!!!! i removed my lambda edges for Cache and add index.html as you once told me here
#208 (comment)
and now its working !!!

it was redirected in 2 stages
from 301 to 302 then to 200, no idea why :D but i am now super happy its working .

image

i want to thank you very much for your time and effort , i would like to tip you or buy you a beer :) please let me know how can i do that.

also would like to know what is the next step to fix this bug so i can switch back to your version and removed my Temp NPM .

Thanks a lot

@YoshiWalsh
Copy link
Collaborator

YoshiWalsh commented Nov 16, 2020

302 redirects in S3 Static Website Hosting are usually to add a trailing slash. If you add a trailing slash to your redirect's toPath it should only need a single 301 redirect.

Thanks for the offer to buy me a beer. We don't have any mechanism set up for tipping, and I'm just happy that it's working for you.

Fixing the bug permanently should be fairly straightforward, but first I want guidance from the Gatsby Core team on how they think redirects should be handled. Unfortunately they recently restructured their community, and now it's harder to get in touch with them. I might try resubmitting my question as a bug instead of a discussion to see if that gets better visibility.

At the moment I'm thinking the best solution would be for us to subtract pathPrefix from the beginning of the fromPath of all redirects, then to add bucketPrefix. This is basically just a workaround for what I consider to be a bug in Gatsby.

@aborza-c
Copy link

@JoshuaWalsh has there been any progress on this issue?

@kevin907
Copy link

Hello, Any updates on this issue? Was this fixed? current version of gatsby-plugin-s3 is 0.4.1, which was published 2 months ago. Does that resolve this error?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants