Site is broken while syncing large site to s3

[marcus13371337]

Hello, and thanks for this great plugin.

I'm building a large e-commerce site with gatsby and s3-hosting where we have approx 25k pages which means there are approx 50k files that need to be synced (counting the page-data-files) to s3 when we deploy a new version. We have used your plugin to sync to the bucket which we use CloudFront in front of the bucket. Normally, everything goes smoothly.

However, when we are doing larger code changes (meaning almost all files will be changed), there are some issues with that the files are not uploaded fast enough. This means that the end-users will get an index.html file that points to page-data files that return 404 since they are not uploaded yet. This is a quite big deal for us as that means the site is unavailable for a couple of minutes (it takes a while to upload all 50k pages).

I'm really afraid of when we run into the problem that the sync to s3 fails halfway through, which would mean a broken build until the next build is up and running.

Do you have any good ideas on how we can solve this issue? I guess a good solution would be to do some kind of "bucket"-rotation, i.e ask your plugin to sync to a separate bucket and once that's finished, point CloudFront to the new bucket. Any ideas on how that can be done easily?

Thanks!

[YoshiWalsh]

Hi Marcus,

I think your approach of using multiple buckets is a good idea. If you'd like to automate the process, you can use AWS CLI (or AWS SDK) to change the Origin of your CloudFront distribution. Keep in mind that changes to CloudFront distributions can take some time to apply to all edge nodes.

When you run gatsby-plugin-s3 deploy you can use the --bucket CLI argument to specify which bucket you'd like to upload to.

Hope this helps,
Josh

[marcus13371337]

Cool! I will investigate that.

Another idea I came up with was that if we would upload the files in the following order:

1. js/json-files
2. html-files
3. Delete previous files

The issue I was experiencing would have less impact as well. What do you think about that feature request? 😀

[drestrepom]

Hi Marcus

In my team we also have the same problem, but we are uploading the files directly to s3 after compilation without using this plugin, I just wanted to use the plugin to solve the problem, but I see that not work

Did you solve the problem? If so, I would like to know how

Thanks

[marcus13371337]

Hi @drestrepom ,

No, unfortunately, we haven't solved this yet. It's a bit complex as we need to reconfigure our bucket-origin in our CloudFront configuration (if we would publish new versions in a new bucket), which requires some integration work. We also considered hosting it with a bucket prefix, but as far as I know, we need to add some edge-lambda functions in order to properly handle redirects and similar stuff.

As far as I know, the problem only occurs during code-changes (and not content changes), and a lot of times it's almost not even noticeable except if all our pages (approx 30k) changes at the same time. We try to do that when there are just a few users on our page. However, not an optimal setup!