Skip to content
This repository has been archived by the owner on Feb 25, 2022. It is now read-only.

Add option to save data to s3 #373

Open
tripodsan opened this issue May 20, 2021 · 2 comments
Open

Add option to save data to s3 #373

tripodsan opened this issue May 20, 2021 · 2 comments
Labels
enhancement New feature or request

Comments

@tripodsan
Copy link
Contributor

tripodsan commented May 20, 2021

@trieloff's idea to store (larger) data directly in the underlying storage would make the data processing faster, especially for larger data-sets.

suggest to:

  1. content-bus generates a presigned url for the respective destination object
  2. send request to content-proxy, including a presignedStorageUrl parameter
  3. content-proxy sends along the presignedStorageUrl to data-embed, ideally using a PUT1
  4. data-embed stores the .json directly in the storage using the presignedStorageUrl
  5. data-embeds responds with a 3072, including a location header to the location of the stored object (if possible)
  6. content-proxy returns the same

1 I'm not sure about using PUT or GET, but writing content on GET feels wrong
2 I'm not sure about the redirect response. maybe a 200 when using PUT is better.

@trieloff
Copy link
Contributor

  1. Definitely not GET, but POST would be acceptable

I'd use the universat storage abstraction to generate the presign URLs.

How do we handle multi-cloud? Run the trigger twice, in both clouds or have it generate multiple URLs?

@tripodsan
Copy link
Contributor Author

How do we handle multi-cloud? Run the trigger twice, in both clouds or have it generate multiple URLs?

for helix3, the content-bus needs to store the json into the s3/cs when requested (e.g. on preview). since the content-bus is the intitiator, the corresponding content-bus will invoke content-proxy/data-embed on the corresponding cloud which will store it in the corresponding storage.

so who ever trigger content-bus, needs to trigger it in all clouds...probably helix-admin or admin.hlx3.page

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants