Support for Files, Images, Videos to be stored in a cloud provider in addiiton to filesystem

[ravensorb]

It would be great from a scalability perspective to support storing the underling files (videos, images, etc) within a cloud provider (CDN, AWS, Azure, etc). This would allow us to leverage those from a scalability and distribution perspective while still maintaining the metadata within structr.

[eric-schleicher]

This is a really interesting subject and can be implemented within structr. though it's absolutely non-trivial and you would have to learn a lot of the deeper functionality the system offers.

I've dabbled a little bit with using a combination of extension, inheritance, function attributes and entity methods (onCreate/onSave) to manage where user content resides in the system. I use entity methods extensively for organizing projects (a core asset in my system) in the user's content folder for a user, and setting up (automagically) where their generated content goes when the user is created. I haven't made the jump just yet to publishing them on a CDN, which i is something i've been thinking about for some time

I like the idea of doing this since then Structr can be the system of record for the images, etc, and serves directly if required, but can seamlessly support an asynchronous publishing model that as they become published asset are automatically served from the right place, and it's non-destructive & easy to "undo". makes for pretty seamless management and the file can stay in the system (for management) but aren't a drag on serving resources once published on a CDN.

Here is the basic concept/plan i expect to use to accomplish this.

Extend entity "File":

• Add new attribute 'publishableFile'/boolean
  • This will be wired up to onCreate, onSave and onDelete methods to "trigger" publishing of the appropriate types of files
  • It will also cascade over into any entity type that inherits from file (like image, video if you so choose)
• Add new attribute 'publishedUri'/string
  • This is the final hosting location of the asset
• Add new attribute 'bestUri'/function
  • This is the "smart" attribute that picks the best place to serve the assets from
  • Code for the function should evaluate whether the published uri is empty, if so, then use the current url, otherwise return the publishedUri.

the function code for that "bestUri" attribute would look something like:

  ``` if(empty(this.publishedUri), this.path, this.publishedUri) ```

Entity Methods

1. create an onCreate method for the file class
   • In that method, evaluate the see if the type of file is in a list i've defined (most likely in another Entity in the System). If so, either put in in a publishing queue (recommended), or try to publish it immediately (may not work for large file).
2. create an onSave/OnDelete methods to the file class
   • Conditionally, If it's one of these "managed types", run whatever process might be used to request or register a change/sync/delete

####Usage change
Then anywhere the one of these assets is used you use the bestUri attribute (instead of the .path). not anything that inherits or descends from file will have the bestUri and can be intelligently served.

You would need to add the bestUri to the public view of file, and perhaps each of the entities that you wanted to do this with to make sure it's available (if you're using the REST services externally).

This obviously doesn't address the publishing part, but there are a number of ways to achieve this. I haven't settled on what i'm going to do for my system just yet, but i'll post back once i figure it out.

it may be possible to have to use the the inbuild POST function to publish the assets directly, thought i've not used it extensively. Perhaps, @amorgner or @cmorgner can comment on whether the post function would allow you to control the headers/auth that might be required to publish on a CDN.
then throughout the system, the attribute for an asset (video, picture, or other that youve set) can be served from either place, but if it becomes published,

[amorgner]

Great ideas, Eric, this could definitely work like this.

The POST function supports headers, just set them before actual request:

${(
  add_header('X-User', 'abc'),
  POST('http:/external.service/api/foo', '{name:"bar"}', 'application/json', 'utf-8') 
)}

Typically, uploading an asset to a CDN is initiated by a POST request, telling the CDN the location of the original file, and the CDN returns an UUID which then can be used to access the file on the CDN. Here's a nice example: https://uploadcare.com/documentation/upload/

[ravensorb]

I love the ideas and it would be great to see this almost as a provider model concept with the ability to create and configure different providers for different destinations. Then possibly we could have a set of default options for things like AWS, AzureBlob, Samba, and local mountpoint.

[eric-schleicher]

@ravensorb I'm getting closer to going this. I'll post the schema JSON for whatever tracking entities i create.