Push files directly to a CDN

[lonasozo]

PROBLEM
Hello Guys, This is an awesome project.
I'm looking for a way to push files directly to a CDN.

EXPECTATION
I imagine a flow like this:

1. Start a live streaming with RTMP
2. While streaming is going on, the server will push (for example) HLS playlist and segments on the CDN provider
3. Fetch with the player directly the contents from the CDN provider

I've seen the documentations but hadn't find something like this, seems there are mainly "Pull" way to do this, but not with the "Push".
I've missed something?

Thanks!

[getroot]

Is there a standard protocol to push to the CDN? Or is it a feature provided by a specific CDN? I have experience working with AWS CloudFront or Akamai, and other CDNs, but I haven't experienced anything with "Push" yet. If you have specific documents or data, please provide them.
If you mean something like S3 upload, that's a feature that OME can't provide. You need to upload it with OME's Recording or HLS Dump and the script or program you developed.

[lonasozo]

Ok clear, yes there are not a standard for the Push, like Akamai gives to you 2 different approaches:
1 Pull -> so basically the CDN will pull the data and spread them
2 Push -> You push data to the CDN and CDN spread them

For the push you can say directly what you you'll push to the CDN, so CDN can do all actions needed to let it works. (For example use caching only on some files, avoid caching on other files.)

Probably yes, what I mean is much similar to the S3 upload.
So basically I should build my own "uploader" that will push the playlist and files on my CDN, there is no option to say "Hey Oven, push files here", correct?

[getroot]

Yes there are too many upload methods in the world. It is difficult to implement them all.

[json-m]

why push over pull to a CDN, out of curiosity? in live streaming scenarios at least, pull is generally the preferred method for lowest latency.

[lonasozo]

I'm happy you asked for that!

Actually I don't have an answer about that, we've build the system in this way so just don't wont to change the logic.
I'm agree with you about the lowest latency, but in my opinion "on push" is a little bit less risky if need to scale up fast.

• No one hit the origin
• Don't need to manage affinity sessions and other things on the Load Balancing for connections

I've never used the "on pull" for live streaming, so maybe I'm wrong 🙌

[lonasozo]

I'm installing Oven on a VM, what it's not clear to me is the path where the file are encoded so I can take them and push to a CDN. Haven't find anything on doc, do you have some hint for me?
Thank you

[getroot]

@lonasozo
What do you want to push LLHLS live to CDN? Let me give you some additional information.

1. All files in LLHLS are in memory.
2. If you use the LLHLS Dump function, you can write a file to the desired directory. But its Playlist is changed for VoD.
3. Adding the ability to dump the live stream as-is to a file is not that difficult. But for reasons [4, 5], it doesn't make any sense.
4. LLHLS must provide "Blocking Playlist Reloads" and "Blocking of Media Downloads" functions as mandatory. However, files pushed to CDN cannot provide this function. Check out Apple's documentation for more details. https://developer.apple.com/documentation/http_live_streaming/enabling_low-latency_http_live_streaming_hls
5. The chunklist of LLHLS must be delivered to the player once within the chunk duration (default: 0.5 seconds). It's not easy to upload this to a CDN once every half a second without fail.

Let me know if you have any other comments.

[lonasozo]

Clear, thank you for your reply and for this clarification.
Sorry I haven't specified, I've seen that legacy HLS is deprecated but still supported, it's enough for me to use legacy HLS at the moment.
After that I'll make some test on LLHLS and test if I'm able to use it in this way or not.

P.s. I've found the DumpSegmentToFile() function, is this the function you've mentioned before?

To be more specific, I need this only for live streaming and not for VoD contents.

[getroot]

I don't have time to support it right now, so if you can do it yourself, here's a tip.

Just remove #EXT-X-ENDLIST from the code below and it will be dumped as a live version of legacy HLS.

https://github.com/AirenSoft/OvenMediaEngine/blob/master/src/projects/publishers/llhls/llhls_chunklist.cpp#L328

[getroot]

As you may already know, the settings for HLS dump are below.
#902

[lonasozo]

Awesome! Thank you, I'll try to do by myself and come back if have any other question 🙏

[radar155]

@lonasozo What do you want to push LLHLS live to CDN? Let me give you some additional information.

1. All files in LLHLS are in memory.
2. If you use the LLHLS Dump function, you can write a file to the desired directory. But its Playlist is changed for VoD.
3. Adding the ability to dump the live stream as-is to a file is not that difficult. But for reasons [4, 5], it doesn't make any sense.
4. LLHLS must provide "Blocking Playlist Reloads" and "Blocking of Media Downloads" functions as mandatory. However, files pushed to CDN cannot provide this function. Check out Apple's documentation for more details. https://developer.apple.com/documentation/http_live_streaming/enabling_low-latency_http_live_streaming_hls
5. The chunklist of LLHLS must be delivered to the player once within the chunk duration (default: 0.5 seconds). It's not easy to upload this to a CDN once every half a second without fail.

Let me know if you have any other comments.

This is pretty clear. But I still have a question: in #766, in your first post you wrote

as you know, it's HTTP-based, so you can deploy with an existing CDN.

So what do you mean with "you can deploy with an existing CDN" ? What I understood was exactly what @lonasozo stated here in his first post. Maybe I'm missing something too?

[getroot]

@radar155
Most HTTP CDNs operate in the HTTP pull method (Reverse Proxy). And what lonasozo wants is to push to CDN.

[radar155]

So basically if OME is setting for us the proper caching headers, it's possible to deliver LLHLS segments (and playlist) across thousands of users with a single OME Origin istance only hitted by the CDN (once per segment) ?

[lonasozo]

@radar155 consider that it is not "one call per segment", it depend on how much fast is the CDN to fetch the content and how many concurrency requests are you getting.

Basic example: if you get 1000 request concurrently, maybe 200 can hit the origin because content is not available into CDN and CDN need to fetch it, 800 get the content directly from CDN. (numbers are totally random, just to let you understand the logic)

[getroot]

In most CDNs I know, when there are 1000 simultaneous requests, only one session goes to the origin, and the remaining 999 requests are hit.

[lonasozo]

On high cost CDNs it could work, if there are optimization for live streaming services.
For mid-low CDNs that does not have live streaming optimization I'm pretty sure it's not the same. (from my experience)

P.s. Time ago (1/2 years) I got a consultancy from one of the "high cost" CDNs, they said to me that with the Pull origin has from 10% to 20% of hit. Maybe something in the while is changed.

[json-m]

in a pull scenario advanced/"high cost" CDNs offer you to designate a specific POP/datacenter to act as an "origin" within their network, which all other POPs will make requests to before your actual origin. utilizing this with request collapsing to origin from the CDN you can achieve high cache HIT ratio (90%+) at the CDN with minimal bandwidth usage to origin.

[getroot]

LLHLS has a sequence number (_HLS_msn, _HLS_part) in the playlist. Therefore, LLHLS has a much higher hit rate than legacy HLS when using a CDN. I'm offering a commercial LLHLS service with CloudFront, and Origin's output throughput is roughly equal to number of edges * media bitrate.

[gaf2419]

@getroot Hi, I'm interested. I'm interested in the service you offer, how can I purchase your services?

[getroot]

@gaf2419 You mean CloudFront? It's not my service, it's AWS's CDN service.

[gaf2419]

@getroot Thanks for answering, I have ovenmediaengine installed, and I also have an amazon account, I would like to configure the amazon cloudfront CDN to deliver the LL-HLS content to me, do you have a tutorial which I can follow step by step? to achieve this? Or do you provide the service of configuring a CDN for ovenmediaengine? I would appreciate any help you can give me.

[getroot]

@gaf2419 #908
Maybe this issue will help you.

And you can find CF tutorials on AWS. I'm just an OME developer. Or does that mean you want to hire me or our company?

[gaf2419]

@getroot I would like to hire you, to configure what I need, for this reason I have previously asked you how I can acquire your services.

[getroot]

@gaf2419 Please email our support team. (support@airensoft.com)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.