Offline downloads are slow #4166
Comments
|
We request individual segments, but in some cases, those segments are byte ranges from the same file. We don't have a special case for that, and still download segments one at a time. Options you could explore:
|
Add the possibility to configure the number of downloads in parallel per stream. If you put a higher value the download time can decrease. Related to #4166 Co-authored-by: Joey Parrish <joeyparrish@users.noreply.github.com>
|
I built an automated test that stores a 4k VP9 version of Sintel, 10 times over to get an average. With the latest code in main (including Alvaro's round-robin 5-parallel download improvement), I get: mean storage time: 75.352 seconds Fetching the same file via wget, I get a transfer rate of 373 Mbit/s. Now I'm going to see if there are any cheap modifications to the system I can make to speed that up. I'm going to try to avoid fully reverting @theodab's changes to prepare for background fetch. |
|
Commenting out the call to |
|
Without touching the database, I get: mean storage time: 26.3522 seconds This is a 2.86x improvement in throughput. So I'm going to try to offload the database work as much as possible, and avoid queuing it with the downloads. |
|
Moving database operations to a separate queue (without collapsing them) doesn't help at all:
|
|
I also tried decoupling manifest updates from segments and batching manifest updates together, but I never quite got it right. My implementation resulted in inconsistent test failures. The results for that in the table below are just estimates based on what little data I did get. I then tried deferring all manifest updates until the end (but writing individual segments right away). I think this might be the winner. Results below:
At least for my internet connection, this is 187% of the baseline throughput after @avelad's recent change, and 65% of the full throughput we would achieve if never wrote anything to the database. And I think writing segments to the database is a hard requirement for offline playback. 😁 |
joeyparrish
added
type: performance
|
I'm blown away by this. For me, the nightly demo site is already ~2x faster than the 3.3.4 demo site (~15s vs ~33s, or 49 Mbit/s vs 22 Mbit/s, for the same Sintel download). I'm excited to see the further improvements. Thanks for tackling this so quickly, too. I can understand that downloads may not be the most popular feature so I wasn't expecting to see much movement on this. |
|
Thanks belong to @avelad! He not only put together the PR for the first improvement, but quickly followed up by finding the root cause. I would not have been able to put together my tests or my PR without that help. |
By waiting for all segment data to be written to the database before updating the manifest, we can speed up offline storage in the foreground by ~87%. Closes #4166
If a storage operation is aborted (via API, not via closing the page), this will now be cleaned up from the database. More work is needed to find and remove orphaned segments in the database. Related to shaka-project#4166 and follow-up to PR shaka-project#4176.
By waiting for all segment data to be written to the database before updating the manifest, we can speed up offline storage in the foreground by ~87%. Closes #4166
github-actions
bot
added
the
status: archived
Have you read the FAQ and checked for duplicate open issues?
Yes
What version of Shaka Player are you using?
3.3.2
Can you reproduce the issue with our latest release version?
Yes, on the 3.3.4 demo site
Can you reproduce the issue with the latest code from
main?Yes, on the nightly demo site
Are you using the demo app or your own custom app?
Custom
If custom app, can you reproduce the issue using our demo app?
Yes
What browser and OS are you using?
Chrome 100/Windows 10
For embedded devices (smart TVs, etc.), what model and firmware version are you using?
What are the manifest and license server URIs?
On the demo app, I've been using the Sintel content on the main page.
https://storage.googleapis.com/shaka-demo-assets/sintel-widevine/dash.mpd
What configuration are you using? What is the output of
player.getConfiguration()?Config object
From your demo site's
storage.getConfiguration(), right before theawait storage.store(...):{ "drm": { "retryParameters": { "maxAttempts": 2, "baseDelay": 1000, "backoffFactor": 2, "fuzzFactor": 0.5, "timeout": 30000, "stallTimeout": 5000, "connectionTimeout": 10000 }, "servers": { "com.widevine.alpha": "https://cwip-shaka-proxy.appspot.com/no_auth" }, "clearKeys": {}, "advanced": { "com.widevine.alpha": { "distinctiveIdentifierRequired": false, "persistentStateRequired": false, "videoRobustness": "", "audioRobustness": "", "sessionType": "", "serverCertificate": {}, "serverCertificateUri": "", "individualizationServer": "" }, "com.microsoft.playready": { "distinctiveIdentifierRequired": false, "persistentStateRequired": false, "videoRobustness": "", "audioRobustness": "", "sessionType": "", "serverCertificate": {}, "serverCertificateUri": "", "individualizationServer": "" }, "com.apple.fps": { "distinctiveIdentifierRequired": false, "persistentStateRequired": false, "videoRobustness": "", "audioRobustness": "", "sessionType": "", "serverCertificate": {}, "serverCertificateUri": "", "individualizationServer": "" }, "com.adobe.primetime": { "distinctiveIdentifierRequired": false, "persistentStateRequired": false, "videoRobustness": "", "audioRobustness": "", "sessionType": "", "serverCertificate": {}, "serverCertificateUri": "", "individualizationServer": "" }, "org.w3.clearkey": { "distinctiveIdentifierRequired": false, "persistentStateRequired": false, "videoRobustness": "", "audioRobustness": "", "sessionType": "", "serverCertificate": {}, "serverCertificateUri": "", "individualizationServer": "" } }, "delayLicenseRequestUntilPlayed": false, "logLicenseExchange": false, "updateExpirationTime": 1, "preferredKeySystems": [] }, "manifest": { "retryParameters": { "maxAttempts": 2, "baseDelay": 1000, "backoffFactor": 2, "fuzzFactor": 0.5, "timeout": 30000, "stallTimeout": 5000, "connectionTimeout": 10000 }, "availabilityWindowOverride": null, "disableAudio": false, "disableVideo": false, "disableText": false, "disableThumbnails": false, "defaultPresentationDelay": 0, "dash": { "clockSyncUri": "https://shaka-player-demo.appspot.com/time.txt", "ignoreDrmInfo": false, "disableXlinkProcessing": false, "xlinkFailGracefully": false, "ignoreMinBufferTime": false, "autoCorrectDrift": true, "initialSegmentLimit": 1000, "ignoreSuggestedPresentationDelay": false, "ignoreEmptyAdaptationSet": false, "ignoreMaxSegmentDuration": false, "keySystemsByURI": { "urn:uuid:1077efec-c0b2-4d02-ace3-3c1e52e2fb4b": "org.w3.clearkey", "urn:uuid:edef8ba9-79d6-4ace-a3c8-27dcd51d21ed": "com.widevine.alpha", "urn:uuid:9a04f079-9840-4286-ab92-e65be0885f95": "com.microsoft.playready", "urn:uuid:79f0049a-4098-8642-ab92-e65be0885f95": "com.microsoft.playready", "urn:uuid:f239e769-efa3-4850-9c16-a903c6932efb": "com.adobe.primetime" } }, "hls": { "ignoreTextStreamFailures": false, "ignoreImageStreamFailures": false, "useFullSegmentsForStartTime": false, "defaultAudioCodec": "mp4a.40.2", "defaultVideoCodec": "avc1.42E01E" } }, "streaming": { "retryParameters": { "maxAttempts": 2, "baseDelay": 1000, "backoffFactor": 2, "fuzzFactor": 0.5, "timeout": 30000, "stallTimeout": 5000, "connectionTimeout": 10000 }, "rebufferingGoal": 2, "bufferingGoal": 10, "bufferBehind": 30, "ignoreTextStreamFailures": false, "alwaysStreamText": false, "startAtSegmentBoundary": false, "gapDetectionThreshold": 0.1, "smallGapLimit": 0.5, "jumpLargeGaps": false, "durationBackoff": 1, "forceTransmuxTS": false, "safeSeekOffset": 5, "stallEnabled": true, "stallThreshold": 1, "stallSkip": 0.1, "useNativeHlsOnSafari": true, "inaccurateManifestTolerance": 2, "lowLatencyMode": false, "autoLowLatencyMode": false, "forceHTTPS": false, "preferNativeHls": false, "updateIntervalSeconds": 1, "dispatchAllEmsgBoxes": false, "observeQualityChanges": false }, "offline": { "usePersistentLicense": true }, "abr": { "enabled": true, "useNetworkInformation": true, "defaultBandwidthEstimate": 1000000, "switchInterval": 8, "bandwidthUpgradeTarget": 0.85, "bandwidthDowngradeTarget": 0.95, "restrictions": { "minWidth": 0, "maxWidth": null, "minHeight": 0, "maxHeight": null, "minPixels": 0, "maxPixels": null, "minFrameRate": 0, "maxFrameRate": null, "minBandwidth": 0, "maxBandwidth": null }, "advanced": { "minTotalBytes": 128000, "minBytes": 16000, "fastHalfLife": 2, "slowHalfLife": 5 } }, "preferredAudioLanguage": "en%3Btextlang%3Den%3Builang%3Den%3Bpanel%3DHOME%3Bbuild%3Dcompiled", "preferredTextLanguage": "en", "preferredVariantRole": "", "preferredTextRole": "", "preferredAudioChannelCount": 2, "preferredVideoCodecs": [], "preferredAudioCodecs": [], "preferForcedSubs": false, "preferredDecodingAttributes": [], "restrictions": { "minWidth": 0, "maxWidth": null, "minHeight": 0, "maxHeight": null, "minPixels": 0, "maxPixels": null, "minFrameRate": 0, "maxFrameRate": null, "minBandwidth": 0, "maxBandwidth": null }, "playRangeStart": 0, "playRangeEnd": null, "cmcd": { "enabled": false, "sessionId": "", "contentId": "", "useHeaders": false } }What did you do?
On the Shaka demo site, I opened the Chrome dev tools, cleared the network tab, and clicked the download button for Sintel.
What did you expect to happen?
I expected the transfer to download fairly quickly, ideally using all the available bandwidth. For reference, using
wget, I can download a webm track from Sintel at around 27.25MB/s (or 218 Mbit/s, which is about the limit of my connection).What actually happened?
Once the download via Shaka completed, I took the MB transferred from the Chrome dev tools network page (91.6MB), divided by the last request's start time (33.39 seconds for me), and got 2.74 MB/s (or 21.95Mbit/s) to get a rough estimate. (I also verified that the MPD request was queued at 0ms so I could in theory trust the dev tool timings.) In our custom app, we do calculate an estimated download speed, which is in the same ballpark as Sintel, but for our own content/CDN.
Since Shaka is downloading each byte range as a separate request, I don't expect it to be able to match the throughput of
wget, but I wasn't expecting it to be ~10x slower.We've also tested our own content being served from the same network (1Gb/s theoretical speeds), and we're still only downloading at around 13MB/s (or 104Mbit/s), so again about 10x below theoretical maxes.
It looks like Shaka is downloading each byte range serially, with the only parallelism being the unique files it's downloading (each audio track and video track will happen in parallel). Eventually though, the download will have completed all the audio tracks and low bitrate video tracks leaving one file still downloading, and it'll only be requesting one byte range at a time. Since there's no ability to pause or resume a download in Shaka, this is pretty inconvenient when downloading a large video, such as a movie.
Also, for more context, I went to YouTube and downloaded a video in the web app with YouTube Premium, which seemed to download at around 9MB/s (or 73Mbit/s). Their downloads are serially transferred as well, yet several times faster. YouTube may have larger segments to reduce each request's overhead. I also realize that YouTube is likely not using Shaka Player at all, so it's comparing apples to oranges.
Is this the expected behavior? Is there a bug preventing faster downloads? Would it even be possible to download multiple byte ranges in parallel for the same track/file?
The text was updated successfully, but these errors were encountered: