bug: Allow resumable download #255

riderx · 2023-08-29T15:54:30Z

Currently, if a network issue happen or the user kill the app, the download is lost and restarted from 0, so implementing resumable download will improve a lot the bandwidth usage and for user with bad connexion

riderx · 2023-08-29T15:56:34Z

/bounty $200

algora-pbc · 2023-08-29T15:56:37Z

💎 $200 bounty • Capgo

Steps to solve:

Start working: Comment /attempt #255 with your implementation plan
Submit work: Create a pull request including /claim #255 in the PR body to claim the bounty
Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Additional notes:

ℹ️ If something is not clear ask before working on it, otherwise your chance to rework it is high
🎥 To claim you need to provide in your PR a demo video of the change
👨‍👩‍👧‍👦 Join the Discord to get help
📏 Check all Bounty rules

Additional opportunities:

🔴 Livestream on Algora TV while solving this bounty & earn $200 upon merge! Comment /livestream once live

Thank you for contributing to Cap-go/capacitor-updater!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🔴 @neo773	Aug 29, 2023, 3:58:42 PM	WIP
🔴 @ayewo	Sep 20, 2023, 9:32:06 AM	WIP
🔴 @yashk7oo	Oct 19, 2023, 5:04:34 AM	WIP
🔴 @Sambit003	Dec 20, 2023, 1:36:13 PM	WIP
🔴 @Samankhalid01	Feb 6, 2024, 7:23:59 PM	WIP

neo773 · 2023-08-29T15:58:41Z

/attempt #255 continuing this from our DM in discord

Options

Cancel my attempt

WcaleNieWolny · 2023-08-29T19:15:24Z

Isn't the partial download (individual file download) already a partial fix for this?

riderx · 2023-08-29T22:50:35Z

@WcaleNieWolny it seems resumable is easier to do than partial.

So I had conversation with @neo773 that easier to do, so I want to add this now and partial when ready

algora-pbc · 2023-09-05T15:58:51Z

@neo773: Reminder that in 7 days the bounty will become up for grabs, so please submit a pull request before then 🙏

algora-pbc · 2023-09-12T15:58:51Z

The bounty is up for grabs! Everyone is welcome to /attempt #255 🙌

ayewo · 2023-09-20T09:32:04Z

/attempt #255

Options

Cancel my attempt

WcaleNieWolny · 2023-09-20T09:38:56Z

@ayewo as far as I know @neo773 has already wrote some code for this and he is nearly completed this. Although he said that it is 90% complete 5 days ago

ayewo · 2023-09-20T09:45:13Z

I'm sure @neo773 has no trouble speaking for him/herself so there's absolutely no need for you @WcaleNieWolny to speak on their behalf.

WcaleNieWolny · 2023-09-20T09:47:58Z

And I am sure he will. I am not saying you should not attempt this. Go ahead, just letting you know what I know about the status of this issue

neo773 · 2023-09-21T11:09:47Z

@ayewo Feel free to work on it, my attempt wasn't successful
@WcaleNieWolny that "90% completion" was actually partial downloads

ayewo · 2023-09-22T08:32:26Z

@neo773 gracias 👍!

@riderx
I did some preliminary research and I have a few quick questions:

the response headers for requests to the https://api.capgo.app/updates endpoint includes this header:
CF-Cache-Status: DYNAMIC which suggests you have Cloudflare caching enabled, correct?

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
CF-Ray: 7fac66fd7dd30b85-GPS
CF-Cache-Status: DYNAMIC
...
...
Server: cloudflare
Content-Encoding: br

if you have caching enabled, I want to assume you also have caching enabled for updates that are downloaded from the capgo bucket hosted on R2, correct? In other words, downloads from a presigned URL like https://9ee3d7479a3c359681e3fab2c8cb22c0.r2.cloudflarestorage.com/capgo/apps/d94748c9-e609-449e-84d1-30cbfbd4dde5/ee.forgr.demoapp/versions/1.0.0.zip are cached, correct?
if yes to 1.0 and 2.0, then Cloudflare's caching is going to interfere with the ability to resume downloads from the iOS/Android plugin unless caching is turned off for the capgo R2 bucket through cache rules.

For instance, with caching disabled, if a 100MB download is interrupted after 70% has been downloaded, when the download is resumed, the server will send the remaining 30MB. But with caching enabled, the server will always send the full 100MB any time an end user attempts to continue an interrupted download.

Do you still need this feature now that the partial update feature is ready for review?

@WcaleNieWolny it seems resumable is easier to do than partial.

So I had conversation with @neo773 that easier to do, so I want to add this now and partial when ready

I can assure it is not easier 😀. It is closer in complexity to the partial update feature as it will require opening PRs in this repo as well as the Cap-go/capgo repo.

For this repo, the background download logic will have to be refactored so that background downloads can survive network interruptions, end user interruption or termination by the operating system.

For the Cap-go/capgo repo, an additional endpoint will be added to add support for resuming a download from your R2 bucket since you currently use presigned URLs to authorise access.

The payment on this issue completely underestimates the effort: it is currently too low.

neo773 · 2023-09-22T09:22:12Z

@ayewo

For the Cap-go/capgo repo, an additional endpoint will be added to add support for resuming a download from your R2 bucket since you currently use presigned URLs to authorise access.

Pre-signed URLs have content-length header so not sure why you need to an additional API endpoint
I did some initial testing with MZDownloadManager and it did work, but couldn't manage to integrate the library since I'm new to Swift.

WcaleNieWolny · 2023-09-22T09:56:28Z

Can't you just add "Range: bytes=0-NN" header from iOS or android and then just get the r2 file?

I think you can download part of a S3 file https://stackoverflow.com/questions/36436057/s3-how-to-do-a-partial-read-seek-without-downloading-the-complete-file

ayewo · 2023-09-22T10:51:08Z

To support resumable downloads you need 2 headers. The Content-Length: header is one piece. The other piece is the Range: header.

There are 2 reasons why a new endpoint is required:

Capgo currently creates a presigned URL with a validity of 120s (2 min). If a download interruption lasts for more than 2mins, the presigned URL would have expired, so the plugin will need the Capgo server to generate another presigned URL. There is currently no endpoint that handles the renewal of presigned URLs and this is needed for downloads to be resumable.
In order to support arbitrary range seeking using the Range: header, the presigned URL must first encode the actual Range: for a valid URL to be generated. So, if a client wants to fetch Range: bytes=0-19 the Capgo server must encode that extra bit of information before returning a presigned URL. Here's an example using the AWS S3 library in JS: Pre-signed GetObject requests with Range header throws SignatureDoesNotMatch exception aws/aws-sdk-js-v3#4823 (comment).

As for why caching will need to be disabled for downloads from R2, it because we don't want Cloudflare to ignore the Range: header when it is included by the plugin. This Cloudflare forum discusion makes the case for why caching will need to be turned off.

WcaleNieWolny · 2023-09-22T11:28:47Z

Capgo currently creates a presigned URL with a validity of 120s

That is not true, capgo has a way longer validity due to redis caching the url

This used to be true, but is no longer the case

ayewo · 2023-09-25T10:16:00Z

That is not true, capgo has a way longer validity due to redis caching the url

lol 😂🤣!

This was true just 2 weeks ago so not sure what point you are trying to make. In any case your rebuttal doesn't invalidate what I wrote as to why another endpoint is needed.

You could have corrected me by saying Martin has since changed it from 120 to 604800 ... instead you chose to come out guns blazing like someone trying to win an argument (that I didn't start).

Anyway just take it easy as I can't possibly be on top of all the changes you and Martin are making✌️.

WcaleNieWolny · 2023-09-25T10:34:37Z

I was just trying to inform you about the change to the R2 validity time and I was trying to argue against creating a new endpoint

I am sorry if I did not made it clear enough

algora-pbc · 2023-09-27T09:32:10Z

@ayewo: Reminder that in 7 days the bounty will become up for grabs, so please submit a pull request before then 🙏

ayewo · 2023-09-27T10:07:33Z

@algora-pbc yes, I know but without @riderx's input, I can't really forge ahead.
Martin is working on fixing issues with billing (which is of higher priority).

neo773 · 2023-09-27T10:08:46Z

@ayewo Bruh, you’re replying to a bot 💀

ayewo · 2023-09-27T10:44:10Z

@neo773 I know, right 😁!

(I also know that the founders of Algora keep an eye on the bot's email notifications too 😉).

algora-pbc · 2023-09-27T10:46:31Z

Roger that 🫡

algora-pbc · 2023-10-04T09:32:10Z

The bounty is up for grabs! Everyone is welcome to /attempt #255 🙌

yashk7oo · 2023-10-19T05:04:28Z

/attempt #255

Options

Cancel my attempt

algora-pbc · 2023-10-24T05:04:42Z

@yashk7oo: Reminder that in 5 days the bounty will become up for grabs, so please submit a pull request before then 🙏

algora-pbc · 2023-10-29T05:04:41Z

The bounty is up for grabs! Everyone is welcome to /attempt #255 🙌

Sambit003 · 2023-12-20T13:36:11Z

/attempt #255

Options

Cancel my attempt

algora-pbc · 2023-12-25T13:36:20Z

@Sambit003: Reminder that in 5 days the bounty will become up for grabs, so please submit a pull request before then 🙏

algora-pbc · 2023-12-30T13:36:21Z

The bounty is up for grabs! Everyone is welcome to /attempt #255 🙌

Samankhalid01 · 2024-02-06T19:23:57Z

/attempt #255

Options

Cancel my attempt

algora-pbc · 2024-02-11T19:24:09Z

@Samankhalid01: Reminder that in 5 days the bounty will become up for grabs, so please submit a pull request before then 🙏

algora-pbc · 2024-02-16T19:24:05Z

The bounty is up for grabs! Everyone is welcome to /attempt #255 🙌

Sambit003 · 2024-03-02T14:38:01Z

/attempt #255

algora-pbc bot added the 💎 Bounty label Aug 29, 2023

bug: Allow resumable download #255

bug: Allow resumable download #255

Comments

riderx commented Aug 29, 2023

riderx commented Aug 29, 2023

algora-pbc bot commented Aug 29, 2023 • edited

💎 $200 bounty • Capgo

Steps to solve:

Additional notes:

Additional opportunities:

neo773 commented Aug 29, 2023 • edited by algora-pbc bot

WcaleNieWolny commented Aug 29, 2023 • edited

riderx commented Aug 29, 2023

algora-pbc bot commented Sep 5, 2023

algora-pbc bot commented Sep 12, 2023

ayewo commented Sep 20, 2023 • edited by algora-pbc bot

WcaleNieWolny commented Sep 20, 2023

ayewo commented Sep 20, 2023

WcaleNieWolny commented Sep 20, 2023

neo773 commented Sep 21, 2023

ayewo commented Sep 22, 2023

neo773 commented Sep 22, 2023

WcaleNieWolny commented Sep 22, 2023

ayewo commented Sep 22, 2023

WcaleNieWolny commented Sep 22, 2023 • edited

ayewo commented Sep 25, 2023

WcaleNieWolny commented Sep 25, 2023 • edited

algora-pbc bot commented Sep 27, 2023

ayewo commented Sep 27, 2023

neo773 commented Sep 27, 2023

ayewo commented Sep 27, 2023 • edited

algora-pbc commented Sep 27, 2023

algora-pbc bot commented Oct 4, 2023

yashk7oo commented Oct 19, 2023 • edited by algora-pbc bot

algora-pbc bot commented Oct 24, 2023

algora-pbc bot commented Oct 29, 2023

Sambit003 commented Dec 20, 2023 • edited by algora-pbc bot

algora-pbc bot commented Dec 25, 2023

algora-pbc bot commented Dec 30, 2023

Samankhalid01 commented Feb 6, 2024 • edited by algora-pbc bot

algora-pbc bot commented Feb 11, 2024

algora-pbc bot commented Feb 16, 2024

Sambit003 commented Mar 2, 2024

algora-pbc bot commented Aug 29, 2023 •

edited

neo773 commented Aug 29, 2023 •

edited by algora-pbc bot

WcaleNieWolny commented Aug 29, 2023 •

edited

ayewo commented Sep 20, 2023 •

edited by algora-pbc bot

WcaleNieWolny commented Sep 22, 2023 •

edited

WcaleNieWolny commented Sep 25, 2023 •

edited

ayewo commented Sep 27, 2023 •

edited

yashk7oo commented Oct 19, 2023 •

edited by algora-pbc bot

Sambit003 commented Dec 20, 2023 •

edited by algora-pbc bot

Samankhalid01 commented Feb 6, 2024 •

edited by algora-pbc bot