feat: add AssetManager

[sea-snake]

Description

Currently there's the AssetManager implementation available here.

This PR improves on this implementation and moves it to @dfinity/assets.

The AssetManager has list, store, delete, get, batch and clear methods to manage assets in an asset storage canister.

Fixes #602

How Has This Been Tested?

• AssetManager has e2e tests (Node.js)
• Readable has unit tests

Checklist:

• My changes follow the guidelines in CONTRIBUTING.md.
• The title of this PR complies with Conventional Commits.
• I have edited the CHANGELOG accordingly.
• I have made corresponding changes to the documentation.

[dfinity-droid-prod]

Dear @sea-snake,

In order to potentially merge your code in this open-source repository and therefore proceed with your contribution, we need to have your approval on DFINITY's CLA¹.

If you decide to agree with it, please visit this issue and read the instructions there.

— The DFINITY Foundation

Footnotes

1. Contributor License Agreement ↩

[sea-snake]

PR is work in progress.

Main points that are still TODO:

• Implement a way to track events and/or progress
• Additional unit and e2e tests
• Documentation
• Example application

[sea-snake]

Example application PR: dfinity/examples#355

[krpeacock]

This is close to ready, and the main logic looks solid! To manually run the lint checks, you can run npm run lint --workspace @dfinity/assets

[sea-snake]

This is close to ready, and the main logic looks solid! To manually run the lint checks, you can run npm run lint --workspace @dfinity/assets

Thanks for the helpful feedback! I've resolved these points in the latest commit.

[domwoe]

@sea-snake Seems there is still an issue with the timeout for the E2E tests.

[sea-snake]

@sea-snake Seems there is still an issue with the timeout for the E2E tests.

Ended up reducing the asset size in the tests (e.g. from 1MB to 1KB) to decrease the test duration. On a local environment, the processing speed is more in line with expectation (2s) for each chunk but it seems things take a bit longer on the automated test env.

[domwoe]

Thanks @sea-snake!
@krpeacock Seems to be ready to be merged now.

[krpeacock]

@sea-snake is it concerning that the tests were timing out with a minute to run for a 3mb upload?

[sea-snake]

@krpeacock
I've benchmarked all steps involved. Some observations I've made:

• Sha256 in JS is slow, sadly the current native web implementation doesn't allow incremental hashing so it's not an alternative (You don't want to load a big file in memory if avoidable).
• Every update call takes approx. 2s. For a 3MB asset test there are 5 calls (create_batch, create_chunk, create_chunk, commit_batch and delete_asset). If all these calls were made sequentially, this would be 10s. The create_chunk calls are made in parallel in theory but the dfx environment (local, production, system it's running on) might make a difference there.
• The psuedo random bytes generator in the test isn't the fastest code out there but didn't seem to make a big influence on the overall test duration.
• Maybe the cbor encoding of the update call messages might also affect the speed depending on the system it's running on.

[sea-snake]

@krpeacock
One difference between node and web is that fetch in node is still using http1.1 instead of http2 if I remember correctly. Might be interesting to see if there's a difference in performance when tests are run in cypress.

[krpeacock]

I'll go ahead and merge so we can get this work in front of people, and I'll create an issue to follow up on automating some benchmarks

[sea-snake]

Doing some more debugging now to see if there are some other issues slowing things down.