Disable shared worker, use a single IPFS instance per app #404

icidasset · 2022-08-10T11:44:58Z

Summary

This PR disables the shared worker which we've had increasingly issues with lately. This should resolve the CBOR decoding issues and CID related issues, such as the ones in fission-codes/auth-lobby#115

~~The result should be slower, but more stable apps.~~ Faster and more stable.

Monitoring functions have been exposed through the webnative.ipfs.node object, example:

webnative.ipfs.node.monitorBitswap( await webnative.ipfs.get() )

Test plan (required)

Try a webnative app with this build. You can make a webnative build with yarn build and then use it in your app by pointing to the local webnative directory with { "webnative": "file:../relative-path-to-webnative/" }

Closing issues

Closes #401

icidasset · 2022-08-10T15:43:34Z

I'm converting all the old browser tests to node tests, because it loads a new js-ipfs node for all these tests now, essentially flooding the production ipfs node with new connections (connects to all peers for each test).

tests/ucan/ucan.node.test.ts

src/ipfs/node.ts

bgins · 2022-08-10T18:20:50Z

Testing this out with the React TodoMVC app, and I'm seeing some bundler issues.

Create React App (`main` branch)

Both the dev server and the build fail with a FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory error.

Full error and stacktrace

<--- Last few GCs --->

[15879:0x140008000]    32103 ms: Mark-sweep 3994.5 (4128.2) -> 3981.3 (4130.9) MB, 2115.0 / 0.0 ms  (average mu = 0.085, current mu = 0.003) allocation failure scavenge might not succeed
[15879:0x140008000]    34234 ms: Mark-sweep 3997.9 (4131.9) -> 3984.7 (4134.3) MB, 2125.2 / 0.0 ms  (average mu = 0.045, current mu = 0.003) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0x104247380 node::Abort() [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
2: 0x104247508 node::errors::TryCatchScope::~TryCatchScope() [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
3: 0x1043938a4 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
4: 0x104393838 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
5: 0x10452fda4 v8::internal::Heap::GarbageCollectionReasonToString(v8::internal::GarbageCollectionReason) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
6: 0x104533314 v8::internal::Heap::CollectSharedGarbage(v8::internal::GarbageCollectionReason) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
7: 0x10453055c v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
8: 0x10452dff8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
9: 0x104539998 v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
10: 0x104539a2c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
11: 0x10450ce6c v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
12: 0x10483f788 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
13: 0x104b4958c Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
14: 0x108e27dfc 
15: 0x108e2db38 
16: 0x104b5aaa8 Builtins_ArrayForEach [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
17: 0x104addcb8 Builtins_InterpreterEntryTrampoline [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
18: 0x104addcb8 Builtins_InterpreterEntryTrampoline [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
19: 0x104addcb8 Builtins_InterpreterEntryTrampoline [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
20: 0x104addcb8 Builtins_InterpreterEntryTrampoline [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
21: 0x104addcb8 Builtins_InterpreterEntryTrampoline [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
22: 0x104b0efac Builtins_GeneratorPrototypeNext [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
23: 0x104addcb8 Builtins_InterpreterEntryTrampoline [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
24: 0x104b0efac Builtins_GeneratorPrototypeNext [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
25: 0x104addcb8 Builtins_InterpreterEntryTrampoline [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
26: 0x104addcb8 Builtins_InterpreterEntryTrampoline [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
27: 0x104addcb8 Builtins_InterpreterEntryTrampoline [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
28: 0x104b8f898 Builtins_PromiseFulfillReactionJob [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
29: 0x104aff734 Builtins_RunMicrotasks [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
30: 0x104adb984 Builtins_JSRunMicrotasksEntry [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
31: 0x1044bd02c v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
32: 0x1044bd460 v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
33: 0x1044bd528 v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*, v8::internal::MaybeHandle<v8::internal::Object>*) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
34: 0x1044e01e8 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
35: 0x1044e0064 v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
36: 0x104196d8c node::InternalCallbackScope::Close() [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
37: 0x10419736c node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
38: 0x1041ac4ac node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
39: 0x10424b600 node::fs::FSReqCallback::Resolve(v8::Local<v8::Value>) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
40: 0x10424c028 node::fs::AfterMkdirp(uv_fs_s*) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
41: 0x104abc0f4 uv__work_done [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
42: 0x104abf854 uv__async_io [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
43: 0x104ad1568 uv__io_poll [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
44: 0x104abfce4 uv_run [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
45: 0x104197e00 node::SpinEventLoop(node::Environment*) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
46: 0x1042806c0 node::NodeMainInstance::Run(int*, node::Environment*) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
47: 0x10428038c node::NodeMainInstance::Run(node::EnvSerializeInfo const*) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
48: 0x10421934c node::Start(int, char**) [/Users/brian/.nvm/versions/node/v16.9.0/bin/node]
49: 0x108a9d08c

Parcel

Module resolution issues. Doesn't appear to be anything new.

Vite

With latest Vite (at 3.0.5), the dev server fails with Big integer literals are not available in the configured target environment ("chrome87", "edge88", "es2020", "firefox78", "safari13" + 2 overrides) errors.

Updating vite.config.ts per vitejs/vite#9062 seems to get past that:

export default defineConfig({
    build: { target: "es2020" },
    optimizeDeps: {
      esbuildOptions: { target: "es2020", supported: { bigint: true } }
    },
  plugins: [reactRefresh()],
})

But then there is a Could not resolve "electron" error.

Full error details

✗ npm start

react-todomvc@0.1.0 start
vite

VITE v3.0.5 ready in 107 ms

➜ Local: http://localhost:5173/
➜ Network: use --host to expose
✘ [ERROR] Could not resolve "electron"

../webnative/node_modules/electron-fetch/lib/index.es.js:1257:21:
  1257 │   electron = require('electron');
       ╵                      ~~~~~~~~~~

You can mark the path "electron" as external to exclude it from the bundle, which will remove this
error. You can also surround this "require" call with a try/catch block to handle this failure at
run-time instead of bundle-time.

/Users/brian/code/fission/react-todomvc/node_modules/esbuild/lib/main.js:1624
let error = new Error(${text}${summary});
^

Error: Build failed with 1 error:
../webnative/node_modules/electron-fetch/lib/index.es.js:1257:21: ERROR: Could not resolve "electron"
at failureErrorWithLog (/Users/brian/code/fission/react-todomvc/node_modules/esbuild/lib/main.js:1624:15)
at /Users/brian/code/fission/react-todomvc/node_modules/esbuild/lib/main.js:1266:28
at runOnEndCallbacks (/Users/brian/code/fission/react-todomvc/node_modules/esbuild/lib/main.js:1046:63)
at buildResponseToResult (/Users/brian/code/fission/react-todomvc/node_modules/esbuild/lib/main.js:1264:7)
at /Users/brian/code/fission/react-todomvc/node_modules/esbuild/lib/main.js:1377:14
at /Users/brian/code/fission/react-todomvc/node_modules/esbuild/lib/main.js:678:9
at handleIncomingPacket (/Users/brian/code/fission/react-todomvc/node_modules/esbuild/lib/main.js:775:9)
at Socket.readFromStdout (/Users/brian/code/fission/react-todomvc/node_modules/esbuild/lib/main.js:644:7)
at Socket.emit (node:events:394:28)
at addChunk (node:internal/streams/readable:315:12) {
errors: [
{
detail: undefined,
id: '',
location: {
column: 21,
file: '../webnative/node_modules/electron-fetch/lib/index.es.js',
length: 10,
line: 1257,
lineText: " electron = require('electron');",
namespace: '',
suggestion: ''
},
notes: [
{
location: null,
text: 'You can mark the path "electron" as external to exclude it from the bundle, which will remove this error. You can also surround this "require" call with a try/catch block to handle this failure at run-time instead of bundle-time.'
}
],
pluginName: '',
text: 'Could not resolve "electron"'
}
],
warnings: []
}

The build succeeds with and without the vite.config.ts changes, but fails at runtime in the browser with

Uncaught ReferenceError: process is not defined
    <anonymous> http://localhost:8080/assets/index.a21b2827.js:144
    <anonymous> http://localhost:8080/assets/index.a21b2827.js:153

Webpack 5

With the latest Webpack 5 libraries, both the dev server and the build fail at runtime in the browser with Uncaught ReferenceError: process is not defined.

Webpack versions include:

webpack at 5.74.0
webpack-cli at 4.10.0
webpack-dev-server at 4.10.0

bgins · 2022-08-15T16:26:09Z

Testing the latest changes after bundling ipfs-core with the React TodoMVC app.

Create React App (`main` branch)

Same FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory error as before for both the dev server and the build.

Parcel

Untested. Will have issues because of lack of support for export maps.

Vite

A new issue with the dev server. At runtime there is a Uncaught (in promise) SyntaxError: missing name after . operator error. More detail from the console:

Also, there are a couple of warnings from the dev server:

[BABEL] Note: The code generator has deoptimised the styling of /Users/brian/code/fission/webnative/lib/vendor/ipfs.js as it exceeds the max of 500KB.
Sourcemap for "/Users/brian/code/fission/webnative/lib/vendor/ipfs.js" points to missing source files

The build has a couple of issues. First, it makes a failing call to gu1.troper.report:

Not sure what that is, seems pretty odd. 🤔

Thinking about it some more, this may have been a peer that was gossiped at some point but was no longer available. The failed call went away after a little while.

The other issue is that it's not possible to make changes to the filesystem. In that app, when I try to add a todo, I get the "Adding to the MMPT again with a different value" warning, but I don't see any logs for adding to the CID log or updating the DNS link messages. Also, nothing seems to be persisted on refreshing the page.

Webpack 5

Webpack works great for the dev server and the build.

It also has that call to gu1.troper.report that fails, but it doesn't seem to cause any issues.

icidasset · 2022-08-16T08:27:07Z

In that app, when I try to add a todo, I get the "Adding to the MMPT again with a different value" warning

I think that's a separate issue no? I've seen that in current production apps too.

A new issue with the dev server.

Not good 😞 Is this with the same Vite config as in oddsdk/odd-bundler-test#2 with optimizeDeps config as well?

icidasset · 2022-08-16T11:42:27Z

Ok, since we still have so many bundler issues, I added the CDN option as well and made that the default. To switch to the bundled ipfs-core, you can do:

wn.ipfs.pkgFromBundle()
  .then(wn.ipfs.nodeWithPkg)
  .then(wn.ipfs.set)

Let me know if you want better names for that.

In addition to that I've added an ES6 build next to the UMD build. The ES6 build is code splitted, whereas the UMD build is not. Meaning the bundled ipfs-core is a separate file with the ES6 build, so that won't be loaded when you use the CDN.

I've also done some testing with the various bundlers from https://github.com/fission-codes/webnative-bundler-test to see what the application code would look like.

Vite:

Code splitting by default
IPFS is excluded when not used (no extra file generated)

Esbuild:

Code splitting with splitting: true and ESM format
IPFS is excluded when not used (no extra file generated)

Webpack:

Code splitting by default, but files are way larger than with other bundlers (2MB + 3.6MB IPFS)
IPFS is not excluded when not used (not that big of a problem, since code splitting by default)

icidasset · 2022-08-16T12:58:08Z

CI tests keep failing because of timeouts, did I mess something up or was this happening before?

bgins · 2022-08-16T16:57:41Z

In that app, when I try to add a todo, I get the "Adding to the MMPT again with a different value" warning

I think that's a separate issue no? I've seen that in current production apps too.

Ah yeah, was just mentioning that for context, but nothing we need to fix here.

A new issue with the dev server.

Not good 😞 Is this with the same Vite config as in fission-codes/webnative-bundler-test#2 with optimizeDeps config as well?

Adding the optimizeDeps section and using the new CDN default seems to have helped, but there this odd behavior where the performance degrades rapidly. An example sequence that I've reproduced a couple of times:

User freshly authenticates with the app and everything works great for a few writes to the filesystem
After a few updates, any further updates get slower to the point of no response or a very delayed response
Refreshing the page in this state, webnative won't even initialize

This sequence of events happens with both the Vite dev server and build. Will try things out with the other bundlers next.

bgins · 2022-08-16T17:30:12Z

Alright, more test results with the latest pulling in ipfs-core from a CDN. Create React app has the same issues as above, and I'm skipping Parcel for now. The Vite testing is in my last comment above this one.

That leaves Webpack 5. The dev server behaves similar to Vite, but slightly different. On initializes, things work great for a while but they slow down and get inconsistent after a little while. Refreshing the page in this state, webnative does initialize, but the filesystem is paralyzed and cannot do anything.

By contrast, the Webpack 5 build is solid. Works great at initialization, keeps up after using it for a few minutes, and works fine after refreshing the page.

bgins

Looks good and working great!

As a summary, I've tested this in the React TodoMVC codebase with Create React App, Vite, and Webpack 5. In all cases, these changes work for both dev server and builds.

I've also tested in the WIP Webnative App Template which handles it own auth without a trip to the lobby. Both the dev server and the build work great there as well!

Excellent work on this! Exciting to have per-app IPFS nodes 🎉

gotjoshua · 2022-08-20T20:38:38Z

scripts/build-minified.js

+    splitting: true,
+    minify: true,
+    sourcemap: true,
+    platform: "browser",


will / should this work in a web worker?

i tried

const webnativeCDNURL = 'https://unpkg.com/webnative@0.34.0-alpha1//dist/index.umd.min.js' const importRes = self.importScripts(webnativeCDNURL)

but i get an ReferenceError: document is not defined

Ah good point, we totally forgot about that. Will fix 👍 Thanks!

gotjoshua · 2022-08-23T06:55:28Z

a failing call to gu1.troper.report:

this is back

is it indeed harmless?

icidasset · 2022-08-23T10:07:23Z

@gotjoshua It's indeed harmless, just IPFS trying to connect to peers. We aren't sure where that's coming from exactly though, we're trying to disable all these random peer connections.

gotjoshua · 2022-08-23T20:13:14Z

Will this strategy block the UI thread while initialize is running or the fs is loading?

i wonder if adding a blobURL to an iframe may be an alternative
https://stackoverflow.com/questions/29667922/set-a-blob-as-the-src-of-an-iframe

icidasset · 2022-08-24T08:36:10Z

@gotjoshua I'm not entirely sure tbh, it doesn't feel like it's blocking it when testing it. We previously used a shared worker for IPFS to optimise this as much as possible out of the box. But, we had a bunch of issues with passing data across postMessage using https://www.npmjs.com/package/ipfs-message-port-protocol So to avoid all those issues we want to keep it simple for now, as we move away from js-ipfs/ipfs-core entirely. We've had our time with iframes, no more please 😅

All that said, you can pass the loadFileSystem: false option to initialize and then call loadFileSystem in a web worker yourself if you don't want to block the main thread.

gotjoshua · 2022-08-24T14:00:17Z

you can pass the loadFileSystem: false option to initialize and then call loadFileSystem in a web worker yourself if you don't want to block the main thread.

exactly what i'm doing. Thanks!

i still feel some ui lag sometimes, which i was concerned may be due to initialize, but it may be due to other factors.

icidasset requested a review from bgins August 10, 2022 11:45

bgins mentioned this pull request Aug 10, 2022

App owned: Add local IPFS implementation #375

Closed

7 tasks

icidasset commented Aug 10, 2022

View reviewed changes

tests/ucan/ucan.node.test.ts Outdated Show resolved Hide resolved

bgins reviewed Aug 10, 2022

View reviewed changes

src/ipfs/node.ts Outdated Show resolved Hide resolved

Disable shared worker and create a faster single-node per app

4f414bd

icidasset force-pushed the icidasset/local-ipfs branch from 53651a2 to 4f414bd Compare August 18, 2022 13:48

bgins approved these changes Aug 18, 2022

View reviewed changes

icidasset added 2 commits August 18, 2022 19:01

Clean up

9c11343

Disable pubsub & re-enable bundled ipfs

bfd1977

icidasset merged commit 8f50d78 into main Aug 19, 2022

icidasset deleted the icidasset/local-ipfs branch August 19, 2022 08:29

gotjoshua reviewed Aug 20, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disable shared worker, use a single IPFS instance per app #404

Disable shared worker, use a single IPFS instance per app #404

icidasset commented Aug 10, 2022 •

edited

icidasset commented Aug 10, 2022

bgins commented Aug 10, 2022

bgins commented Aug 15, 2022 •

edited

icidasset commented Aug 16, 2022

icidasset commented Aug 16, 2022 •

edited

icidasset commented Aug 16, 2022

bgins commented Aug 16, 2022

bgins commented Aug 16, 2022

bgins left a comment

gotjoshua Aug 20, 2022

icidasset Aug 21, 2022

gotjoshua commented Aug 23, 2022

icidasset commented Aug 23, 2022

gotjoshua commented Aug 23, 2022

icidasset commented Aug 24, 2022

gotjoshua commented Aug 24, 2022

Disable shared worker, use a single IPFS instance per app #404

Disable shared worker, use a single IPFS instance per app #404

Conversation

icidasset commented Aug 10, 2022 • edited

Summary

Test plan (required)

Closing issues

icidasset commented Aug 10, 2022

bgins commented Aug 10, 2022

Create React App (main branch)

Parcel

Vite

Webpack 5

bgins commented Aug 15, 2022 • edited

Create React App (main branch)

Parcel

Vite

Webpack 5

icidasset commented Aug 16, 2022

icidasset commented Aug 16, 2022 • edited

icidasset commented Aug 16, 2022

bgins commented Aug 16, 2022

bgins commented Aug 16, 2022

bgins left a comment

Choose a reason for hiding this comment

gotjoshua Aug 20, 2022

Choose a reason for hiding this comment

icidasset Aug 21, 2022

Choose a reason for hiding this comment

gotjoshua commented Aug 23, 2022

icidasset commented Aug 23, 2022

gotjoshua commented Aug 23, 2022

icidasset commented Aug 24, 2022

gotjoshua commented Aug 24, 2022

icidasset commented Aug 10, 2022 •

edited

Create React App (`main` branch)

bgins commented Aug 15, 2022 •

edited

Create React App (`main` branch)

icidasset commented Aug 16, 2022 •

edited