Fix bug in node json with string more than 500mb

[lexich]

Problem

For big response It can happen error Error: Cannot create a string longer than 0x1fffffe8 characters

Summary of Changes

Add library @discoveryjs/json-ext for parsing response from arrayBuffer with chunks not more the 100mb.
It works right now, but I've made a PR discoveryjs/json-ext#7 for supporting typings for typescript.

Fixes #

[jstarry]

Is it really necessary that web3 supports responses this large?

[lexich]

Is it really necessary that web3 supports responses this large?

Yes we have cases in https://github.com/metaplex-foundation/metaplex

[jstarry]

Is it really necessary that web3 supports responses this large?

Yes we have cases in https://github.com/metaplex-foundation/metaplex

Can you please elaborate? 0.5GB is crazy big

[t-nelson]

Can you please elaborate? 0.5GB is crazy big

This. The default JSON reply buffer is 15MB max (which is also crazy big)

[lexich]

const { Connection } = require('@solana/web3.js');
const METADATA_PROGRAM_ID = 'metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s';
const conn = new Connection('https://api.metaplex.solana.com/', 'recent');
const args = conn._buildArgs([METADATA_PROGRAM_ID], undefined, 'base64');
const resp = conn._rpcRequest('getProgramAccounts', args);
resp.then(e => console.log('OK'), err => console.log(err));
/*
> Error: Cannot create a string longer than 0x1fffffe8 characters
    at Buffer.toString (buffer.js:777:17)
    at /.../node_modules/node-fetch/lib/index.js:284:18
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async ClientBrowser.callServer (/.../node_modules/@solana/web3.js/lib/index.cjs.js:2745:20) {
  code: 'ERR_STRING_TOO_LONG'
*/

[vercel]

@lexich is attempting to deploy a commit to the Solana Team on Vercel.

A member of the Team first needs to authorize it.

[lexich]

@jstarry What do you think about it?

[jstarry]

@jstarry What do you think about it?

I still don't think it's necessary and you haven't provided a reason for why you need this

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[austbot]

Is it really necessary that web3 supports responses this large?

Yes we have cases in https://github.com/metaplex-foundation/metaplex

Can you please elaborate? 0.5GB is crazy big

@lexich @jstarry I know this PR is labeled stale, but I do agree we should actually fix this at the filtering level, return 500mb to web3 clients is perhaps too much. I'm working on adding correct filters to the getProgramAccounts calls in metaplex.

[jstarry]

Is it really necessary that web3 supports responses this large?

Yes we have cases in https://github.com/metaplex-foundation/metaplex

Can you please elaborate? 0.5GB is crazy big

@lexich @jstarry I know this PR is labeled stale, but I do agree we should actually fix this at the filtering level, return 500mb to web3 clients is perhaps too much. I'm working on adding correct filters to the getProgramAccounts calls in metaplex.

Sounds great!