Status update: Puppeteer "ERR_CONNECTION_REFUSED"

[rexagod]

matcher-cli > Error: net::ERR_CONNECTION_REFUSED at http://127.0.0.1:9996/demo/
    at navigate (/mnt/c/Users/Rexagod/cli/node_modules/puppeteer/lib/FrameManager.js:121:37)          at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:189:7)
  -- ASYNC --
    at Frame.<anonymous> (/mnt/c/Users/Rexagod/cli/node_modules/puppeteer/lib/helper.js:111:15)       at Page.goto (/mnt/c/Users/Rexagod/cli/node_modules/puppeteer/lib/Page.js:674:49)
    at Page.<anonymous> (/mnt/c/Users/Rexagod/cli/node_modules/puppeteer/lib/helper.js:112:23)        at invoke (/mnt/c/Users/Rexagod/cli/src/matcher-summon.js:53:14)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:189:7)

matcher-cli uses a puppeteer env to serve results through node. This error (in puppeteer) would mean that the cli will not be able to fetch results when used in a node environment (this also goes for all matcher integrations that "require" it instead of including the min files in the <script> tags. I've been trying to fix this and/or find a workaround to this issue for the past few days and nothing seems to resolve this issue, so I guess the best thing to do right now is open an issue there and see if we can fix this quickly.

Will update here.

---

@jywarren To sum it all up, the library currently integrates and works fine when the minified version is included in the head. But due to the aforementioned puppeteer issue, I'm unable to deploy the core in a virtual env and thus, I'm unable to collect any results off of the virtual webpage. Linking to the corresponding issue @ puppeteer in a bit.

[rexagod]

I think it's worth mentioning that over the past two days I've checked out every single branch of the cli just to verify if there's something up in my code and I'm almost certain there's not, since I remember these working okay a while back. Maybe this is rooted in https://github.com/GoogleChrome/puppeteer/releases/tag/v1.19.0 (23rd July)?

[rexagod]

Posted: puppeteer/puppeteer#4776

[rexagod]

Status update: Came across some promising workarounds today, and have been trying to convert to node env directly. Will keep this thread updated.

[rexagod]

Still debugging, I'm basically refactoring the whole library for this "browser-env" to work and resolve all of the errors popping out of it. Exhausted.

[rexagod]

Also waiting for the folks at puppeteer to respond to this issue. No comments yet. 😕

[rexagod]

Just finished up on this, no errors now, but I guess for some reason node is going into en endless "await" state.

[jywarren]

Aww this looks very frustrating!! I'm sorry!

Could it have anything to do with the chrome chromedriver problem were working on in plots2? With the bump to chrome 76 now that 74 is no longer supported? publiclab/plots2#6095

[rexagod]

@jywarren Running Chrome instance with the modified options in the PR above, i.e., changing this,

https://github.com/publiclab/matcher-cli/blob/c001e7f5644dc2f75078c73f83f3cfe89db8649d/src/matcher-summon.js#L30

to this,

https://github.com/publiclab/plots2/blob/2f1a29e930d8ed7c63ba593827c234af9bc73f04/test/application_system_test_case.rb#L4

Will keep this thread updated.

[rexagod]

(node:1546) UnhandledPromiseRejectionWarning: Error: Page crashed!
    at Page._onTargetCrashed (/mnt/c/Users/Rexagod/cli_/node_modules/puppeteer/lib/Page.js:215:24)
    at CDPSession.Page.client.on.event (/mnt/c/Users/Rexagod/cli_/node_modules/puppeteer/lib/Page.js:123:56)    at emitOne (events.js:116:13)
    at CDPSession.emit (events.js:211:7)    at CDPSession._onMessage (/mnt/c/Users/Rexagod/cli_/node_modules/puppeteer/lib/Connection.js:200:12)    at Connection._onMessage (/mnt/c/Users/Rexagod/cli_/node_modules/puppeteer/lib/Connection.js:112:17)    at WebSocketTransport._ws.addEventListener.event (/mnt/c/Users/Rexagod/cli_/node_modules/puppeteer/lib/WebSocketTransport.js:44:24)    at WebSocket.onMessage (/mnt/c/Users/Rexagod/cli_/node_modules/ws/lib/event-target.js:120:16)    at emitOne (events.js:116:13)
    at WebSocket.emit (events.js:211:7)
(node:1546) UnhandledPromiseRejectionWarning: Unhandled promise rejection. 
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:1546) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

[rexagod]

Sorry, forgot to mention here. Have been working on a NON-puppeteer implementation of this, which might render most of cli library redundant, but enabling us to play around in node w/o any page crashes, mem leaks, connection errors, et cetera.

Will continue to update here.

[rexagod]

The best thing being one library for both browser and node envionments, obviously. Just thought I'd point this out.

Working.

[rexagod]

Focusing on:

• netcat (debugging)
• tcp (final implementation for both client and server through node)
• sockets (through node, included in tcp)

These should be more that enough to allow me to send and recieve packets over a port.

Let's see.

[rexagod]

@jywarren Axios, native fetch, or any HTTP/2 services won't be of any use since the results are being uploaded and fetched back from a localhost port on the user's machine! That's why I was getting the connection refused error! After a bit of thought on how to send and recieve data over the same port, it clicked me that socket programming was made for this! Below is a simple node implementation for the same that (if everything goes well 🤞 ) I'll modify heavily and incorporate into this library!

[jywarren]

Hi, should this really be in the matcher-cli repo, as your code in matcher-core shouldn't really need much awareness of whether it's being run in node or browser? I assume the sockets work you're doing is in matcher-cli, not here, but it's hard to follow with lots of commits going to the main branch here without passing through pull requests. I think it's probably best to work on a feature branch if you're doing major refactoring, until you've resolved the issue.

For reference, see how image-sequencer' loads images and runs tests in node: https://github.com/publiclab/image-sequencer/blob/main/test/core/sequencer/image-sequencer.js#L30 (there is also a browser-based suite of tests for Image Sequencer)

Sockets are interesting, but what exactly are you trying to achieve here? The basic matcher-core code should be executable in a pure node environment, shouldn't it?

[jywarren]

Is the issue the local loading of images in node? Perhaps image-sequencer is a good guide to doing this simply?

[rexagod]

@jywarren The recent commits were to update the main with PRs you had already reviewed, and solve a few merge conflicts here and there,, since a while back, you suggested to pass commits through PRs, and I've been following that ever since. I also did a sweep and removed all the unnecessary branches from this library. There have been no new pushes since, and if there were, I would definitely submit those through PRs.

The basic matcher-core code should be executable in a pure node environment, shouldn't it?

The heart of this algorithm uses canvas.getdata (along with window, requestAnimationFrame, and other browser-specific apis) to operate on the image data in a 'browser' environment, therefore I had no problem integrating this with LDI (a browser app). But for node purposes I used puppeteer (in cli) to collect that data off of a webpage until the error above forced me to try an alternative approach.

[rexagod]

matcher-core/src/orb.core.js

Line 88 in 6460948

imgData = ctxx.getImageData(

matcher-core/src/orb.core.js

Line 210 in 6460948

window.requestAnimationFrame(findPoints);

matcher-core/src/orb.core.js

Line 284 in 6460948

window.onload = function() {

[rexagod]

Just leaving this here: puppeteer/puppeteer#1401 (comment)

[jywarren]

Ah ok that makes sense, thanks! Yeah, actually that's one reason we didn't base image-sequencer on canvas, although that would've been easiest! In this case, I think the puppeteer or headless browser approach makes sense. But it's probably worth looking at what parts of the system absolutely require canvas sometime in the future. Not urgent at all but there may be possibilities for pure js, non canvas ways to do this that'd make node usage easier. Thanks for the clarification! As to the PRs, just checking, as you'd mentioned refactoring here and I saw a commit that seemed to reference refactoring in the past couple days. But glad to hear it, and thanks for your working with this flow, I appreciate it! I hope the puppeteer approach or another similar one works out... headless browsers seem to be a bit less stable than I'd hoped based on the PR in plots2 I linked to, but they sure do allow for some impressive cross environment possibilities. Don't worry, Pranshu I'm sure you'll crack this issue!

…

On Sat, Aug 3, 2019, 1:26 PM Pranshu Srivastava ***@***.***> wrote: Just leaving this here: puppeteer/puppeteer#1401 (comment) <puppeteer/puppeteer#1401 (comment)> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9?email_source=notifications&email_token=AAAF6JYBC4DJUVVWQ5ZRPCDQCVMK5A5CNFSM4IIADNYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3PL4XA#issuecomment-517914204>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAF6J4ULNXVQEATZFWRS73QCVMK5ANCNFSM4IIADNYA> .

[rexagod]

@jywarren I'M SO CLOSE! It's as if I just can't avoid incorporating puppeteer into this lib, or maybe I can do the same with phantomJS? Anyway, as you can see below things are working pretty good, with the side effect that chrome has to launch atleast once on the user's machine to instantiate a browser instance to parse all the frontend apis. This is the last stage, if I can just get a headless browser to sent POST requests to my server's port without actually opening a chrome instance, that'd be just pure joy for me atm.

But it's probably worth looking at what parts of the system absolutely
require canvas sometime in the future. Not urgent at all but there may be
possibilities for pure js, non canvas ways to do this that'd make node
usage easier.

I've tried to eliminate this headless deployment not now but several times in the past as well, and what I've learnt through all of those unseccessful attempts is the fact that some frontend apis are just not meant to be parsed by node. From creating a so-called browser env inside node to spending days experimenting with the npm's "window" package for node to trying to implement individual features in node, I have tried almost everything, which takes me back to the headless implementation I did in the first place. Not so while back I used to make several PRs in the same day in LDI, but now I'm just spending several days on the same PR.

[rexagod]

puppeteer/puppeteer#494 (comment)