Finding out why an app crashed

[scripting]

I am adding Forever functionality to my web server, PagePark.

Basically where ever you can put web content, you can put a Node app.

We assign a port and route requests to the domain name you've declared. It works nicely, and I'm already using it in deployed apps.

My question -- how do you get a handle on the error when one of the apps crashes, ie when Forever relaunches it after an error.

I'll provide the code below that I use that I think should catch the error, but it clearly doesn't in at least some circumstances.

I got the error to show up by running the app standalone, and then the OS gave me the error and stack crawl, and I was on my way. I want the same info when Forever is managing the app.

Thanks in advance for any help.

var child = new (foreverMonitor.Monitor) (f, options);
forever.startServer (child); 
child.on ("stdout", function (linetext) { 
	addToLogFile (f, linetext);
	});
child.on ("stderr", function (data) { 
	addToLogFile (f, "pagePark on stderr: " + data.toString ());
	});
child.on ("error", function (err) {
	addToLogFile (f, "pagePark on error: " + utils.trimWhitespace (err.toString ()));
	});
child.on ("exit", function () {
	addToLogFile (f, "pagePark on exit: f == " + f + "\n");
	});
child.start ();

[kibertoad]

Are you sure you want to use forever, which is basically on life support, in non-legacy system? pm2 and nodemon are better choices these days. That said, if you are quite sure, I can take a look.

[tamaker]

@scripting I'd agree with the prior comment about Forever being 'on life support', I still have it in production on a few projects, but have been using PM2 as my production process manager for Node.js. I believe it can provide what you're seeking here. But instead of relying on it solely, I still do my own error handling and std out to console (which pipes to PM2 logging) for trouble shooting. (Love the blog -- keep up the great work!)

[scripting]

Thanks @kibertoad and @tamaker -- not sure what the pragmatic cost of being "on life support" is -- but I have it working, and I don't want to redo all this.

Anyway I want to get this working properly and then get back to my other projects, so if you can help me figure out how to hook up to the system's error reporting mechanism that would be great.

I have a feeling it doesn't have that much to do with Forever, it's just figuring out what to ask the OS for. I assume they're not ripping up the OS too. ;-)

[tamaker]

@scripting - my reference to 'life support' has more to do with just the shallow set of features and low-energy development trajectory (or my perception of these things anyway). Forever is rock solid, I think you're at a fork in the road where your needs are moving beyond it's limitations -- at least that's where I was a few years ago on a big project.

To test this out with your existing code, it's shockingly straight forward with PM2 (I really like that it doesn't have tentacles deeply embedded in my codebase! I don't recall ever having to change my code to accommodate it actually.)

TO TEST YOUR APP WITH PM2 PROCESS MANAGER:
1.) install pm2 using npm: npm install pm2 -g
2.) then STOP your running app and instead run it with pm2 from the command line: pm2 start ./daves-cool-node-app.js --name Cool_App_1 --max-memory-restart 100M (optional)

Once running, here's some commands you'd find useful:

• view status of running processes: pm2 status
• view live monitor of processes: pm2 monit
• stop a process: pm2 stop [ID] | [NAME]
• restart a process: pm2 restart [ID] | [NAME] (picks up newly modified codebase changes)
• view logging of ALL or individual running processes: pm2 log OR pm2 log [ID] (this starts by showing you the default paths to a.) std out and b.) std err log files for each process started by pm2 (this is where, using your scenario / use case / needs, I find it useful to know WHY my app crashed or what PM2 determines to be the reason.

In my experience it has proved useful to:

• run a (self-contained) process / app seriously, in a production environment, and have it be manageable and monitor-able, and self-healing (automated max memory restart, etc.)
• have an app auto start on boot and run as a daemon process or service with PM2 as a wrapper
• have (rotate-able) logging that gives me insight into the 'health' of the app, instead of just a black box of unknowns in terms of how it's running
• catch errors in a global sense, given that my app is running INSIDE the PM2 ecosystem
• bonus: you can also leverage keymetrics.io to have a nice dashboard / monitoring platform for ease of access to your app (without changing your code)

I think you could test this out in 5 minutes, if that -- just install PM2 and run your app from PM2, doesn't work out for you, no harm no foul and just uninstall it with npm via command line.

Keep us posted, good luck! (Apologies for grammar and structure, I keep picturing you marking up my post with red ink!)

[scripting]

@tamaker -- maybe my initial post wasn't clear. I'm not using Forever in an interactive mode. I'm using it as an API within my web server app. You're giving me instructions for using it interactively. I appreciate all the energy you put into this, and you seem to really enjoy PM2, but this is one of those times when a little reading would have saved you a lot of writing. ;-)

[kibertoad]

Note that PM2 has programmatic usage mode as well.
problem with forever is that codebase is a mess, tests are a mess, development is hard, and there is no active development happening, everything is purely fueled by community contributions, hence forever is not really futureproof solution.

[tamaker]

@scripting no worries, @kibertoad is correct, I'm starting to dabble with PM2 in a programmatic sense as well, and have my own express-based API to accept commands -- note: THAT route does indeed involve writing code to integrate it. But in my use case, it's worth the time to give me a full-featured, interface to 'manage' / 'control' my app. At any rate, what you're describing would indeed be a neat extension to forever! Good luck!

[scripting]

I still would like to figure this out. Thanks for respecting that and let's move on please.

[kibertoad]

Part of the implications of what was mentioned before is that there are no active duty maintainers with decent knowledge how forever works. I'll take a look when I have some time, but results are not guaranteed :). PRs to improve current situation are definitely welcome, though.

[imsickofmaps]

Have you considered using an Exception handling service like Sentry? You can self-host the backend fairly painlessly so it's not like you're introducing a black hole of cost.

See: https://docs.sentry.io/platforms/javascript/