Configure logging

[ojongerius]

Where do our logs go, what do other freeCodeCamp project do? Can and if so, are we happy to re-use this?

/cc @freeCodeCamp/open-api

[raisedadead]

Currently none of the projects have a persistence for logs, sure we do have some tools like opbeat and datadog but, we would want to have our own solution preferably cheap (as in free) like

https://www.elastic.co/solutions/metrics
https://www.elastic.co/solutions/apm
https://www.elastic.co/solutions/logging

We are exploring options for hosting them on our own.

[ojongerius]

Thanks!

Is Datadog giving us a good deal, the lack of data retention over 24 hours in the free tier sounds like a blocker. As Opbeat has been acquired by Elastic thats a dead end.

The Elastic offerings sound interesting. I have seen quite a high operational load on keeping an ELK stack performing at scale, which makes me a little wary of using it in a small team. However
if we get enough AWS credits to run the Elastic products for free it would definitely be an interesting option to explore!

How far are you in exploring running a self hosted ELK? Shall we set up a call this week and catch up?

[raisedadead]

I am planning to re-route the beta instance logs to a small EC2 instance on AWS as a spike with the elastic stack. I am unsure when I will get that ready though. It will need some changes on the beta itself.

But yeah I am up for a call around Thursday if you would like to catch up. Will send you an invite.

[ojongerius]

We've had our meeting and decided to use Cloudwatch for now. What remains to do for this issue is to improve log messages by adding the API GW ID

Lambda see https://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-logging.html
Interesting trick to be able to track which API GW request caused a Lambda log message https://aws.amazon.com/blogs/compute/techniques-and-tools-for-better-serverless-api-logging-with-amazon-api-gateway-and-aws-lambda/
API gw see https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html
Usage: https://docs.aws.amazon.com/lambda/latest/dg/monitoring-functions-logs.html