Logging in the v1.11 runtime

[dchenk]

Issue #154 raises the question of structured logging in general, but it needs to be established what is the correct and recommended way to log messages at the various log levels (Critical/Error/Info, etc.) in the second generation standard environment Go 1.11 runtime. Official documentation here lists the google.golang.org/appengine/log package among the packages that "have been superseded by the Go standard library packages listed below"; instead the standard library "log" package is listed.

The sample app now also uses the standard "log" package.

Yet the appengine/log package does not contain any indication that the package is deprecated for the Go 1.11 runtime. Is it, really? If not, is any additional authentication step needed to use the package now within an app?

How must we now log errors distinctly from info or debug log entries?

[camstuart]

I use github.com/op/go-logging and so far messages seem to come out quite nicely in stackdriver:

If you would like an example, let me know

[apstndb]

You can still use google.golang.org/appengine/log in Go 1.11 if you need it.

• Call appengine.Main in func main()
  • https://cloud.google.com/appengine/docs/standard/go111/go-differences#writing_a_main_package
• Use google.golang.org/appengine@'>=v1.3.0'
  • https://cloud.google.com/appengine/docs/standard/go111/go-differences#importing-appengine-sdk
  • google.golang.org/appengine/log isn't compatible with GAE/Go 1.11 #163 is fixed in v1.3.0

[apstndb]

Related to this issue,
google.golang.org/appengine/log emit log for each line to stderr other than request_log.
https://github.com/golang/appengine/blob/v1.3.0/internal/api.go#L582

It seems too noisy and useless without structured logging.
I am currently set exclusion of stderr log to save the billing of Stackdriver Logging.

[dchenk]

@apstndb thank you for that. I was able to figure out to use appengine.Main.

Googlers maintaining this project: I suggest that the docs here, which show this example, are changed to state that the recommended way to run an app in go111 is to call appengine.Main and use google.golang.org/appengine/log instead of the stdlib log package. That is, the docs should specify that this is the right approach if you want the same logging functionality that you had in Go runtime 1st generation.

[sbuss]

I suggest that the docs [...] are changed to state that the recommended way to run an app in go111 is to call appengine.Main and use google.golang.org/appengine/log instead of the stdlib log package

The recommended way is not to use google.golang.org/appengine, but instead use stackdriver logging. github.com/op/go-logging looks promising, too!

[dchenk]

Hi @sbuss
Are you speaking for Google there? Or is that something you were told or read somewhere? Honest question.

[apstndb]

IMO, the "Migrating your App Engine app from Go 1.9 to Go 1.11" docs seems mainly for migrating to more open APIs rather than for migrating to Go 1.11 runtime.
It is mismatched with the document title and confusing.

I think that many users want to use Go 1.11 and want to get benefits of 2nd gen runtime as soon as possible but want to stay to use App Engine API unless their migration plan is ready.
It is making sense to separate the document, for migrating to 1.11 from 1st gen and for migrating to more open APIs from App Engine APIs.

[hazcod]

Has anything been done for documenting this?

[oyvindsk]

Anything happening on the logging issue? Not having log messages grouped by request makes the logging almost useless and is a huge regression from AE standard 1st gen runtimes. (Along with many, many others)

It works in 1.11 since you can still use google.golang.org/appengine/log but, according to the docs, it will not work in the future :'(

[mtraver]

You can use the Stackdriver Logging API to group logs by request. Here's an example of how to do so: https://github.com/mtraver/gaelog

[iamacarpet]

Alternatively, instead of writing the logs to the Stackdriver API synchronously which can be slow, App Engine allows you to write out to virtual "files" that automatically get passed to Stackdriver in the background.

It is less well written than the one from @mtraver I'm afraid, but feel free to try this one: https://github.com/a1comms/go-gaelog

[evmar]

I just wanted to comment on this bug to ensure it isn't lost. To summarize, there appear to be three different 'official' logging packages (Go, appengine, stackdriver) and no clear recommendation between them.

[derat]

This is an old bug, but just to save others some trouble: I think that structured logging (almost) works using the google.golang.org/appengine/v2/log package now, as long as you're using the standard environment. It looks like there was a typo in a header name that was fixed by #263, though, so you may need to run go get google.golang.org/appengine/v2@e0054249252f2a095b5f75c417b41d7cef9cbd10 instead of using the v2.0.1 release. Log entries are grouped by request for me with the go115 runtime.