Datadog trace and stat exporters immediately stop

[colinodell]

Describe the bug

No traces or stats are actually sent to Datadog when the TraceExporter or StatsExporter is set to datadog.

Error Message
N/A

To Reproduce

Run or debug athens with the following environment variables:

ATHENS_TRACE_EXPORTER=datadog
ATHENS_STATS_EXPORTER=datadog

If debugging, also set a breakpoint inside of the ex.Stop function returned by registerDatadogExporter().

Run the application.

Expected behavior

Traces and spans get flushed every few seconds.

Actual behavior

Only one flush occurs at the very end of App(), before the server is even listening.

Environment (please complete the following information):

• OS: [e.g. Linux 64bit] any
• Go version : 1.21.0
• Proxy version : v0.21.1
• Storage (fs/mongodb/s3 etc.) : s3

Additional context

According to the code comments within func App(), both observ.RegisterExporter() and observ.RegisterStatsExporter() are supposed to:

[return] the function that all traces are flushed to the exporter and the exporter needs to be stopped. The function should be called when the exporter is no longer needed.

However, just a few lines lower is a call to defer flushTraces(), causing that function to be invoked almost immediately, which seems like incorrect behavior. For Jaeger and Stackdriver users, that function only calls ex.Flush on the underlying exporter, so nothing negative happens and they'll keep accumulating traces/stats. However, for Datadog, that function actually calls ex.Stop instead, which stops the background loop and thus no new traces/spans will get flushed after App() returns. From the ex.Stop docs:

Stop cleanly stops the exporter, flushing any remaining spans to the transport and reporting any errors. Make sure to always call Stop at the end of your program in order to not lose any tracing data. Only call Stop once per exporter. Repeated calls will cause panic.

(emphasis added)