Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lambda instrumentation enhancements #1390

Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
dcc01de
added failure scenario when getting container fails
bhautikpip Feb 5, 2021
1dd54d9
fix test case failure
bhautikpip Feb 5, 2021
03f6fb4
add changelog
bhautikpip Feb 5, 2021
5919be5
Merge branch 'main' into main
bhautikpip Feb 5, 2021
ebd45b4
Merge branch 'main' into main
Aneurysm9 Feb 5, 2021
e1ff7d0
fix ecs resource detector bug
bhautikpip Feb 7, 2021
041d9b8
Merge branch 'main' of https://github.com/bhautikpip/opentelemetry-go…
bhautikpip Feb 7, 2021
e12a4b1
fix struct name as per golint suggestion
bhautikpip Feb 7, 2021
e906d2a
fix merge conflict
bhautikpip Feb 7, 2021
86c04d1
minor changes
bhautikpip Feb 7, 2021
047f5d0
added NewResourceDetector func and interface assertions
bhautikpip Feb 9, 2021
21db8fa
fix golint failure
bhautikpip Feb 9, 2021
5204d27
minor changes to address review comments
bhautikpip Feb 10, 2021
c9c1bca
Merge branch 'main' into main
MrAlias Feb 10, 2021
c956d4b
Merge branch 'main' of https://github.com/open-telemetry/opentelemetr…
bhautikpip Feb 10, 2021
12c6c74
Merge branch 'main' of https://github.com/bhautikpip/opentelemetry-go…
bhautikpip Feb 10, 2021
e042a6f
Merge branch 'main' of https://github.com/open-telemetry/opentelemetr…
bhautikpip Jun 3, 2021
1f6b745
Merge branch 'main' of https://github.com/open-telemetry/opentelemetr…
bhautikpip Jun 15, 2021
ada0137
Merge branch 'main' of https://github.com/open-telemetry/opentelemetr…
bhautikpip Aug 12, 2021
9c6a430
Merge branch 'main' of https://github.com/open-telemetry/opentelemetr…
bhautikpip Aug 20, 2021
79d6d15
Merge branch 'main' of https://github.com/open-telemetry/opentelemetr…
bhautikpip Aug 24, 2021
e21f9a6
Merge branch 'main' of https://github.com/open-telemetry/opentelemetr…
bhautikpip Nov 3, 2021
3470b86
added async flusher as an option
bhautikpip Nov 3, 2021
fbcf1ac
modified AllRecommendedConfig API and fix asyncSafeFlusher option
bhautikpip Nov 4, 2021
f03fca4
minor enhancements to improve AllRecommendedOptions further
bhautikpip Nov 4, 2021
4a3ea70
ran makeprecommit for lint
bhautikpip Nov 4, 2021
0a90830
README changes
bhautikpip Nov 4, 2021
b9fede8
README changes: part 2
bhautikpip Nov 5, 2021
38663b0
restore README changes
bhautikpip Nov 5, 2021
9d2d737
restore README changes: 2
bhautikpip Nov 5, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
38 changes: 19 additions & 19 deletions instrumentation/github.com/aws/aws-lambda-go/otellambda/README.md
Expand Up @@ -47,7 +47,7 @@ import "go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go

// wrap lambda handler function
func main() {
lambda.Start(otellambda.InstrumentHandler(HandleRequest))
lambda.Start(otellambda.InstrumentHandler(HandleRequest))
}
```

Expand All @@ -66,14 +66,14 @@ func main() {
var someHeaderKey = "Key" // used by propagator and EventToCarrier function to identify trace header

type mockHTTPRequest struct {
Headers map[string][]string
Body string
Headers map[string][]string
Body string
}

func mockEventToCarrier(eventJSON []byte) propagation.TextMapCarrier{
var request mockHTTPRequest
_ = json.unmarshal(eventJSON, &request)
return propogation.HeaderCarrier{someHeaderKey: []string{request.Headers[someHeaderKey]}}
var request mockHTTPRequest
_ = json.unmarshal(eventJSON, &request)
return propogation.HeaderCarrier{someHeaderKey: []string{request.Headers[someHeaderKey]}}
}

type mockPropagator struct{}
Expand All @@ -82,28 +82,28 @@ type mockPropagator struct{}
// Fields

func HandleRequest(ctx context.Context, request mockHTTPRequest) error {
return fmt.Sprintf("Hello %s!", request.Body ), nil
return fmt.Sprintf("Hello %s!", request.Body ), nil
}

func main() {
exp, _ := stdouttrace.New()
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exp))
lambda.Start(otellambda.InstrumentHandler(HandleRequest,
otellambda.WithTracerProvider(tp),
otellambda.WithFlusher(tp),
otellambda.WithEventToCarrier(mockEventToCarrier),
otellambda.WithPropagator(mockPropagator{})))
exp, _ := stdouttrace.New()

tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exp))

lambda.Start(otellambda.InstrumentHandler(HandleRequest,
otellambda.WithTracerProvider(tp),
otellambda.WithFlusher(tp),
otellambda.WithEventToCarrier(mockEventToCarrier),
otellambda.WithPropagator(mockPropagator{})))
}
```

## Useful links

- For more information on OpenTelemetry, visit: <https://opentelemetry.io/>
- For more about OpenTelemetry Go: <https://github.com/open-telemetry/opentelemetry-go>
- For help or feedback on this project, join us in [GitHub Discussions][discussions-url]
- For help or feedback on this project, join us in [GitHub Discussions][discussions-url]

## License

Expand All @@ -114,4 +114,4 @@ Apache 2.0 - See [LICENSE][license-url] for more information.
[goref-image]: https://pkg.go.dev/badge/go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go/otellambda.svg
[goref-url]: https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go/otellambda
[discussions-url]: https://github.com/open-telemetry/opentelemetry-go/discussions
[lambda-detector-url]: https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/detectors/aws/lambda
[lambda-detector-url]: https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/detectors/aws/lambda
Expand Up @@ -16,6 +16,7 @@ package otellambda

import (
"context"
"runtime"

"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/trace"
Expand Down Expand Up @@ -92,6 +93,20 @@ type config struct {
Propagator propagation.TextMapPropagator
}

type asyncSafeFlusher struct {
flusher Flusher
}

func (f asyncSafeFlusher) ForceFlush(ctx context.Context) error {
// yield processor to attempt to ensure all spans have
// been consumed and are ready to be flushed
// - see https://github.com/open-telemetry/opentelemetry-go/issues/2080
// to be removed upon resolution of above issue
runtime.Gosched()

return f.flusher.ForceFlush(ctx)
}

Comment on lines +96 to +109
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have just landed open-telemetry/opentelemetry-go#2335 which will enable us to remove this entirely and simply call ForceFlush on the provided TracerProvider. The entire Flusher type and its implementations and configuration options can be removed.

func WithTracerProvider(tracerProvider trace.TracerProvider) Option {
return optionFunc(func(c *config) {
c.TracerProvider = tracerProvider
Expand All @@ -115,3 +130,9 @@ func WithPropagator(propagator propagation.TextMapPropagator) Option {
c.Propagator = propagator
})
}

func WithAsyncSafeFlusher(f Flusher) Option {
return optionFunc(func(c *config) {
c.Flusher = asyncSafeFlusher{f}
})
}
Expand Up @@ -34,7 +34,7 @@ func HandleRequest(ctx context.Context, name MyEvent) (string, error) {
}

func main() {
lambda.Start(otellambda.WrapHandlerFunction(HandleRequest))
lambda.Start(otellambda.InstrumentHandler(HandleRequest))
}
```

Expand All @@ -46,7 +46,7 @@ import "go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go

// add options to WrapHandlerFunction call
func main() {
lambda.Start(otellambda.WrapHandlerFunction(HandleRequest, xrayconfig.AllRecommendedOptions()...))
lambda.Start(otellambda.InstrumentHandler(HandleRequest, xrayconfig.AllRecommendedOptions()...))
}
```
## Recommended AWS Lambda Instrumentation Options
Expand All @@ -56,7 +56,7 @@ func main() {
| `WithTracerProvider` | An `sdktrace.TracerProvider` configured to export in batches to an OTel Collector running locally in Lambda | Not individually exported. Can only be used via `AllRecommendedOptions()`
| `WithFlusher` | An `otellambda.Flusher` which yields before calling ForceFlush on the configured `sdktrace.TracerProvider`. Yielding mitigates data delays caused by asynchronous nature of batching TracerProvider when in Lambda | Not individually exported. Can only be used via `AllRecommendedOptions()`
| `WithEventToCarrier` | Function which reads X-Ray TraceID from Lambda environment and inserts it into a `propagtation.TextMapCarrier` | Individually exported as `EventToCarrier()`, also included in `AllRecommendedOptions()`
| `WithPropagator` | An `xray.propagator` | Individually exported as `EventToCarrier()`, also included in `AllRecommendedOptions()`
| `WithPropagator` | An `xray.propagator` | Individually exported as `Propagator()`, also included in `AllRecommendedOptions()`


## Useful links
Expand All @@ -74,4 +74,4 @@ Apache 2.0 - See [LICENSE][license-url] for more information.
[license-image]: https://img.shields.io/badge/license-Apache_2.0-green.svg?style=flat
[goref-image]: https://pkg.go.dev/badge/go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go/otellambda/xrayconfig.svg
[goref-url]: https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go/otellambda/xrayconfig
[discussions-url]: https://github.com/open-telemetry/opentelemetry-go/discussions
[discussions-url]: https://github.com/open-telemetry/opentelemetry-go/discussions
Expand Up @@ -18,7 +18,6 @@ import (
"context"
"log"
"os"
"runtime"

lambdadetector "go.opentelemetry.io/contrib/detectors/aws/lambda"
"go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go/otellambda"
Expand All @@ -35,48 +34,37 @@ func xrayEventToCarrier([]byte) propagation.TextMapCarrier {
return propagation.HeaderCarrier{"X-Amzn-Trace-Id": []string{xrayTraceID}}
}

type asyncSafeFlusher struct {
tp *sdktrace.TracerProvider
}

func (f asyncSafeFlusher) ForceFlush(ctx context.Context) error {
// yield processor to attempt to ensure all spans have
// been consumed and are ready to be flushed
// - see https://github.com/open-telemetry/opentelemetry-go/issues/2080
// to be removed upon resolution of above issue
runtime.Gosched()

return f.tp.ForceFlush(ctx)
}

// tracerProviderAndFlusher returns a list of otellambda.Option(s) to
// enable using a TracerProvider configured for AWS XRay via a collector
// and an otellambda.Flusher to flush this TracerProvider.
// tracerProviderAndFlusher is not exported because it should not be used
// without the provided EventToCarrier function and XRay Propagator
func tracerProviderAndFlusher() ([]otellambda.Option, error) {
ctx := context.Background()

// Do not need transport security in Lambda because collector
// runs locally in Lambda execution environment
// PrepareTracerProvider returns a TracerProvider configured with exporter,
// id generator and lambda resource detector to send trace data to AWS X-Ray via Collector
func PrepareTracerProvider(ctx context.Context) (*sdktrace.TracerProvider, error) {
Comment on lines +37 to +39
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// PrepareTracerProvider returns a TracerProvider configured with exporter,
// id generator and lambda resource detector to send trace data to AWS X-Ray via Collector
func PrepareTracerProvider(ctx context.Context) (*sdktrace.TracerProvider, error) {
// NewTracerProvider returns a TracerProvider configured with exporter,
// id generator and lambda resource detector to send trace data to AWS X-Ray via Collector
func NewTracerProvider(ctx context.Context) (*sdktrace.TracerProvider, error) {

New*() is idiiomatic.

log.Println("creating trace exporter")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
log.Println("creating trace exporter")

exp, err := otlptracegrpc.New(ctx, otlptracegrpc.WithInsecure())
if err != nil {
return []otellambda.Option{}, err
errorLogger.Println("failed to create exporter: ", err)
return nil, err
}

detector := lambdadetector.NewResourceDetector()
res, err := detector.Detect(ctx)
resource, err := detector.Detect(ctx)
if err != nil {
return []otellambda.Option{}, err
errorLogger.Println("failed to detect lambda resources: ", err)
return nil, err
}

tp := sdktrace.NewTracerProvider(
return sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exp),
sdktrace.WithIDGenerator(xray.NewIDGenerator()),
sdktrace.WithResource(res),
)
sdktrace.WithResource(resource),
), nil
}

return []otellambda.Option{otellambda.WithTracerProvider(tp), otellambda.WithFlusher(asyncSafeFlusher{tp: tp})}, nil
// tracerProviderAndFlusher returns a list of otellambda.Option(s) to
// enable using a TracerProvider configured for AWS XRay via a collector
// and an otellambda.Flusher to flush this TracerProvider.
// tracerProviderAndFlusher is not exported because it should not be used
// without the provided EventToCarrier function and XRay Propagator
Comment on lines +64 to +65
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why should these two options not be used without the other two options?

func tracerProviderAndFlusher(tp *sdktrace.TracerProvider) ([]otellambda.Option, error) {
return []otellambda.Option{otellambda.WithTracerProvider(tp), otellambda.WithAsyncSafeFlusher(tp)}, nil
}

// EventToCarrier returns an otellambda.Option to enable
Expand All @@ -89,14 +77,13 @@ func EventToCarrier() otellambda.Option {

// Propagator returns an otellambda.Option to enable the xray.Propagator
func Propagator() otellambda.Option {

return otellambda.WithPropagator(xray.Propagator{})
}

// AllRecommendedOptions returns a list of all otellambda.Option(s)
// recommended for the otellambda package when using AWS XRay
func AllRecommendedOptions() []otellambda.Option {
options, err := tracerProviderAndFlusher()
func AllRecommendedOptions(tp *sdktrace.TracerProvider) []otellambda.Option {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
func AllRecommendedOptions(tp *sdktrace.TracerProvider) []otellambda.Option {
func RecommendedOptions(tp *sdktrace.TracerProvider) []otellambda.Option {

The All feels superfluous. Can we drop it?

options, err := tracerProviderAndFlusher(tp)
if err != nil {
// should we fail to create the TracerProvider, do not alter otellambda's default configuration
errorLogger.Println("failed to create recommended configuration: ", err)
Expand Down
Expand Up @@ -154,6 +154,10 @@ func assertSpanEqualsIgnoreTimeAndSpanID(t *testing.T, expected *v1trace.Resourc
func TestWrapEndToEnd(t *testing.T) {
setEnvVars()

ctx := context.Background()

tp, _ := PrepareTracerProvider(ctx)

customerHandler := func() (string, error) {
return "hello world", nil
}
Expand All @@ -163,7 +167,7 @@ func TestWrapEndToEnd(t *testing.T) {
}()
<-time.After(5 * time.Millisecond)

wrapped := otellambda.InstrumentHandler(customerHandler, AllRecommendedOptions()...)
wrapped := otellambda.InstrumentHandler(customerHandler, AllRecommendedOptions(tp)...)
wrappedCallable := reflect.ValueOf(wrapped)
resp := wrappedCallable.Call([]reflect.Value{reflect.ValueOf(mockContext)})
assert.Len(t, resp, 2)
Expand Down