Oversampling mitigation

[atshaw43]

Issue #, if available:

Description of changes:
Oversampling Mitigation and SQS Helper

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov-commenter]

Codecov Report

Merging #354 (fcba4a0) into master (9487f19) will increase coverage by 0.23%.
The diff coverage is 66.07%.

@@             Coverage Diff              @@
##             master     #354      +/-   ##
============================================
+ Coverage     58.30%   58.54%   +0.23%     
- Complexity     1209     1234      +25     
============================================
  Files           131      133       +2     
  Lines          4948     4986      +38     
  Branches        590      595       +5     
============================================
+ Hits           2885     2919      +34     
- Misses         1788     1793       +5     
+ Partials        275      274       -1     

Impacted Files	Coverage Δ
...core/src/main/java/com/amazonaws/xray/AWSXRay.java	59.18% <0.00%> (-1.24%)	⬇️
...va/com/amazonaws/xray/contexts/SegmentContext.java	88.23% <ø> (ø)
...a/com/amazonaws/xray/entities/DummySubsegment.java	0.00% <0.00%> (ø)
...va/com/amazonaws/xray/lambda/SQSMessageHelper.java	33.33% <33.33%> (ø)
.../main/java/com/amazonaws/xray/AWSXRayRecorder.java	63.58% <50.00%> (-0.17%)	⬇️
...onaws/xray/contexts/ThreadLocalSegmentContext.java	74.46% <57.14%> (-3.31%)	⬇️
.../amazonaws/xray/contexts/LambdaSegmentContext.java	76.71% <63.63%> (-1.86%)	⬇️
...n/java/com/amazonaws/xray/entities/Subsegment.java	83.33% <75.00%> (-16.67%)	⬇️
...va/com/amazonaws/xray/entities/SubsegmentImpl.java	71.92% <80.00%> (+2.11%)	⬆️
...va/com/amazonaws/xray/handlers/TracingHandler.java	42.91% <100.00%> (+4.77%)	⬆️
... and 4 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[atshaw43]

Sample code in Lambda

package example;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.SQSEvent;
import com.amazonaws.services.lambda.runtime.events.SQSEvent.SQSMessage;

import com.amazonaws.xray.AWSXRayRecorder;
import com.amazonaws.xray.sqs.SQSMessageHelper;
import com.amazonaws.xray.AWSXRay;

import java.lang.StringBuilder;
import java.util.Map;
import java.util.List;
import java.util.concurrent.CompletableFuture;

import io.opentelemetry.api.trace.Span;

public class Handler implements RequestHandler<SQSEvent, String> {
  public Handler(){
  }

  @Override
  public String handleRequest(SQSEvent event, Context context) {

    // This variable will override the SamplingStrategy to whatever we need.
    boolean toSample = false;

    for (SQSMessage message: event.getRecords()) {
      // This helper function makes it easier to tell if a SQS message is sampled.
      // There are some magic strings involved that we are hiding from the customer.
      if (SQSMessageHelper.isSampled(message)) {
        toSample = true;
      }
    }
    
    // This is the new function that allows the override of the SamplingStrategy.
    AWSXRay.getGlobalRecorder().beginSubsegmentWithSamplingOverride("Batch Trace 1", toSample);
    AWSXRay.getGlobalRecorder().endSubsegment();

    // Testing making two in a single event for good measure.
    AWSXRay.getGlobalRecorder().beginSubsegmentWithSamplingOverride("Batch Trace 2", toSample);
    AWSXRay.getGlobalRecorder().endSubsegment();
  }
}

Test 1 - This is the JSON sent to lambda

{
  "Records": [
    {
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "receiptHandle": "MessageReceiptHandle",
      "body": "Hello from SQS!",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000",
        "SenderId": "123456789012",
        "ApproximateFirstReceiveTimestamp": "1523232000001",
        "AWSTraceHeader": "Root=1-632BB806-bd862e3fe1be46a994272793;Sampled=1"
      },
      "messageAttributes": {},
      "md5OfBody": "{{{md5_of_body}}}",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:MyQueue",
      "awsRegion": "us-east-1"
    }
  ]
}

Test 2 - Note the different Sampled value

{
  "Records": [
    {
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "receiptHandle": "MessageReceiptHandle",
      "body": "Hello from SQS!",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000",
        "SenderId": "123456789012",
        "ApproximateFirstReceiveTimestamp": "1523232000001",
        "AWSTraceHeader": "Root=1-632BB806-bd862e3fe1be46a994272793;Sampled=0"
      },
      "messageAttributes": {},
      "md5OfBody": "{{{md5_of_body}}}",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:MyQueue",
      "awsRegion": "us-east-1"
    }
  ]
}

[wangzlei]

XRayLambdaSQSMessageHelper? it might be a horrible name.

[atshaw43]

I think it makes sense to change the package/module name from SQS to lambda. But let's keep the class name concise. This is what they did in the Lambda library where SQSMessage lives.

[wangzlei]

Module name could be lambda helper since it is help Lambda user, not for sqs instrumentation

[atshaw43]

Done

[wangzlei]

Not quite get why have to use DISABLED.
Can we use existing API, then update the sampled flag in SegmentImpl?

[atshaw43]

If we are changing the parent's flag (the Lambda Frontend Segment), then will that not cause an issue in multithreading?

[wangzlei]

When parentSegment.isRecording() == false and samplingStrategyOverride == SamplingStrategyOverride.TRUE, should we create a new subsegment?

[atshaw43]

Yes. The override to true will ignore any other indicators.
I guess this creates a problem of an orphaned subsegment?