Co-authored-by: Julio Guerra <julio@datadog.com>

Upgrades golang.org/x/net from v.0.3.0 to v.0.7.0.

See:
https://pkg.go.dev/vuln/GO-2022-1144
https://pkg.go.dev/vuln/GO-2023-1571

Reverts #1708
Dependent on another PR that is not ready yet, so reverting changes to avoid release of half-complete features

…kwards compat (#1759)

…1757)

…ollow-up) (#1763)

Change event.json timestamp precision from seconds to nanoseconds.

The current precision limitations were discovered while playing with
prof-tester locally and seeing a profile that should have been 1s
reported as 2s due to rounding effects caused by the limited precision.

There is probably no immediate benefit for improving the precision for
normal 60s+ profiles.

... rather than the end. This way, setting a tracing period matching the
profile period will do the expected thing of sending a trace with every
profile upload, rather than every _other_ profile upload.

… Raw() API call. (#1779)

* Export TracedConn, as we need it when using the db.Conn Raw() API call.

We would prefer to be able to type assert to this interface type, but we cannot type
assert without the type being exported. The alternative would be to use reflection here.

In addition, we want to be able to, on occasion, access the underlying connection object, hence
the addition of the WrappedConn() function.

Signed-off-by: Alec Warner <alec.warner@panther.io>

---------

Signed-off-by: Alec Warner <alec.warner@panther.io>
Co-authored-by: Alec Warner <alec.warner@panther.io>
Co-authored-by: Ahmed Mezghani <38987709+ahmed-mez@users.noreply.github.com>

Co-authored-by: Julio Guerra <julio@datadog.com>

This PR enables the APM end-to-end (E2E) tests in Github Actions CI.

At the moment, the default test is enabled for all variants, but the
Single Spans test is only implemented for the chi variant. All other
variants will still run the jobs, but are skipped (in system-tests terms
they are expectedly failing).

Once we verify that this works well, we can implement the rest variants
too, in follow-up PRs.

We had several incidents with features not working correctly due to
incorrect assumptions, and bugs, in some component of the whole APM
tracing pipeline (tracer, agent, backend).

We want to enable end-to-end tests for key use-cases to avoid regressions
in the future. See ATI-2419 for more details.

The code that retrieves the OS name for Linux has a bug because it tries to
match for Name in the os-release file but the key for the distro name in that
file is actually NAME (in uppercase).

See https://www.freedesktop.org/software/systemd/man/os-release.html

* Update how benchmark results are stored

1. Save converted JSONs to S3 and Gitlab Artifacts.
2. Ensure same data are in S3 and Gitlab Artifacts.

* Explicitly check if baseline is identified for benchmarks

* Fix pr-commenter path

* Fix commit branch and sha collection

* Fix repo name

* Use benchmark analyzer directly

* Add analysis step when no baseline detected

---------

Co-authored-by: Andrew Glaude <andrew.glaude@datadoghq.com>

Adds BenchmarkSingleSpanRetention, BenchmarkInjectW3C to the benchmarking
platform.

Start will first stop the profiler if it is already running, and the
re-start it with the given configuration. This behaviour is not
documented. It is also not tested--the current test suite passes if
Start is changed to return early when the profiler is already active.
It is possible that there are apps out there which rely on this behavior
(some of our internal apps do), and it would be a breaking change for
these apps to make Start do nothing if the profiler is already started.
Given that there isn't a strong reason to change the behavior otherwise,
we should document it and make sure it's actually tested.

This commit adds a test to lock in this behavior. It also removes the
TestStartStopIdempotent test. This test made no assertions, and at any
rate Start was not idempotent to begin with.

If we want to enable a merge queue, we should have our CI also run on PR
that get added to the queue. If CI fails, the PR will be removed from
the queue. Right now we require a PR to be up-to-date with main, and
have green CI, before merging into main. This basically replicates that
requirement for merge queues.

…ot registered via sqltrace.Register (#1635)

* use RecordLogger instead of testLogger in tracer log tests, they are the same thing

* add ignored field to recordlogger to ignore logs with certain substrings

* reset ignored logs on Reset RecordLogger

---------

Co-authored-by: Andrew Glaude <andrew.glaude@datadoghq.com>

The PprofDiff function was added to our public API unintentionally as
part of #1511. The function is only meant for internal use. This was
missed during code review, likely due to the size of the change and the
fact that most of our attention was devoted to making sure the core
parts of the change were implemented correctly.

Mark the function as deprecated, to be removed in a later release.
This should hopefully give users a chance to notice that it's going away
and stop using it before it's removed. pkg.go.dev will show that the
the function is deprecated, and some IDEs and linters will flag use of
deprecated functions.

This is an exception to our normal v1 compatibility guarantees. It is
very unlikely that any of our users depend on this function. It serves
no purpose in the context of the profiler's public API, and anybody who
might have a reason to use this function would probably have already
found a way to do what they want using the
github.com/google/pprof/profile package.

* Add profiler runs on regression

* Add uploading to DD backend

* Remove duplication of code in bash scripts

---------

Co-authored-by: Felix Geisendörfer <felix@datadoghq.com>

Fixes #1153

There was an issue with path to config file introduced in 10e9cce. Jobs with regressions were not printing out PR comments. This commit fixes it.

…ptions (#1159)

Adds an errCheck function to the net/http roundtripper to decide whether a span should be treated as errored or not.

Co-authored-by: Diana Shevchenko <40775148+dianashevchenko@users.noreply.github.com>
Co-authored-by: Zarir Hamza <zarir.hamza@datadoghq.com>

Co-authored-by: Zarir Hamza <zarir.hamza@datadoghq.com>

Adding support to trace the dimfeld/httptreemux/v5 router package. The changes are almost identical to those that were required to instrument julienschmidt/httprouter, since httptreemux is based on httprouter.

- Implement telemetry client that supports the v2 instrumentation telemetry API. 
- Create global telemetry client that both the profiler and tracer can access



* move core telemetry events back to client.go file

- makes code review easier
- decisions about file re-org can happen when more telemetry events are added to telemetry.go

* add option and telemetry fiels

* remove telemetry from profiler

* no need to remove black box testing for now

* no need to use applyops in testconcurrentclient

* fix lint

* omit api key validation

* validate config types

* add telemetry constructors

* Update internal/telemetry/telemetry.go

Co-authored-by: Katie Hockman <katie@hockman.dev>

* Update internal/telemetry/option.go

Co-authored-by: Katie Hockman <katie@hockman.dev>

* Update internal/telemetry/telemetry.go

Co-authored-by: Katie Hockman <katie@hockman.dev>

* metrics should not persist between start/stop

* clarify enable meaning in app-product-change

* fix err handling for agentless retry

* do not start client if there is an api key err

* use filepath for service backup name

* logger can be standalone

* add more status codes to do not retry list

* Update internal/telemetry/message.go

Co-authored-by: Andrew Glaude <andrew.glaude@datadoghq.com>

* Update internal/telemetry/message.go

Co-authored-by: Andrew Glaude <andrew.glaude@datadoghq.com>

* rename applyfallbackops fn

* change comment for fallbackOps

* clarify comment for Request struct

* send profiler productdetails on start as well

* clarify why we are sending profiler product detail on start

* spelling error

* spell

---------

Co-authored-by: Katie Hockman <katie@hockman.dev>
Co-authored-by: Andrew Glaude <andrew.glaude@datadoghq.com>

* fix flaky agentless retry test

* buffer the channel and have the send be non-blocking

* ddtrace/tracer: modify a test case to detect the race condition on span.SetUser.

* ddtrace/tracer: fix a race condition on `trace.SetUser`

---------

Co-authored-by: Andrew Glaude <andrew.glaude@datadoghq.com>
Co-authored-by: lievan <42917263+lievan@users.noreply.github.com>
Co-authored-by: Ahmed Mezghani <38987709+ahmed-mez@users.noreply.github.com>

…d profiler (#1769)

Send instrumentation telemetry data from the tracer and profiler.

Only the tracer can start the instrumentation telemetry client on tracer.Start. If the profiler attempts to send telemetry info before the tracer has started, no info is sent.

Nothing telemetry-related happens when the profiler stops.

* start telemetry client when tracer starts

* enable telemetry by default and refactor transport unit tests

* fixed race condition in transport tests

* add new telemetry function with starter options to telemetry package

* add options file

* find API key when agentless and disable telemetry for clean stop test

* remove telemetry logs in tracer log tests

* call telemetry.NewClient to init profiler telemetry client, turn telemetry on by default for profiler

* change test telemetry enabled

* lint

* disable instrumentation telemetry for transport tests to prevent flakiness

* remove non-valid config values

* change config to single test value

* send empty payload

* add service mappings and global tags to telemetry config

* add product version and enabled details

* add product field to newrequest

* enabled field change to bool

* required field of enabled

* remove profiler from client products info

* remove telemetry client from the profiler for now, keep it for the tracer

* add sampling rules to config data, clarify sampling rule comments

* retry sending telemetry with agentless endpoint given agent errors

* update headers to v2

* update host spec to v2

* remove dependencies from app-started into seperate event

* add support for dependency collection env var

* remove lib fields from metrics

* fix race on temp agentless bool

* add products to app-started and fix errors in configuration schema

* error should not be arr

* lint

* fix mismatched type err'

* send start error data for telemetry

* changed config key values to be consistent with telemetry backend normalization

* del withstarterrors fn

* user tracer log for instrumentation telemetry client:

* fix lint and flaky tests

* implement global telemetry client

* fix race cond on temp agentless var

* refactor telemetry starting logic, reset global telemetry client default values on tracer start

* enable instrumentation telemetry for the profiler

* fix hanging test case

* fix timeout in profiler test, rememebr to stop telemetry

* remove sending error telemetry information for now

* refactor telemetry code in the tracer and profiler

* remove flush timer and just flush during heartbeat interval

* fix lint

* remove the ability for the profiler to start telemetry client

* dont send app-product-change if client hasn't started, update api version in headers

* Update ddtrace/tracer/telemetry.go

Co-authored-by: Andrew Glaude <andrew.glaude@datadoghq.com>

* Update internal/telemetry/client.go

Co-authored-by: Andrew Glaude <andrew.glaude@datadoghq.com>

* Update internal/telemetry/message.go

Co-authored-by: Andrew Glaude <andrew.glaude@datadoghq.com>

* update comments and reformat

* add api key to headers when retrying:

* add agentlessRetry function for retry logic

* switch statement for namespace selection when sending product data

* git rid of black blox testing to speed up the tests, minimize possibility of ci timeout

* fix lint

* move around some functions for clarity

* differentiate namespace for each metric

* Update ddtrace/tracer/telemetry.go

Co-authored-by: Katie Hockman <katie@hockman.dev>

* Update internal/telemetry/client.go

Co-authored-by: Katie Hockman <katie@hockman.dev>

* Update internal/telemetry/client.go

Co-authored-by: Katie Hockman <katie@hockman.dev>

* Update internal/telemetry/client.go

Co-authored-by: Katie Hockman <katie@hockman.dev>

* Update internal/telemetry/option.go

Co-authored-by: Katie Hockman <katie@hockman.dev>

* Update internal/telemetry/client.go

Co-authored-by: Katie Hockman <katie@hockman.dev>

* export Disabled fn for the purpose of checking whether instrumentation telemetry is disabled

* log at info level

* rename submit fn and add comments describing functionality

* add kernel architecture

* dont stop telemetry client when profiler stops

* move all env var config logic into Start

* take away responsibility of disabling the client from With.. functions

* removed unused field

* add an events file and move all events-related code to the file

* rename events file to telemetry

* send produc change on profiler stop

* lint

* ignore telemetry logs

* set api key manually for testing

* no need to check for api key in applydefaultops

* api key validation not our job i think

* readd checking fro agentless for setting api key

* update with latest telemetry changes

* fix lint

* resolve missing telemetry when start is called multiple times without stop

* Update ddtrace/tracer/tracer_test.go

Co-authored-by: Andrew Glaude <andrew.glaude@datadoghq.com>

* add note about telemetry when tracing is disabled

---------

Co-authored-by: Andrew Glaude <andrew.glaude@datadoghq.com>
Co-authored-by: Katie Hockman <katie@hockman.dev>

… found (#1848)

…#1859)

DD_CLIENT_HOSTNAME_ENABLED can be set to false to disable client hostname detection. This may be useful for systems where aggressive security policies throw alerts when applications attempt to access cloud provider APIs