Alexgallotta/svls 4506 serverless sidecar rebasemain2 #25514

alexgallotta · 2024-05-10T14:59:16Z

No description provided.

* feat(installer): Add installer in path * test remove * fix(installer): Skip symlink if exists

* Warn when GOROOT is not set * Fix rsync command * Use GOPATH instead of GOROOT

…t_verification_stats when the verifier_stats json doesn't exist) we skip the iteration on the NoneObject (#25189)

* update gui path concatenation functions * improve returned HTTP respose in case of error

* Add json annotation to Subnet and Alias * omit empty

…utput (#25185) * - removed verification_time field from the output - simplified a bit the original cleanup stats function in ebpf.collect_verification_stats task * fixed file permissions and truncate any possible leftovers when overriding the file

…#24922) * split tagger component into component and implementation * use an embedded struct to reduce boiler plate

* [CWS] Add 7.54 release notes * Address comments * Address comment * Changed spelling of mmaped

* Use in-memory overlayfs to scan container images * Add missing file * Fix host scan request type conversion * Make it optional * Add support for containerd * Bump trivy to get: - Ignore permission errors when walking in dpkg post analyzer - Handle absolute path in mapfs * Do not populate the metadata id field * Disable overlayfs scanning by default * Keep host.scanRequest private * Set scan output log level to debug * Remove double check of overlayfs support * Add a comment with the Trivy fork branch * Copy img metadata when scanning using overlayfs * Add containerd lease * Bump Trivy to start walking from the root path

… it safe to use GOMAXPROCS in component constructors (#25198)

A recent fix in viper increased the startup time from serverless.

…25194) * Remove noisy log * Update comment

Co-authored-by: pducolin <paola.ducolin@datadoghq.com>

* (fleet) fix remote updates permissions Couple fixes to allow the deamon to write runtime files. * tests * fx fun * fix tests * lol

* Remove usage of cgosymbolizer * Update go.mod go.sum to remove cgosymbolizer dep * Update license

We weren't properly sending the ApplyError to the RC backend.

* remove helper * add timeout

Co-authored-by: jszwedko <jesse.szwedko@datadoghq.com>

* Add logs agent pipeline files * Add modules * Replace logs agent in otel-agent with logs agent pipeline * Fix dependency version * Add health liveness * PR feedback * Move component * Fix otel-agent * Fix lint * Fix imports * Make dependencies public * Fix lint * Edit component interface * Fix fx * Rename method * Update comments * Fix license year * Fix go mods * Fix lint

Co-authored-by: kacper-murzyn <89013263+kacper-murzyn@users.noreply.github.com>

* chore: add e2e tests examples using WithFileWithPermissions for Unix Signed-off-by: Nicolas Guerguadj <nicolas.guerguadj@datadoghq.com> * refacto(e2e): create and use secret helpers Signed-off-by: Nicolas Guerguadj <nicolas.guerguadj@datadoghq.com> * fix: add missing files Signed-off-by: Nicolas Guerguadj <nicolas.guerguadj@datadoghq.com> * feat: add secret client to easily create files containing secrets Signed-off-by: Nicolas Guerguadj <nicolas.guerguadj@datadoghq.com> * chore: go mod tidy Signed-off-by: Nicolas Guerguadj <nicolas.guerguadj@datadoghq.com> * fix: actually use secrets.WithUnixSecretSetupScript Signed-off-by: Nicolas Guerguadj <nicolas.guerguadj@datadoghq.com> * lint: remove unused var Signed-off-by: Nicolas Guerguadj <nicolas.guerguadj@datadoghq.com> * test: add e2e tests for config-refresh on *nix Signed-off-by: Nicolas Guerguadj <nicolas.guerguadj@datadoghq.com> * fix: change testsuite name to run them in CI Signed-off-by: Nicolas Guerguadj <nicolas.guerguadj@datadoghq.com> --------- Signed-off-by: Nicolas Guerguadj <nicolas.guerguadj@datadoghq.com>

* add event id filter with hardcoded config * add eventdata filter on eventids * read filters from file * read confd_path from config comp * fix integration test invocation * add profiles * codeowners * add missing close * avoid formatting the message a second time

* prefix output with pkg/security-agent * package security

…ver.replace` (#25226)

* [usm] Get socket information from all namespaces * [usm] Reduce number of `fstat` calls * [usm] Use /proc/<PID>/net/tcp files instead

* Remove GetGlobalStore() from NewDefaultContainerProvider * Remove unused workloadmeta from containerd * Remove GetGlobalStore from newDockerCollector * Remove GetGlobalStore from kubelet collector.go * Remove GetGlobalStore from docker_util * Remove GetGlobalStore in launcher.go * Remove GetGlobalStore from scanner.go * Remove GetGlobalStore from workloadmeta.go * Remove workload meta global store * Fix linux build * Fix TestDiagnoseCommand * Fix merge conflicts * Fix linux build * Fix compilation on Windows * Fix typo * Fix linter issues * Fix conflicts * Fix when orchestrator is not set * Fix DockerListenerTestSuite * Fix testUDSOriginDetection

pr-commenter · 2024-05-10T19:14:17Z

Test changes on VM

Use this command from test-infra-definitions to manually test this PR changes on a VM:

inv create-vm --pipeline-id=34934213 --os-family=ubuntu

pr-commenter · 2024-05-10T19:58:47Z

Regression Detector

Regression Detector Results

Run ID: f31aed87-4b89-436f-a4f9-28fff820e681
Baseline: 4a91499
Comparison: 19c5f25

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI
➖	file_to_blackhole	% cpu utilization	+0.44	[-0.83, +1.71]

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	+2.32	[-0.49, +5.13]
➖	tcp_syslog_to_blackhole	ingress throughput	+1.85	[-19.71, +23.41]
➖	pycheck_1000_100byte_tags	% cpu utilization	+1.11	[-3.75, +5.97]
➖	file_to_blackhole	% cpu utilization	+0.44	[-0.83, +1.71]
➖	file_tree	memory utilization	+0.05	[-0.04, +0.14]
➖	tcp_dd_logs_filter_exclude	ingress throughput	+0.03	[+0.01, +0.06]
➖	uds_dogstatsd_to_api	ingress throughput	+0.00	[-0.20, +0.20]
➖	trace_agent_msgpack	ingress throughput	+0.00	[-0.00, +0.00]
➖	trace_agent_json	ingress throughput	-0.00	[-0.01, +0.01]
➖	idle	memory utilization	-0.06	[-0.09, -0.02]
➖	otel_to_otel_logs	ingress throughput	-0.08	[-0.43, +0.27]
➖	basic_py_check	% cpu utilization	-0.44	[-2.85, +1.97]

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

BaptisteFoy and others added 30 commits April 26, 2024 16:03

feat(installer): Add installer in path (#25181)

93500bf

* feat(installer): Add installer in path * test remove * fix(installer): Skip symlink if exists

[EBPF] Fix kmt.prepare task on macOS (#25174)

9578dc9

* Warn when GOROOT is not set * Fix rsync command * Use GOPATH instead of GOROOT

in case the function is called with default arguments (like from prin…

1e0b5b9

…t_verification_stats when the verifier_stats json doesn't exist) we skip the iteration on the NoneObject (#25189)

[ASCII-1543] Update GUI path concatenation fonctions (#25087)

f1fe1d0

* update gui path concatenation functions * improve returned HTTP respose in case of error

[network] Add json annotation to Subnet and Alias (#25191)

7184d2c

* Add json annotation to Subnet and Alias * omit empty

[ASCII-1531] Split tagger component into interface and implementation (…

9581bfe

…#24922) * split tagger component into component and implementation * use an embedded struct to reduce boiler plate

[CWS] Add 7.54 release notes (#25188)

34ac000

* [CWS] Add 7.54 release notes * Address comments * Address comment * Changed spelling of mmaped

Invokes 'SetMaxProcs' even earlier in the component lifecycle, making…

b725666

… it safe to use GOMAXPROCS in component constructors (#25198)

Temporary increase serverless threshold while we fix the issue (#25199)

3680d23

A recent fix in viper increased the startup time from serverless.

[process-agent] Remove noisy log about ECS Fargate Misconfiguration (#…

400b0da

…25194) * Remove noisy log * Update comment

Enable new-e2e ndm/netflow test in CI. (#25144)

0746f97

Co-authored-by: pducolin <paola.ducolin@datadoghq.com>

(fleet) fix remote updates permissions (#25170)

4c65fb2

* (fleet) fix remote updates permissions Couple fixes to allow the deamon to write runtime files. * tests * fx fun * fix tests * lol

Remove cgosymbolizer from Agent (#25140)

440b349

* Remove usage of cgosymbolizer * Update go.mod go.sum to remove cgosymbolizer dep * Update license

[RC] Send the ApplyError for configs as well to the RC backend (#24884)

232ed97

We weren't properly sending the ApplyError to the RC backend.

remove helper (#25196)

a6ae268

* remove helper * add timeout

logs/sds: RC / reconfiguration telemetry. (#24722)

c847ad9

Co-authored-by: jszwedko <jesse.szwedko@datadoghq.com>

Skip flaky tests (#25206)

1dd94e2

omnibus cache: fix overzealous updates (#25203)

3d9a706

Co-authored-by: kacper-murzyn <89013263+kacper-murzyn@users.noreply.github.com>

Update release.json and Go modules for 6/7.54.0-rc.1 (#25212)

6235382

[e2e] remove functional agent tests from e2e_pre_test (#25215)

5c51df7

Use default as default namespace (#25232)

5ea4c06

Fix security agent testsuite output path (#25231)

79adffa

* prefix output with pkg/security-agent * package security

normalize otel and trace fields in serverless integration tests (#25204)

392692d

[CWS] add missing config bind for `runtime_security_config.hash_resol…

722187a

…ver.replace` (#25226)

[usm] Get socket information from all namespaces (#25208)

d4d7ba2

* [usm] Get socket information from all namespaces * [usm] Reduce number of `fstat` calls * [usm] Use /proc/<PID>/net/tcp files instead

alexgallotta and others added 23 commits May 10, 2024 11:55

fix test

e3aaf51

fix linter

cd9a9e2

add tags to file log source

5172740

azure change

2fa53ad

"more azure changes"

3304486

SVLS-4796

9876905

fix new arg after rebase

b7f5fb8

use fx for health probe

7eb4194

add back correct tagger

fa5f200

fix serverless build with noop autodiscovery listeners

ecd7b95

remove serverless build tag types

33680f5

removed unused method from conflicts and fix new debug log

e5b35ff

add required params to have health probe working

017968c

make nnop factory private for linter

e1db75b

put noop in a sls build tag file since it is used only in sls

c0bd826

avoid double invocation of setup

15fef65

simplify tags handling

ca53ac0

add mode to version

465210f

use total different tag for init and sidecar

882d51c

fix linter issue

051a85b

xix unit test

6bd28ea

fix rebase issue

9919eab

alexgallotta force-pushed the alexgallotta/SVLS-4506-serverless-sidecar-rebasemain2 branch from 7c40534 to 9919eab Compare May 10, 2024 15:55

fix health probe merge

19c5f25

jcstorms1 and others added 3 commits May 15, 2024 11:05

update AAS recognition

bddbea5

fix double import

fa1171a

SVLS-4847

76eaa73

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Alexgallotta/svls 4506 serverless sidecar rebasemain2 #25514

Alexgallotta/svls 4506 serverless sidecar rebasemain2 #25514

alexgallotta commented May 10, 2024

pr-commenter bot commented May 10, 2024 •

edited

pr-commenter bot commented May 10, 2024

Experiments ignored for regressions

Fine details of change detection per experiment

Explanation

Alexgallotta/svls 4506 serverless sidecar rebasemain2 #25514

Are you sure you want to change the base?

Alexgallotta/svls 4506 serverless sidecar rebasemain2 #25514

Conversation

alexgallotta commented May 10, 2024

pr-commenter bot commented May 10, 2024 • edited

Test changes on VM

pr-commenter bot commented May 10, 2024

Regression Detector

Regression Detector Results

No significant changes in experiment optimization goals

Experiments ignored for regressions

Fine details of change detection per experiment

Explanation

pr-commenter bot commented May 10, 2024 •

edited