v2.35.0 proposal #3206

uurien · 2023-06-01T13:25:42Z

Features

waf: Support RC custom rules (implement RC custom rules #3126)
waf: Update blocking page and status from RC (Update blocking page and status from RC or rules file #3195)
iast: Detect SSRF vulnerabilities (Detect SSRF vulnerabilities #3115)
iast: Detect Insecure cookie vulnerabilities (Insecure cookie vulnerability detection #3184)

Improvements

profiling: Use process as default strategy for oom export (Use 'process' as default strategy for oom export #3136)
tracer: Service Naming API (Service Naming API #3161, Introduce service naming framework #2941, Apply service naming flow to messaging integrations #2961)
tracer: Cache integrations - Service Naming (Cache integrations - service naming #3056)
tracer: More beautiful debug logs (More beautiful debug logs #3171)
tracer: postgres: DBM full service fallback w/ prepared statements (postgres: DBM full service fallback w/ prepared statements #3186)
tracer: Make HTTP clients fit in the plugin hierarchy (Make HTTP clients fit in the plugin hierarchy #3178)
ci-visibility: Extract code coverage from cypress ([ci-visibility] Extract code coverage from cypress #3159)
ci-visibility: Change gitlab's pipeline URL extraction ([ci-visibility] Change gitlab's pipeline URL extraction #3183)
ci-visibility: Test skipping logic for cypress ([ci-visibility] Test skipping logic for cypress #3167)
waf: Update AppSec blocking templates (Update AppSec blocking templates #3181)
waf: Update AppSec rules to 1.7.1 (Update Appsec rules to 1.7.1 #3185)
iast: Detect SQL injection with sequelize (Detect SQL injection with sequelize #3154)

Bug Fixes

iast: Fix evidence redaction (Fix IAST evidence redaction #3160)
iast: Fix path traversal vulnerability detection on close file (Fix path traversal vulnerability detection on close #3172)
ci-visibility: Fix cucumber parallel mode ([ci-visibility] Fix cucumber parallel mode #3156)
ci-visibility: Remove git.properties error log (Remove git.properties error log #3179)
ci-visibility: Fix playwright@1.30.0 ([ci-visibility] Fix playwright@1.30.0 #3180)
waf: Fix ASM_DD batch update (Fix ASM_DD batch update #3165)

* Add test for vulnerability evidence scrubber * Fix IAST SQL redaction with tainted contained in sensitive

'process' is the safer export strategy for heap profile export upon oom.

…3166)

* introduce DD_TRACE_SPAN_ATTRIBUTE_SCHEMA env var * add attribute schema v0 for rhea * add attribute schema v1 for rhea * add test harness for service/operation naming * grab config object from pluginManager init * provide schema autoresolution in consumer/producer plugins * bind service to schema manager at configure time * rename plugins to match inbound/outbound service naming terminology * minimize test dependencies * naming resolution wrt version in the test object uses the Nomenclature, instead of being resolved by the test fixture * we no longer use the test fixture for _all_ existing tests, rather let the default resolution do the work * the testing fixture accepts a callback which is a minimal viable trace retrieval, on which we examine _only_ service and name * split test naming schema from test code * Apply service naming flow to messaging integrations (#2961) * add v0 to all messaging plugins * add v1 to all messaging plugins * test naming schema for all messaging integrations * add naming schema tests for other versions * bake messaging data into producer/consumer plugins * persist kind in plugins and infer naming subtype from kind+type * no logs on empty DD_SPAN_TRACE_ATTRIBUTE_SCHEMA * don't compute service name unless necessary

- attempting to fix flaky windows tests

Co-authored-by: Ugaitz Urien <ugaitz.urien@datadoghq.com>

Adopt service naming schema in cache integrations --------- Co-authored-by: Thomas Hunter II <tlhunter@datadog.com>

* implement RC custom rules * update waf bindings to 3.2.0

--------- Co-authored-by: Stephen Belanger <stephen.belanger@datadoghq.com> Co-authored-by: Igor Unanua <igor.unanua@datadoghq.com>

* Taint cookies and headers * Bump minimum node version for v4 on cookie plugin test * Add test with latest node version for cookie plugin test * Provide iastContext from index when tainting headers * Add test for cookie tainting in taint tracking plugin * Remove iast transaction after taint tracking plugin tests to avoid hiting setMaxTransactions in tests * Add test for taintObject with taintingKeys flag * Address header tainting test for keys shorter than 10 chars

This reverts commit cc51732.

- when DBM is set to full service - the DBM comment falls back to service mode - but only when sending a prepared statement - without this, each prepared statement query is technically different - this causes the pg library to fail as it does an exact string check of the query - ideally the pg library would somehow parse out and not consider comments - at any rate this brings parity with other tracer implementations - see brianc/node-postgres#2735 - previously attempted in 0ff9465

* Detect SSRF vulnerabilities * Fix test * Add space * Understand arguments in publisher instead of the subscriber * Redact sensitive information in SSRF vulnerabilities * Tiny style change * Tiny code styles * Use SSRF enum instead of literal * Try to reduce flaky test * Rename originalArgs to originalUrlAndOptions * Fix comment in PR * Do not normalize arguments twice in http/client.js

* Initial version of insecure-cookie vulnerability * Small fix * Add tests * rename const * Fix test * Exclude express file from insecure cookie stack trace * Some code styles + tests * Small code spaces * intermediate-cookies-analyzer to set-cookies-header-intercepteor * Comments in the PR * Comments in PR * Ignore insecure cookie when cookie value is empty * Reuse excluded paths array

* Update blocking page and status from RC or rules file * Use if/else instead of return * Code styles * Split block in two methods * Fix test * Unapply after test * Fix tests * Reorder params in method * Change the signature of updateBlockingConfiguration method * Clear blocking configuration on clear rules * Update blocking response type by configuration * Fix lint

* move http client to clientPlugin * move http2 client to clientPlugin

github-actions · 2023-06-01T13:26:26Z

Overall package size

Self size: 4.23 MB
Deduped: 68.76 MB
No deduping: 68.81 MB

Dependency sizes

name	version	self size	total size
@datadog/pprof	2.2.1	14.24 MB	15.12 MB
@datadog/native-iast-taint-tracking	1.4.1	14.85 MB	14.86 MB
@datadog/native-appsec	3.2.0	13.38 MB	13.39 MB
@datadog/native-metrics	1.6.0	7.88 MB	7.89 MB
protobufjs	7.2.3	2.77 MB	6.65 MB
@types/node	18.11.19	3.58 MB	3.58 MB
@datadog/native-iast-rewriter	2.0.1	2.09 MB	2.1 MB
opentracing	0.14.7	194.81 kB	194.81 kB
lru-cache	7.18.3	133.92 kB	133.92 kB
semver	7.3.8	88.2 kB	118.6 kB
@datadog/sketches-js	2.1.0	109.9 kB	109.9 kB
lodash.sortby	4.7.0	75.76 kB	75.76 kB
ipaddr.js	2.0.1	59.52 kB	59.52 kB
ignore	5.2.4	51.22 kB	51.22 kB
import-in-the-middle	1.3.5	34.34 kB	38.81 kB
istanbul-lib-coverage	3.2.0	29.34 kB	29.34 kB
retry	0.10.1	27.44 kB	27.44 kB
lodash.uniq	4.5.0	25.01 kB	25.01 kB
limiter	1.1.5	23.17 kB	23.17 kB
lodash.kebabcase	4.1.1	17.75 kB	17.75 kB
node-abort-controller	3.1.1	16.89 kB	16.89 kB
lodash.pick	4.4.0	16.33 kB	16.33 kB
crypto-randomuuid	1.0.0	11.18 kB	11.18 kB
diagnostics_channel	1.1.0	7.07 kB	7.07 kB
path-to-regexp	0.1.7	6.78 kB	6.78 kB
koalas	1.0.2	6.47 kB	6.47 kB
methods	1.1.2	5.29 kB	5.29 kB
module-details-from-path	1.0.3	4.47 kB	4.47 kB

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

codecov · 2023-06-01T13:29:27Z

Codecov Report

Merging #3206 (2b5778d) into v2.x (fbc2899) will decrease coverage by 6.40%.
The diff coverage is 73.83%.

@@            Coverage Diff             @@
##             v2.x    #3206      +/-   ##
==========================================
- Coverage   93.51%   87.11%   -6.40%     
==========================================
  Files         222      330     +108     
  Lines        8829    12088    +3259     
  Branches        0       33      +33     
==========================================
+ Hits         8256    10531    +2275     
- Misses        573     1557     +984

Impacted Files	Coverage Δ
integration-tests/ci-visibility/test/sum.js	`69.19% <ø> (ø)`
packages/datadog-instrumentations/src/cucumber.js	`0.00% <0.00%> (-90.77%)`	⬇️
packages/datadog-instrumentations/src/mocha.js	`0.00% <0.00%> (-98.64%)`	⬇️
packages/datadog-instrumentations/src/mongoose.js	`93.33% <ø> (ø)`
packages/datadog-plugin-bunyan/src/index.js	`100.00% <ø> (ø)`
packages/datadog-plugin-connect/src/index.js	`100.00% <ø> (ø)`
packages/datadog-plugin-cucumber/src/index.js	`0.00% <0.00%> (-92.69%)`	⬇️
packages/datadog-plugin-cypress/src/index.js	`100.00% <ø> (ø)`
packages/datadog-plugin-express/src/index.js	`100.00% <ø> (ø)`
packages/datadog-plugin-fastify/src/index.js	`100.00% <ø> (ø)`
... and 71 more

... and 221 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

pr-commenter · 2023-06-01T13:32:58Z

Benchmarks

Comparing candidate commit 2b5778d in PR branch v2.35.0-proposal with baseline commit 000add4 in branch v2.x.

Found 2 performance improvements and 42 performance regressions! Performance is the same for 617 metrics, 47 unstable metrics.

scenario:appsec-control-with-attacks-14

🟥 cpu_user_time [+16.625ms; +22.902ms] or [+6.149%; +8.471%]

scenario:appsec-control-14

🟥 cpu_user_time [+18.181ms; +23.459ms] or [+6.379%; +8.230%]

scenario:log-skip-log-14

🟥 cpu_user_time [+19.663ms; +26.224ms] or [+11.295%; +15.064%]
🟥 execution_time [+22.880ms; +24.510ms] or [+11.510%; +12.330%]
🟥 instructions [+79; +80] or [+16.290%; +16.393%]

scenario:log-with-error-14

🟥 cpu_user_time [+16.550ms; +23.242ms] or [+9.389%; +13.186%]
🟥 execution_time [+22.512ms; +23.664ms] or [+11.308%; +11.887%]
🟥 instructions [+79; +80] or [+16.266%; +16.430%]

scenario:log-with-debug-14

🟥 cpu_user_time [+16.900ms; +23.163ms] or [+9.643%; +13.217%]
🟥 execution_time [+22.188ms; +23.283ms] or [+11.127%; +11.676%]
🟥 instructions [+79; +80] or [+16.325%; +16.394%]

scenario:log-without-log-14

🟥 cpu_user_time [+17.973ms; +24.784ms] or [+10.294%; +14.194%]
🟥 execution_time [+20.134ms; +23.348ms] or [+10.074%; +11.682%]
🟥 instructions [+63; +64] or [+13.076%; +13.136%]

scenario:appsec-appsec-enabled-with-attacks-14

🟥 execution_time [+37.547ms; +46.852ms] or [+9.221%; +11.507%]
🟩 cpu_usage_percentage [-8.363%; -6.473%]

scenario:log-with-debug-16

🟥 cpu_user_time [+29.276ms; +35.947ms] or [+16.651%; +20.445%]
🟥 execution_time [+38.837ms; +39.543ms] or [+19.062%; +19.409%]
🟥 instructions [+84; +85] or [+16.514%; +16.579%]

scenario:log-skip-log-16

🟥 cpu_user_time [+30.113ms; +36.491ms] or [+17.059%; +20.672%]
🟥 execution_time [+39.091ms; +40.051ms] or [+19.179%; +19.651%]
🟥 instructions [+84; +85] or [+16.458%; +16.562%]

scenario:log-with-error-16

🟥 cpu_user_time [+31.028ms; +37.349ms] or [+17.641%; +21.234%]
🟥 execution_time [+38.749ms; +40.607ms] or [+19.016%; +19.928%]
🟥 instructions [+84; +85] or [+16.438%; +16.577%]

scenario:log-without-log-16

🟥 cpu_user_time [+28.863ms; +34.684ms] or [+16.227%; +19.499%]
🟥 execution_time [+35.656ms; +36.549ms] or [+17.398%; +17.834%]
🟥 instructions [+70; +71] or [+13.790%; +13.826%]

scenario:appsec-appsec-enabled-with-attacks-16

🟥 execution_time [+48.414ms; +54.425ms] or [+11.771%; +13.232%]
🟩 cpu_usage_percentage [-8.925%; -7.770%]

scenario:startup-with-tracer-16

🟥 cpu_user_time [+13.695ms; +25.377ms] or [+7.802%; +14.457%]
🟥 execution_time [+22.364ms; +23.297ms] or [+11.109%; +11.573%]

scenario:log-with-debug-18

🟥 cpu_user_time [+16.953ms; +23.209ms] or [+8.768%; +12.003%]
🟥 execution_time [+18.045ms; +20.738ms] or [+8.029%; +9.227%]
🟥 instructions [+70; +71] or [+13.806%; +13.906%]

scenario:log-with-error-18

🟥 cpu_user_time [+15.556ms; +22.537ms] or [+8.037%; +11.643%]
🟥 execution_time [+18.598ms; +19.702ms] or [+8.267%; +8.758%]
🟥 instructions [+70; +71] or [+13.882%; +13.925%]

scenario:log-skip-log-18

🟥 cpu_user_time [+17.557ms; +23.507ms] or [+8.994%; +12.042%]
🟥 execution_time [+18.570ms; +19.789ms] or [+8.234%; +8.774%]
🟥 instructions [+70; +70] or [+13.843%; +13.889%]

scenario:log-without-log-18

🟥 cpu_user_time [+14.063ms; +20.616ms] or [+7.222%; +10.587%]
🟥 execution_time [+16.929ms; +17.955ms] or [+7.502%; +7.956%]
🟥 instructions [+56; +56] or [+11.102%; +11.148%]

juan-fernandez

looks good from ci visibility's perspective. Could we remove this item from the release notes, though, as it isn't interesting for the user at all:

ci-visibility: Use new endpoint for code coverage ([ci-visibility] Use new endpoint for code coverage #3157)

tlhunter and others added 29 commits June 1, 2023 15:11

modifying benchmarks to reduce non-determinism (#3150)

724290f

Fix IAST evidence redaction (#3160)

c2d7379

* Add test for vulnerability evidence scrubber * Fix IAST SQL redaction with tainted contained in sensitive

Use 'process' as default strategy for oom export (#3136)

6b2c5ec

'process' is the safer export strategy for heap profile export upon oom.

Add test for pg native bindings. Pin pg version for v2 appsec test (#…

77b2605

…3166)

test: force a stop before starting metrics (#3173)

c256f0d

- attempting to fix flaky windows tests

Fix path traversal vulnerability detection on close (#3172)

85c4227

More beautiful debug logs (#3171)

d14a7d0

[ci-visibility] Use new endpoint for code coverage (#3157)

74e99cc

[ci-visibility] Fix cucumber parallel mode (#3156)

cb01877

[ci-visibility] Extract code coverage from cypress (#3159)

1b0491e

Co-authored-by: Ugaitz Urien <ugaitz.urien@datadoghq.com>

Cache integrations - service naming (#3056)

a3b1db3

Adopt service naming schema in cache integrations --------- Co-authored-by: Thomas Hunter II <tlhunter@datadog.com>

Fix ASM_DD batch update (#3165)

b00d324

Remove git.properties error log (#3179)

84dd2b9

Implement RC custom rules (#3126)

9b10bd2

* implement RC custom rules * update waf bindings to 3.2.0

[ci-visibility] Fix playwright@1.30.0 (#3180)

38dbe41

Update AppSec blocking templates (#3181)

809623a

Update Appsec rules to 1.7.1 (#3185)

3cf3d66

Detect SQL injection with sequelize (#3154)

979e3f6

--------- Co-authored-by: Stephen Belanger <stephen.belanger@datadoghq.com> Co-authored-by: Igor Unanua <igor.unanua@datadoghq.com>

[ci-visibility] Change gitlab's pipeline URL extraction (#3183)

fa446bd

Add more scenarios to system-tests (#3191)

ee4d7d2

Revert "Taint cookies and headers (#3182)" (#3192)

e0f7af0

This reverts commit cc51732.

[ci-visibility] Test skipping logic for cypress (#3167)

7623ed6

Make HTTP clients fit in the plugin hierarchy (#3178)

62317e1

* move http client to clientPlugin * move http2 client to clientPlugin

uurien added 2 commits June 1, 2023 16:00

Replace rmSync by unlinkSync (#3208)

d9f361d

v2.35.0

2b5778d

uurien marked this pull request as ready for review June 1, 2023 14:47

uurien requested review from a team as code owners June 1, 2023 14:47

tlhunter approved these changes Jun 1, 2023

View reviewed changes

CarlesDD approved these changes Jun 1, 2023

View reviewed changes

uurien changed the title ~~V2.35.0 proposal~~ v2.35.0 proposal Jun 1, 2023

juan-fernandez approved these changes Jun 2, 2023

View reviewed changes

uurien merged commit ef38c1a into v2.x Jun 2, 2023
111 checks passed

uurien deleted the v2.35.0-proposal branch June 2, 2023 07:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v2.35.0 proposal #3206

v2.35.0 proposal #3206

uurien commented Jun 1, 2023 •

edited

github-actions bot commented Jun 1, 2023 •

edited

codecov bot commented Jun 1, 2023 •

edited

pr-commenter bot commented Jun 1, 2023 •

edited

juan-fernandez left a comment

v2.35.0 proposal #3206

v2.35.0 proposal #3206

Conversation

uurien commented Jun 1, 2023 • edited

Features

Improvements

Bug Fixes

github-actions bot commented Jun 1, 2023 • edited

Overall package size

Dependency sizes

codecov bot commented Jun 1, 2023 • edited

Codecov Report

pr-commenter bot commented Jun 1, 2023 • edited

Benchmarks

scenario:appsec-control-with-attacks-14

scenario:appsec-control-14

scenario:log-skip-log-14

scenario:log-with-error-14

scenario:log-with-debug-14

scenario:log-without-log-14

scenario:appsec-appsec-enabled-with-attacks-14

scenario:log-with-debug-16

scenario:log-skip-log-16

scenario:log-with-error-16

scenario:log-without-log-16

scenario:appsec-appsec-enabled-with-attacks-16

scenario:startup-with-tracer-16

scenario:log-with-debug-18

scenario:log-with-error-18

scenario:log-skip-log-18

scenario:log-without-log-18

juan-fernandez left a comment

Choose a reason for hiding this comment

uurien commented Jun 1, 2023 •

edited

github-actions bot commented Jun 1, 2023 •

edited

codecov bot commented Jun 1, 2023 •

edited

pr-commenter bot commented Jun 1, 2023 •

edited