v4.1.0 proposal #3209

uurien · 2023-06-01T14:22:11Z

Features

waf: Support RC custom rules (implement RC custom rules #3126)
waf: Update blocking page and status from RC (Update blocking page and status from RC or rules file #3195)
iast: Detect SSRF vulnerabilities (Detect SSRF vulnerabilities #3115)
iast: Detect Insecure cookie vulnerabilities (Insecure cookie vulnerability detection #3184)

Improvements

profiling: Use process as default strategy for oom export (Use 'process' as default strategy for oom export #3136)
tracer: Service Naming API (Service Naming API #3161, Introduce service naming framework #2941, Apply service naming flow to messaging integrations #2961)
tracer: Cache integrations - Service Naming (Cache integrations - service naming #3056)
tracer: More beautiful debug logs (More beautiful debug logs #3171)
tracer: postgres: DBM full service fallback w/ prepared statements (postgres: DBM full service fallback w/ prepared statements #3186)
tracer: Make HTTP clients fit in the plugin hierarchy (Make HTTP clients fit in the plugin hierarchy #3178)
ci-visibility: Extract code coverage from cypress ([ci-visibility] Extract code coverage from cypress #3159)
ci-visibility: Change gitlab's pipeline URL extraction ([ci-visibility] Change gitlab's pipeline URL extraction #3183)
ci-visibility: Test skipping logic for cypress ([ci-visibility] Test skipping logic for cypress #3167)
waf: Update AppSec blocking templates (Update AppSec blocking templates #3181)
waf: Update AppSec rules to 1.7.1 (Update Appsec rules to 1.7.1 #3185)
iast: Detect SQL injection with sequelize (Detect SQL injection with sequelize #3154)

Bug Fixes

iast: Fix evidence redaction (Fix IAST evidence redaction #3160)
iast: Fix path traversal vulnerability detection on close file (Fix path traversal vulnerability detection on close #3172)
ci-visibility: Fix cucumber parallel mode ([ci-visibility] Fix cucumber parallel mode #3156)
ci-visibility: Remove git.properties error log (Remove git.properties error log #3179)
ci-visibility: Fix playwright@1.30.0 ([ci-visibility] Fix playwright@1.30.0 #3180)
waf: Fix ASM_DD batch update (Fix ASM_DD batch update #3165)

* add migration guide for 3.x to 4.x * Update MIGRATING.md Co-authored-by: Juan Antonio Fernández de Alba <juan.fernandezdealba@datadoghq.com> * Update MIGRATING.md Co-authored-by: Juan Antonio Fernández de Alba <juan.fernandezdealba@datadoghq.com> * Update README.md Co-authored-by: Ugaitz Urien <ugaitz.urien@datadoghq.com> --------- Co-authored-by: Juan Antonio Fernández de Alba <juan.fernandezdealba@datadoghq.com> Co-authored-by: Ugaitz Urien <ugaitz.urien@datadoghq.com>

* update readme to include v4 * use version module instead of semver to match library version

* Add test for vulnerability evidence scrubber * Fix IAST SQL redaction with tainted contained in sensitive

'process' is the safer export strategy for heap profile export upon oom.

…3166)

* introduce DD_TRACE_SPAN_ATTRIBUTE_SCHEMA env var * add attribute schema v0 for rhea * add attribute schema v1 for rhea * add test harness for service/operation naming * grab config object from pluginManager init * provide schema autoresolution in consumer/producer plugins * bind service to schema manager at configure time * rename plugins to match inbound/outbound service naming terminology * minimize test dependencies * naming resolution wrt version in the test object uses the Nomenclature, instead of being resolved by the test fixture * we no longer use the test fixture for _all_ existing tests, rather let the default resolution do the work * the testing fixture accepts a callback which is a minimal viable trace retrieval, on which we examine _only_ service and name * split test naming schema from test code * Apply service naming flow to messaging integrations (#2961) * add v0 to all messaging plugins * add v1 to all messaging plugins * test naming schema for all messaging integrations * add naming schema tests for other versions * bake messaging data into producer/consumer plugins * persist kind in plugins and infer naming subtype from kind+type * no logs on empty DD_SPAN_TRACE_ATTRIBUTE_SCHEMA * don't compute service name unless necessary

- attempting to fix flaky windows tests

Co-authored-by: Ugaitz Urien <ugaitz.urien@datadoghq.com>

Adopt service naming schema in cache integrations --------- Co-authored-by: Thomas Hunter II <tlhunter@datadog.com>

* implement RC custom rules * update waf bindings to 3.2.0

--------- Co-authored-by: Stephen Belanger <stephen.belanger@datadoghq.com> Co-authored-by: Igor Unanua <igor.unanua@datadoghq.com>

* Taint cookies and headers * Bump minimum node version for v4 on cookie plugin test * Add test with latest node version for cookie plugin test * Provide iastContext from index when tainting headers * Add test for cookie tainting in taint tracking plugin * Remove iast transaction after taint tracking plugin tests to avoid hiting setMaxTransactions in tests * Add test for taintObject with taintingKeys flag * Address header tainting test for keys shorter than 10 chars

This reverts commit cc51732.

- when DBM is set to full service - the DBM comment falls back to service mode - but only when sending a prepared statement - without this, each prepared statement query is technically different - this causes the pg library to fail as it does an exact string check of the query - ideally the pg library would somehow parse out and not consider comments - at any rate this brings parity with other tracer implementations - see brianc/node-postgres#2735 - previously attempted in 0ff9465

* Detect SSRF vulnerabilities * Fix test * Add space * Understand arguments in publisher instead of the subscriber * Redact sensitive information in SSRF vulnerabilities * Tiny style change * Tiny code styles * Use SSRF enum instead of literal * Try to reduce flaky test * Rename originalArgs to originalUrlAndOptions * Fix comment in PR * Do not normalize arguments twice in http/client.js

* Initial version of insecure-cookie vulnerability * Small fix * Add tests * rename const * Fix test * Exclude express file from insecure cookie stack trace * Some code styles + tests * Small code spaces * intermediate-cookies-analyzer to set-cookies-header-intercepteor * Comments in the PR * Comments in PR * Ignore insecure cookie when cookie value is empty * Reuse excluded paths array

* Update blocking page and status from RC or rules file * Use if/else instead of return * Code styles * Split block in two methods * Fix test * Unapply after test * Fix tests * Reorder params in method * Change the signature of updateBlockingConfiguration method * Clear blocking configuration on clear rules * Update blocking response type by configuration * Fix lint

* move http client to clientPlugin * move http2 client to clientPlugin

github-actions · 2023-06-01T14:23:11Z

Overall package size

Self size: 4.23 MB
Deduped: 58.43 MB
No deduping: 58.47 MB

Dependency sizes

name	version	self size	total size
@datadog/pprof	2.2.1	14.24 MB	15.12 MB
@datadog/native-iast-taint-tracking	1.4.1	14.85 MB	14.86 MB
@datadog/native-appsec	3.2.0	13.38 MB	13.39 MB
protobufjs	7.1.2	2.76 MB	6.55 MB
@datadog/native-iast-rewriter	2.0.1	2.09 MB	2.1 MB
@datadog/native-metrics	2.0.0	898.77 kB	1.3 MB
opentracing	0.14.7	194.81 kB	194.81 kB
semver	7.3.8	88.2 kB	118.6 kB
@datadog/sketches-js	2.1.0	109.9 kB	109.9 kB
lodash.sortby	4.7.0	75.76 kB	75.76 kB
lru-cache	7.14.0	74.95 kB	74.95 kB
ipaddr.js	2.0.1	59.52 kB	59.52 kB
ignore	5.2.0	48.87 kB	48.87 kB
import-in-the-middle	1.3.5	34.34 kB	38.81 kB
istanbul-lib-coverage	3.2.0	29.34 kB	29.34 kB
retry	0.10.1	27.44 kB	27.44 kB
lodash.uniq	4.5.0	25.01 kB	25.01 kB
limiter	1.1.5	23.17 kB	23.17 kB
lodash.kebabcase	4.1.1	17.75 kB	17.75 kB
lodash.pick	4.4.0	16.33 kB	16.33 kB
node-abort-controller	3.0.1	14.33 kB	14.33 kB
crypto-randomuuid	1.0.0	11.18 kB	11.18 kB
diagnostics_channel	1.1.0	7.07 kB	7.07 kB
path-to-regexp	0.1.7	6.78 kB	6.78 kB
koalas	1.0.2	6.47 kB	6.47 kB
methods	1.1.2	5.29 kB	5.29 kB
module-details-from-path	1.0.3	4.47 kB	4.47 kB

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

codecov · 2023-06-01T14:27:38Z

Codecov Report

❗ No coverage uploaded for pull request base (v4.x@ea026bb). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             v4.x    #3209   +/-   ##
=======================================
  Coverage        ?   86.52%           
=======================================
  Files           ?      337           
  Lines           ?    12088           
  Branches        ?       33           
=======================================
  Hits            ?    10459           
  Misses          ?     1629           
  Partials        ?        0

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

pr-commenter · 2023-06-01T14:39:33Z

Benchmarks

Comparing candidate commit df3b438 in PR branch v4.1.0-proposal with baseline commit ea026bb in branch v4.x.

Found 0 performance improvements and 36 performance regressions! Performance is the same for 412 metrics, 24 unstable metrics.

scenario:log-with-error-16

🟥 cpu_user_time [+26.315ms; +31.055ms] or [+18.970%; +22.386%]
🟥 execution_time [+29.086ms; +30.730ms] or [+17.948%; +18.963%]
🟥 instructions [+80; +80] or [+18.990%; +19.166%]

scenario:log-with-debug-16

🟥 cpu_user_time [+24.952ms; +31.246ms] or [+18.129%; +22.701%]
🟥 execution_time [+28.589ms; +29.959ms] or [+17.658%; +18.504%]
🟥 instructions [+77; +81] or [+18.290%; +19.422%]
🟥 max_rss_usage [+2.550KB; +2.669KB] or [+5.295%; +5.542%]

scenario:log-skip-log-16

🟥 cpu_user_time [+26.987ms; +32.160ms] or [+19.526%; +23.268%]
🟥 execution_time [+29.315ms; +32.155ms] or [+18.097%; +19.850%]
🟥 instructions [+80; +80] or [+19.037%; +19.179%]

scenario:log-without-log-16

🟥 cpu_user_time [+25.417ms; +29.974ms] or [+18.248%; +21.520%]
🟥 execution_time [+26.291ms; +29.641ms] or [+16.129%; +18.185%]
🟥 instructions [+66; +66] or [+15.784%; +15.828%]
🟥 max_rss_usage [+2.479KB; +2.574KB] or [+5.138%; +5.334%]

scenario:net-with-tracer-16

🟥 instructions [+42; +44] or [+6.122%; +6.389%]

scenario:startup-with-tracer-16

🟥 cpu_user_time [+14.743ms; +22.898ms] or [+10.775%; +16.735%]
🟥 execution_time [+17.198ms; +18.350ms] or [+10.738%; +11.458%]

scenario:log-with-debug-18

🟥 cpu_user_time [+22.737ms; +28.948ms] or [+16.100%; +20.498%]
🟥 execution_time [+28.844ms; +29.742ms] or [+17.276%; +17.814%]
🟥 instructions [+80; +80] or [+19.033%; +19.113%]

scenario:log-with-error-18

🟥 cpu_user_time [+26.443ms; +31.148ms] or [+18.775%; +22.116%]
🟥 execution_time [+28.793ms; +30.771ms] or [+17.231%; +18.414%]
🟥 instructions [+80; +80] or [+19.024%; +19.121%]

scenario:log-skip-log-18

🟥 cpu_user_time [+23.664ms; +29.934ms] or [+16.816%; +21.272%]
🟥 execution_time [+28.673ms; +29.717ms] or [+17.124%; +17.747%]
🟥 instructions [+79; +80] or [+18.967%; +19.051%]

scenario:log-without-log-18

🟥 cpu_user_time [+23.320ms; +29.266ms] or [+16.416%; +20.602%]
🟥 execution_time [+26.926ms; +29.596ms] or [+16.061%; +17.654%]
🟥 instructions [+65; +66] or [+15.662%; +15.774%]

scenario:appsec-control-with-attacks-18

🟥 cpu_user_time [+25.423ms; +31.190ms] or [+9.708%; +11.910%]

scenario:appsec-control-18

🟥 cpu_user_time [+20.937ms; +27.582ms] or [+7.489%; +9.866%]

scenario:plugin-bluebird-with-tracer-18

🟥 cpu_user_time [+21.743ms; +27.034ms] or [+9.021%; +11.217%]
🟥 execution_time [+24.190ms; +25.372ms] or [+8.702%; +9.128%]

scenario:startup-with-tracer-18

🟥 cpu_user_time [+8.665ms; +18.343ms] or [+6.159%; +13.038%]
🟥 execution_time [+18.306ms; +18.883ms] or [+11.039%; +11.387%]
🟥 instructions [+21; +21] or [+5.059%; +5.147%]

juan-fernandez

looks good from ci visibility's perspective. Could we remove this item from the release notes, though, as it isn't interesting for the user at all:

ci-visibility: Use new endpoint for code coverage ([ci-visibility] Use new endpoint for code coverage #3157)

rochdev and others added 30 commits June 1, 2023 15:54

update readme for v4 and remove semver for version feature flags (#3148)

81d3bd8

* update readme to include v4 * use version module instead of semver to match library version

modifying benchmarks to reduce non-determinism (#3150)

f21de3d

Fix IAST evidence redaction (#3160)

8076e8c

* Add test for vulnerability evidence scrubber * Fix IAST SQL redaction with tainted contained in sensitive

Use 'process' as default strategy for oom export (#3136)

66a6f55

'process' is the safer export strategy for heap profile export upon oom.

Add test for pg native bindings. Pin pg version for v2 appsec test (#…

75b3b37

…3166)

test: force a stop before starting metrics (#3173)

7b744ca

- attempting to fix flaky windows tests

Fix path traversal vulnerability detection on close (#3172)

65e7985

More beautiful debug logs (#3171)

21367cd

[ci-visibility] Use new endpoint for code coverage (#3157)

ceeb09e

[ci-visibility] Fix cucumber parallel mode (#3156)

5a5e733

[ci-visibility] Extract code coverage from cypress (#3159)

e625253

Co-authored-by: Ugaitz Urien <ugaitz.urien@datadoghq.com>

Cache integrations - service naming (#3056)

75925ac

Adopt service naming schema in cache integrations --------- Co-authored-by: Thomas Hunter II <tlhunter@datadog.com>

Fix ASM_DD batch update (#3165)

f2302b1

Remove git.properties error log (#3179)

b9191de

Implement RC custom rules (#3126)

fca2369

* implement RC custom rules * update waf bindings to 3.2.0

[ci-visibility] Fix playwright@1.30.0 (#3180)

db349cb

Update AppSec blocking templates (#3181)

6a71c45

Update Appsec rules to 1.7.1 (#3185)

4d87546

Detect SQL injection with sequelize (#3154)

58d3fbd

--------- Co-authored-by: Stephen Belanger <stephen.belanger@datadoghq.com> Co-authored-by: Igor Unanua <igor.unanua@datadoghq.com>

[ci-visibility] Change gitlab's pipeline URL extraction (#3183)

5c3bb66

Add more scenarios to system-tests (#3191)

52eb3b9

Revert "Taint cookies and headers (#3182)" (#3192)

3fbbaad

This reverts commit cc51732.

[ci-visibility] Test skipping logic for cypress (#3167)

73e6979

jbertran and others added 2 commits June 1, 2023 15:56

Make HTTP clients fit in the plugin hierarchy (#3178)

c273f1d

* move http client to clientPlugin * move http2 client to clientPlugin

Replace rmSync by unlinkSync (#3208)

5d7d3b3

v4.1.0

df3b438

tlhunter approved these changes Jun 1, 2023

View reviewed changes

uurien marked this pull request as ready for review June 1, 2023 16:31

uurien requested review from a team as code owners June 1, 2023 16:31

CarlesDD approved these changes Jun 1, 2023

View reviewed changes

uurien changed the title ~~V4.1.0 proposal~~ v4.1.0 proposal Jun 1, 2023

juan-fernandez approved these changes Jun 2, 2023

View reviewed changes

uurien merged commit c1b80aa into v4.x Jun 2, 2023
102 checks passed

uurien deleted the v4.1.0-proposal branch June 2, 2023 07:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v4.1.0 proposal #3209

v4.1.0 proposal #3209

uurien commented Jun 1, 2023 •

edited

github-actions bot commented Jun 1, 2023 •

edited

codecov bot commented Jun 1, 2023 •

edited

pr-commenter bot commented Jun 1, 2023

juan-fernandez left a comment

v4.1.0 proposal #3209

v4.1.0 proposal #3209

Conversation

uurien commented Jun 1, 2023 • edited

Features

Improvements

Bug Fixes

github-actions bot commented Jun 1, 2023 • edited

Overall package size

Dependency sizes

codecov bot commented Jun 1, 2023 • edited

Codecov Report

pr-commenter bot commented Jun 1, 2023

Benchmarks

scenario:log-with-error-16

scenario:log-with-debug-16

scenario:log-skip-log-16

scenario:log-without-log-16

scenario:net-with-tracer-16

scenario:startup-with-tracer-16

scenario:log-with-debug-18

scenario:log-with-error-18

scenario:log-skip-log-18

scenario:log-without-log-18

scenario:appsec-control-with-attacks-18

scenario:appsec-control-18

scenario:plugin-bluebird-with-tracer-18

scenario:startup-with-tracer-18

juan-fernandez left a comment

Choose a reason for hiding this comment

uurien commented Jun 1, 2023 •

edited

github-actions bot commented Jun 1, 2023 •

edited

codecov bot commented Jun 1, 2023 •

edited