You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is to report unexpected behavior where the broker selects too many segments to query. The identified scenario is when secondary partition information exists (eg, for range partitioning) but is not being used in conjunction with a query filter which corresponds to the secondary partitions. The result of not completely pruning segments at the broker is a detrimental impact on system operations such as for performance and cost.
Affected Version
28
Description
The /druid/v2 and /druid/v2/candidates broker endpoints are returning segments which should be filtered out based on secondary partition metadata. Returning unneeded segments for planning and executing queries will cause unnecessary I/O throughout the system, causing avoidable detriment to cost and performance.
Steps to reproduce:
create segments with range partitioning, mark as used, load onto historicals. Testing will require multiple segments per combination of datasource and time chunk.
Identify the partition column value(s) corresponding to a single segment (within a datasource/time chunk)
submit /v2 and /v2/candidates requests to the broker for the datasource and time chunk identified. Add a query filter using a partition value previously identified for the first partition column.
Expected/observed behavior summary
The expected behavior is that the response will return only the segment identified and no others for the loaded datasource/time chunk.
The observed behavior is that the response will include additional segments for the loaded datasource/time chunk.
Testing notes
Testing used a segment where the partition/filter column was not a setwise start or end (ie, not indicated with a negative or positive infinity value for the column). Further, to reduce ambiguity, a partition filter value was chosen that was not the first or last for the column within the segment.
The test setup had multiple partition columns in the queried segments but only filtered on the first partition column
Testing both included and omitted the query context value "secondaryPartitionPruning"
Some testing included "bySegment" in the query context to assess broker logic for selecting segments
Testing only queried a single time chunk (aligning with segment granularity). This shouldn't be strictly necessary, but reproductions of the test may need to at least ensure that all segments in the time chunk have the same partitioning.
Representation of the test setup
segments for "src1"
there are multiple segments for the tested time chunk
If partition pruning is reflected at the broker, it would be unexpected to receive back any of "the other two segments" (regardless of server/replication)
Unexpected result example 1: /druid/v2 endpoint - extra (but not all) segments are returned
expected only src1_2025-02-16T01:00:00.000Z_2025-02-16T02:00:00.000Z_2024-04-01T23:59:59.999Z_1
(FYI) unexpected segments are loaded on different servers than the expected segments
The broker should only select and/or query the smallest number of segments matching on datasource/time chunk and partition dimensions when it has enough has enough partition information about the used/loaded segments to prune out segments that don't match on partition dimensions (in addition to those segments not matching on datasource or time chunk)
This is to report unexpected behavior where the broker selects too many segments to query. The identified scenario is when secondary partition information exists (eg, for range partitioning) but is not being used in conjunction with a query filter which corresponds to the secondary partitions. The result of not completely pruning segments at the broker is a detrimental impact on system operations such as for performance and cost.
Affected Version
28
Description
The /druid/v2 and /druid/v2/candidates broker endpoints are returning segments which should be filtered out based on secondary partition metadata. Returning unneeded segments for planning and executing queries will cause unnecessary I/O throughout the system, causing avoidable detriment to cost and performance.
Steps to reproduce:
Expected/observed behavior summary
Testing notes
Representation of the test setup
segments for "src1"
server segments for datasource "src1"
segment metadata - range partition column values
test query
Expected result example - /v2/candidates
If partition pruning is reflected at the broker, it would be unexpected to receive back any of "the other two segments" (regardless of server/replication)
Unexpected result example 1: /druid/v2 endpoint - extra (but not all) segments are returned
src1_2025-02-16T01:00:00.000Z_2025-02-16T02:00:00.000Z_2024-04-01T23:59:59.999Z_1
Unexpected result example 2: /druid/v2/candidates endpoint - all segments are returned
src1_2025-02-16T01:00:00.000Z/2025-02-16T02:00:00.000Z_2024-04-01T23:59:59.999Z_1
Summary
The broker should only select and/or query the smallest number of segments matching on datasource/time chunk and partition dimensions when it has enough has enough partition information about the used/loaded segments to prune out segments that don't match on partition dimensions (in addition to those segments not matching on datasource or time chunk)
Related documentation
The text was updated successfully, but these errors were encountered: