Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is druid.segmentCache.locations is used to configure the maximum storage size for hot data? #16265

Open
lukoou3 opened this issue Apr 11, 2024 · 0 comments

Comments

@lukoou3
Copy link

lukoou3 commented Apr 11, 2024

Is druid.segmentCache.locations is used to configure the maximum storage size for hot data?

config druid.segmentCache.locations 300g, when data increases, kafka ingestion task fail:

WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [15] Segments

Affected Version

26.0.0.

Description

  • Cluster size : 5
  • Configurations in use: config druid.segmentCache.locations 300g, deep-storage: hdfs

when data increases, kafka ingestion task fail:

WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [15] Segments

historical segmentCache.locations Usage : 100%

Is druid.segmentCache.locations is used to configure the maximum storage size for hot data? Delete cold data when there is not enough space, and only pull it from HDFS when querying cold data?

Do druid.segmentCache.locations need to store all undeleted data, not just hot data?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant