Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Include ksqldb-rocksdb-config-setter jar in distributed artifacts #6838

Open
colinhicks opened this issue Jan 7, 2021 · 5 comments · May be fixed by #7372 or #10258
Open

Include ksqldb-rocksdb-config-setter jar in distributed artifacts #6838

colinhicks opened this issue Jan 7, 2021 · 5 comments · May be fixed by #7372 or #10258
Assignees
@wcarlson5
Labels
core-infra Issues owned by the ksqlDB Core Infra team enhancement fix-it-week P1 Slightly lower priority to P0 ;)

Comments

@colinhicks
Copy link
Member

The How to Limit Memory Usage in ksqlDB blog post details the use of KsqlBoundedMemoryRocksDBConfigSetter, but the containing jar is not on the classpath – nor included at all in releases, including the standalone Docker image.

This yields the following when the custom config setter is otherwise correctly configured:

org.apache.kafka.common.config.ConfigException: Invalid value io.confluent.ksql.rocksdb.KsqlBoundedMemoryRocksDBConfigSetter for configuration rocksdb.config.setter: Class io.confluent.ksql.rocksdb.KsqlBoundedMemoryRocksDBConfigSetter could not be found.
	at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:729)
	at io.confluent.ksql.config.ConfigItem$Resolved.parseValue(ConfigItem.java:125)
	at io.confluent.ksql.util.KsqlConfig.lambda$resolveStreamsConfig$2(KsqlConfig.java:902)
	at java.base/java.util.Optional.map(Optional.java:265)

At minimum, we should ensure that the ksqldb-rocksdb-config-setter jar is included in distributed artifacts. We should consider altering ksql-run-class to automatically add it to the classpath.

As a workaround, users of the standalone Docker image would need to build the ksqldb-rocksdb-config-setter jar at the release version, mount it in the container, and ensure it is added to the classpath.
@cprasad1 cprasad1 added P1 Slightly lower priority to P0 ;) and removed needs-triage labels Jan 12, 2021
@guozhangwang guozhangwang added core-infra Issues owned by the ksqlDB Core Infra team query-engine Issues owned by the ksqlDB Query Engine team streaming-engine Tickets owned by the ksqlDB Streaming Team labels Jan 14, 2021
@vcrfxia vcrfxia removed query-engine Issues owned by the ksqlDB Query Engine team streaming-engine Tickets owned by the ksqlDB Streaming Team labels Jan 22, 2021
@wcarlson5 wcarlson5 self-assigned this Mar 10, 2021
@vcrfxia vcrfxia mentioned this issue Mar 18, 2021
Closed
@vcrfxia
Copy link
Contributor

vcrfxia commented Mar 18, 2021

There's another workaround documented in #6644: besides building the config setter module themselves, users can copy the jar from our maven repo.

@vcrfxia
Copy link
Contributor

vcrfxia commented Mar 18, 2021

@wcarlson5 should this be labeled for 0.18?

@wcarlson5 wcarlson5 linked a pull request Apr 12, 2021 that will close this issue
chore: add rocksdb-config-setter to class path #7372
Open
2 tasks
@ChenZhaobin
Copy link

ChenZhaobin commented Jun 16, 2021 •
edited

specifically ,how to alter ksql-run-class in order to use ksqldb-rocksdb-config-setter? Do you mean download ksqldb code ,compile ksqldb-rocksdb-config-setter.jar,upload it to some dir on ksqldb server,then add dir path to KSQL_CLASSPATH in ksql-run-class?

@ChenZhaobin
Copy link

I download jar here :
https://ksqldb-maven.s3-us-west-2.amazonaws.com/maven/io/confluent/ksql/ksql-rocksdb-config-setter/0.7.1/ksql-rocksdb-config-setter-0.7.1.jar

it works when I put it in dir /ksqldb/customclass
and then vi /ksql-run-class ,add one line KSQL_CLASSPATH="/ksqldb/customclass/*:$KSQL_CLASSPATH"

@dberardo-com
Copy link

dberardo-com commented Nov 12, 2022 •
edited

by the way do you know the effect of setting

ksql.plugins.rocksdb.cache.limit.strict=<boolean>

ref: https://github.com/confluentinc/ksql/blob/0.28.2-ksqldb/ksqldb-rocksdb-config-setter/src/main/java/io/confluent/ksql/rocksdb/KsqlBoundedMemoryRocksDBConfig.java

@cefothe cefothe linked a pull request Mar 5, 2024 that will close this issue
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wcarlson5 wcarlson5

Labels
core-infra Issues owned by the ksqlDB Core Infra team enhancement fix-it-week P1 Slightly lower priority to P0 ;)
Projects
None yet
Milestone
No milestone
8 participants
@colinhicks @rodesai @guozhangwang @ChenZhaobin @vcrfxia @wcarlson5 @cprasad1 @dberardo-com