-
Notifications
You must be signed in to change notification settings - Fork 524
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor: Decode StreamChunk
proto into primitive data types with Java instead of Rust/JNI
#16738
Comments
Since we are doing embedded connector node, we can simply pass a pointer of stream chunk from rust to java, so that we don't need serializing and deserializing the stream chunk. We have code to convert stream chunk to arrow. We can try encoding to arrow in rust and pass to java, and on java side we decode with arrow java sdk. We can have a comparison on the cost of encode-decode and jni native method call to see which one to adopt. |
do some bench with arrow. It seems that there is a slight improvement in arrow 50 columns , type is int 5columns type is int int bigint varchar varchar |
the draft test code is in |
Is your feature request related to a problem? Please describe.
Now it calls JNI function (in class
Binding
) to decode every value. The overhead sounds high to me.Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: