Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Headers on KeyValue storage #237

Open
davidmcote opened this issue Aug 24, 2023 · 5 comments
Open

Support for Headers on KeyValue storage #237

davidmcote opened this issue Aug 24, 2023 · 5 comments

Comments

@davidmcote
Copy link

What motivated this proposal?

Message headers encoded separately from a binary-blob payload are extremely handy capability of core NATS and Jetstream.

The lower level library bindings for Jetstream make headers readily accessible for application use, but the KeyValue abstraction does not expose access to message headers.

Since KV storage is just another stream, there doesn't appear to be a technical limitation to enabling headers. There simply isn't API bindings for it.

What is the proposed change?

  1. In https://github.com/nats-io/nats.java/blob/main/src/main/java/io/nats/client/KeyValue.java#L24, overload create/update/put methods with versions that accept a Headers object.
  2. In https://github.com/nats-io/nats.java/blob/main/src/main/java/io/nats/client/api/KeyValueEntry.java#L28, introduce a Headers datamember with accessors. Populate it in the constructor from Message.getHeaders().

Who benefits from this change?

Developers are already using Jetstream to store messages with headers. Some would like to do the same with the KeyValue abstraction.

What alternatives have you evaluated?

One may leverage lower level APIs to store KeyValue entries with headers. This is demonstrated below with the natscli. It unfortunately pierces through the KeyValue abstraction and requires the application developer to string munge stream names and subject patterns to access data in the lower level stream.

Demonstration with natscli:

Create a bucket and store KVs, one with a custom header:

$ nats kv add mybucket

$ nats kv put mybucket foo bar
bar

$ nats req '$KV.mybucket.biz' -H myheaderkey:myheadervalue baz
10:13:40 Sending request on "$KV.mybucket.biz"
10:13:40 Received with rtt 432.206µs
{"stream":"KV_mybucket", "seq":2}

Inspect contents of KV with KV CLI. Headers omitted.

$ nats kv ls mybucket
foo
biz

$ nats kv get mybucket foo
mybucket > foo created @ 24 Aug 23 14:11 UTC

bar

$ nats kv get mybucket biz
mybucket > biz created @ 24 Aug 23 14:13 UTC

baz

Inspect contents of underlying stream to reveal headers.

$ nats stream view KV_mybucket
[1] Subject: $KV.mybucket.foo Received: 2023-08-24T10:11:59-04:00

bar

[2] Subject: $KV.mybucket.biz Received: 2023-08-24T10:13:40-04:00

  myheaderkey: myheadervalue

baz

$ nats stream get  KV_mybucket -S '$KV.mybucket.foo'
Item: KV_mybucket#1 received 2023-08-24 14:11:59.009016 +0000 UTC on Subject $KV.mybucket.foo

bar

$ nats stream get  KV_mybucket -S '$KV.mybucket.biz'
Item: KV_mybucket#2 received 2023-08-24 14:13:40.951116 +0000 UTC on Subject $KV.mybucket.biz

Headers:
  myheaderkey: myheadervalue

baz
@scottf
Copy link
Collaborator

scottf commented Aug 24, 2023

@davidmcote This is an architectural issue spanning the server and all clients. I am moving this request to the appropriate repo.

@scottf scottf transferred this issue from nats-io/nats.java Aug 24, 2023
@ripienaar
Copy link
Contributor

You are saying that it would benefit - but do not describe exactly what the benefit is.

A KV is exactly a 2 part item - key and value. It's not 3 part - KVH. We limit the functionality to match other KVs to keep things simple, understandable and to keep avenues open for replacing the backend later with another.

So for headers people usually store a serialized hash/map/struct where one of the keys are the metadata.

If you can expand why that is not a suitable choice for you we can consider it better.

@davidmcote
Copy link
Author

Thanks for the reply!

The proposal of this ticket is a backward compatible change to the KeyValue API which would allow applications to opt-in to using Headers. Existing application code would continue to use KV in its current form.

An application developer choosing to depend on the new capability would certainly have a harder time migrating to another backend, but whether to limit their use of this feature does seem like it would be their decision to make.


In my specific use case, I am using a serialization library to store and load structures as values in a key value store. My structures happen to be polymorphic.

While serialization of a polymorphic type is no problem, deserialization generally requires type information as input to map back to a class. My serialization library does not encode type information inline and I have limited control over the serialization library. This motivates supplementing the byte[] payload value somehow to communicate a runtime type/class to decode into.

In lieu of this proposal, I'm constrained to either:

  1. Switch to a serialization format that encodes and decodes type information inline.
  2. Develop an envelope structure to encapsulate the real serialized value alongside type information.
  3. Communicate type information in Headers, but forego the KeyValue abstraction and re-implement much of the desired KeyValue behavior on top of regular streams.

Only the solution with Headers appears to avoid changing the encoding scheme of byte[] values.

@ripienaar
Copy link
Contributor

The concern about backends is our ability to move to other backends with out user impacting changes.

To my mind choice 2 seems most compatible with a KV store. Or perhaps pre/postpend a type hint to the serialized data without the whole overhead of a wrapping structure and multiple unmarshals.

The KV implementation is popular specifically because it removes features and complexity from streams. Really reluctant to feature creap into it being a stream.

@davidmcote
Copy link
Author

Thank you. I appreciate you taking the time to consider this proposal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants