Jetstream: Support for multiple "filter subject" patterns in consumer

[bruth]

Feature Request

Similar to how a stream can be configured to bind multiple subject patterns, this request is to allow for a consumer to be configured with multiple filter subject patterns. Currently only one pattern can be specified.

Use Case:

My specific use case is modeling an event sourcing type of workload where I am modeling subjects as entities/state machines/workflows and there are a sequence of events written per subject. There are cases where I need to consume events from more than one subject in order to process a new command.

Currently, I am managing this by consume a broader scope of the stream, e.g. events.> and then doing local filtering. Supporting multiple filter subjects would make the consumers more targeted and reduce the data transferred and latency since it would be filtered server-side.

Also to clarify, this is all scoped to subjects in a single stream, not across streams, since I know that is a different problem (but interesting in its own right!)

Proposed Change:

To not break compatibility with the singular FilterSubject, this would add a new field FilterSubjects to take a slice of filter patterns.

Who Benefits From The Change(s)?

Anyone need to create a consumer with disjoint subjects, specifically not supported by wildcard hierarchy, e.g. []string{"foo.bar", "baz.qux"} or selecting specific subjects where a wildcard match would yield too much data, []string{"foo.1", "foo.2"} (where foo.* may have thousands or millions).

Alternative Approaches

The alternatives are either to do local filtering with a broader scoped filter or to creating multiple consumers per specific filter and combine the events/messages locally.

[nadihagh]

I think this is really needed,
specially for messaging solutions

[kim-energyweb]

Any update on this please? Thank you.

[Jarema]

Yes, we're looking into this.
Expect more info soon.

[kindermoumoute]

Random idea on this topic: why not using Common Expression Language (cf. Go implementation) in order to filter out all kind of message based on their attributes?

# Filter a list of subject or subjects matching a regexp
msg.subject in ["foo.bar", "baz.qux"] || msg.subject.matches("(foo|bars?)_foo")

# Filter all messages sent at 10 AM with a particular header
msg.header["color"] == "blue" && msg.timestamp.getHours() == 10

This could also be extended to do some payload-based filtering with support for some serialization (JSON, protobuf, ...):

msg.payload.JSON().field1 == "foo" || msg.payload.protobuf().field3 == "bar"

[derekcollison]

This might be better served through our callout architecture which is underway. We prefer to try to keep the server as simple as we can.

[autodidaddict]

Just found this issue. My own app architecture would also be greatly simplified if I could comma separate my interest declaration.

Any update on the roadmap for this?

[Jarema]

@autodidaddict PR introducing this change already has been merged into dev branch and will be part od 2.10.0 release.

Additionally, async-nats has that feature released under server_2_10 feature flag (disabled by default until actual nats server 2.10 is released).

#3500 - implementation PR

[autodidaddict]

Sweet thanks for the update

[aeb-dev]

The title start with JetStream but is this only applicable to JetStream? I believe Nats Core also support this case, right?

[bastien-m]

The title start with JetStream but is this only applicable to JetStream? I believe Nats Core also support this case, right?

It only apply to jetstream since consumer is a notion of jetstream.

[stnoa]

@autodidaddict PR introducing this change already has been merged into dev branch and will be part od 2.10.0 release.

Additionally, async-nats has that feature released under server_2_10 feature flag (disabled by default until actual nats server 2.10 is released).

#3500 - implementation PR

Since this issue is still open, this was not part of 2.10 I guess?

[bruth]

Yes this has been released in 2.10. Announcement post, docs, and 2.10 guide coming up next week.

[stnoa]

Ah ok great. Thank you.