Sourced stream with subject + source subject mapping

[ruidscampos]

Feature Request

Support for subject in sourced stream.
Support for subject mapping of each sourced stream.

Use Case:

Sourced streams are great as they are independent from the source, meaning the source doesn't need to know about them.
As per design sourced streams loose the concept of subject and become containers of messages, and we can either use headers or content in the message.

Ideally sourced streams could support their own subject, and sources could be mapped to it, and the stream could also be published to directly.

Proposed Change:

Let's assume the following existing streams:

name: source-stream-1
subject: source.1.subject.*

name: source-stream-2
subject: source2.something.subject.*

name: source-stream-3
no-subject: sourced from another stream, subject could be assumed as empty (or a new identifier created for these cases)

We could then create a sourced stream as follows:

name: new-sourced-stream
subject: new.sourced.stream..
source-mapping for source-stream-1: "source.1.subject." : "sourced.path.source1.{{wildcard(1)}}"
source-mapping for source-stream-2: "source2.something.subject." : "sourced.path.source2.{{wildcard(1)}}"
source-mapping for source-stream-3: "" : "sourced.path.source3.source3"

There are limitations to this.
If a subject is added to the sourced stream, then all source streams must have a subject mapping.
The source streams must either have a subject or the mapping be done to a fixed subject that fits within the stream subject.

All other functionality would work as if a regular subject stream had been created.

Who Benefits From The Change(s)?

This enables advanced scenarios that are easier to design and manage, while avoiding apriori coupling like republish, meaning the source does not need to be changed for something that is handled in parallel or afterwards in the chain.

This benefits some auditing scenarios, and is quite useful for system redesigns / refactors with no downtime + lower risk of errors.

Alternative Approaches

This can be done partially with republish, but requires doing so in all sources, which feels like polution when used for auditing/logging/cyber-security, where the ideal way is to select the sources and remap them into an audit/logging/cyber stream that the origin doesn't need to know about.

[jnmoyne]

As it happens there's been some recent work on introducing subject mapping transform features to streams and which covers what you are asking about (and some more) and should be merged into the dev branch very soon: see nats-io/nats-server#3814

So this may be the quickest ever implementation of a feature request :)

[ruidscampos]

Thanks.
Indeed it's almost a direct match, looking forward to it.

[ruidscampos]

@jnmoyne, any idea when this is expected to be released?
Have a use case that will be complex to implement without this and want to understand if it's worth delaying it to wait for this, or simply use the more complex approach.

[jnmoyne]

@ruidscampos the feature has been merged into the dev branch of nats-server (and exposed in the dev version of nats.go, and there's a PR for it in natscli as well), so you can already start playing with it or developing and testing. The dev branch of nats-server is a beta of 2.10 (will become 2.10 as it's released)

[ruidscampos]

@jnmoyne Perfect, thank you.