Add support for multisubject filtering and transformation in sources …

…and mirror introduced in nats-io/nats-server#4354

Signed-off-by: Jean-Noël Moyne <jnmoyne@gmail.com>

adr/ADR-36.md

@@ -19,20 +19,84 @@ See [ADR-30](ADR-30.md) for Core NATS subject mapping and a description of the a
 
 ## Features introduced
 
-The new features introduced by the [PR](https://github.com/nats-io/nats-server/pull/3814) allow the application of subject mapping transformations in two places in the space configuration:
+The new features introduced by version 2.10 of the NATS server allow the application of subject mapping transformations in multiple places in the stream configuration:
 
+- You can apply a subject mapping transformation as part of a Stream mirror.
 - You can apply a subject mapping transformation as part of a Stream source.
   - Amongst other use cases, this enables the ability to do sourcing between KV bucket  (as the name of the bucket is part of the subject name in the KV bucket streams, and therefore has to be transformed during the sourcing as the name of the sourcing bucket is different from the name(s) of the bucket(s) being sourced).
 - You can apply a subject mapping transformation at the ingres (input) of the stream, meaning after it's been received on Core NATS, or mirrored or sourced from another stream, and before limits are applied (and it gets persisted). This subject mapping transformation is only that, it does not filter messages, it only transforms the subjects of the messages matching the subject mapping source.
   - This enables the ability to insert a partition number as a token in the message subjects.
+- You can also apply a subject mapping transformation as part of the re-publishing of messages.
+
+Subject mapping transformation can be seen as an extension of subject filtering, there can not be any subject mapping transformation without an associated subject filter.
+
+A subject filtering and mapping transform is composed of two parts: a subject filter (the 'source' part of the transform) and the destination transform (the 'destination' part of the transform). An empty (i.e. `""`) destination transform means _NO transformation_ of the subject.
 
 ![](images/stream-transform.png)
 
-In addition, it is now possible to source from the same stream more than once.
+Just like streams and consumers can now have more than one single subject filter, mirror and sources can have more than one set of subject filter and transform destination.
+
+Just like with consumers you can either specify a single subject filter and optional subject transform destination or an array of subject transform configs composed of a source filter and optionally empty transform destination.
+
+In addition, it is now possible to source not just from different streams but also from the same stream more than once.
+
+If you define a single source with multiple subject filters and transforms, in which case the ordering of the messages is guaranteed to be preserved, there can not be any overlap between the filters. If you define multiple sources from the same stream, subject filters can overlap between sources thereby making it possible to duplicate messages from the sourced stream, but the order of the messages between the sources is not guaranteed to be preserved.
+
+For example if a stream contains messages on subjects "foo", "bar" and "baz" and you want to source only "foo" and "bar" from that stream you could specify two subject filters and transforms in a single source, or you can source twice from that stream once with the "foo" subject filter and a second time with the "bar" subject filter.
+
+## Stream config structure changes
+
+From the user's perspective these features manifest themselves as new fields in the Stream Configuration request and Stream Info response messages.
+
+In Mirror and Sources :
+- Additional `"subject_transform_dest"` field and `"subject_transforms"` array in the `"sources"` array and in `"mirror"`. Note that if you use the `"subject_transforms"` array then you can _NOT_ also use either of the single string subject filter or transformation destination fields.
+
+At the top level of the Stream Config:
+- Additional `"subject_transform"` field in Stream Config containing two strings: `"src"` and `"dest"`
+
+## Examples
+
+A stream that mirrors the `sourcedstream` stream using two subject filters and transform (in this example `foo` is transformed, but `bar` is not):
+
+```JSON
+{
+  "name": "sourcingstream",
+  "retention": "limits",
+  "max_consumers": -1,
+  "max_msgs_per_subject": -1,
+  "max_msgs": -1,
+  "max_bytes": -1,
+  "max_age": 0,
+  "max_msg_size": -1,
+  "storage": "file",
+  "discard": "old",
+  "num_replicas": 1,
+  "duplicate_window": 120000000000,
+  "mirror":
+  {
+    "name": "sourcedstream",
+    "subject_transforms": [
+      {
+        "src": "foo",
+        "dest": "foo-transformed"
+      },
+      {
+        "src": "bar",
+        "dest": ""
+      }
+    ]
+  },
+  "sealed": false,
+  "deny_delete": false,
+  "deny_purge": false,
+  "allow_rollup_hdrs": false,
+  "allow_direct": false,
+  "mirror_direct": false
+}
+```
 
-For example if a stream contains messages on subjects "foo", "bar" and "baz" and you want to source only "foo" and "bar" from that stream you could stream twice from that stream once with the "foo" subject filter and a second time with the "bar" subject filter.
+A stream that sources from the `sourcedstream` stream twice, each time using a single subject filter and transform:
 
-E.g.
 ```JSON
 {
   "name": "sourcingstream",
@@ -50,11 +114,13 @@ E.g.
   "sources": [
     {
       "name": "sourcedstream",
-      "filter_subject": "foo"
+      "filter_subject": "foo",
+      "subject_transform_dest": "foo-transformed"
     },
     {
       "name": "sourcedstream",
-      "filter_subject": "bar"
+      "filter_subject": "bar",
+      "subject_transform_dest": "bar-transformed"
     }
   ],
   "sealed": false,
@@ -66,12 +132,8 @@ E.g.
 }
 ```
 
-From the user's perspective these features manifest themselves as new fields in the Stream Configuration request and Stream Info response messages.
-
-- Additional `"subject_transform_dest"` field in objects in the `"sources"` array of the Stream Config
-- Additional `"subject_transform"` field in Stream Config containing two strings: `"src"` and `"dest"`
+A Stream that sources from 2 streams and has a subject transform:
 
-E.g.
 ```JSON
 {
   "name": "foo",
@@ -99,13 +161,16 @@ E.g.
     },
     {
       "name": "source2",
-      "filter_subject": "stream2.foo.>",
-      "subject_transform_dest": "foo.>"
-    },
-    {
-      "name": "source2",
-      "filter_subject": "stream2.bar.>",
-      "subject_transform_dest": "bar.>"
+      "subject_transforms": [
+        {
+          "src": "stream2.foo.>",
+          "dest": "foo2.>"
+        },
+        {
+          "src": "stream2.bar.>",
+          "dest": "bar2.>"
+        }
+      ]
     }
   ],
   "subject_transform": {
@@ -122,6 +187,6 @@ E.g.
 ```
 ## Client implementation PRs
 
-- [jsm.go](https://github.com/nats-io/jsm.go/pull/436)
-- [nats.go](https://github.com/nats-io/nats.go/pull/1200)
-- [natscli](https://github.com/nats-io/natscli/pull/695)
+- [jsm.go](https://github.com/nats-io/jsm.go/pull/436) [and](https://github.com/nats-io/jsm.go/pull/461)
+- [nats.go](https://github.com/nats-io/nats.go/pull/1200) [and](https://github.com/nats-io/nats.go/pull/1359)
+- [natscli](https://github.com/nats-io/natscli/pull/695) [and](https://github.com/nats-io/natscli/pull/845)