jetstream.PublishMsg is not guaranteed to return ErrNoStreamResponse when there is no stream configured

[kung-foo]

Observed behavior

jetstream.PublishMsg is not guaranteed to return ErrNoStreamResponse when there is no stream configured.

https://github.com/nats-io/nats.go/blob/v1.31.0/jetstream/publish.go#L189-L191

The client seems to rely on ErrNoResponders to indicate that "no response from stream". However, if another consumer is doing a simple subscribe on the same subject (i.e. nc.Subscribe(...) then the server doesn't respond with ErrNoResponders.

Since ErrNoResponders is in the critical path for retry logic, this also fails since there is no other indication that something failed.

This became an issue when writing tests for our use of the new JetStream API. One of the tests attempts to handle the case where a stream was not configured for before publication of messages. However, I couldn't make this error bubble up. Turns out, the test harness also has a simple nc.Subscribe("test.>") to track expected messages counts and sometime content. This "outofband" tracking kept the ErrNoResponders from being sent. Even though this was just a unit test, there are cases in production where this could happen. For example we sometimes use a simple subscribe with only headers as a parallel audit capture log.

Expected behavior

I expected jetstream.PublishMsg to return ErrNoStreamResponse if there was no JetStream configured to consume the subject.

Server and client version

client: v1.31.0
server: v2.10.3

Host environment

No response

Steps to reproduce

No response

[piotrpio]

Hello @kung-foo, thanks for reporting the issue. I'm looking at this (and #1462) and will update you when I have a fix.

[piotrpio]

Unfortunately this issue is not something that can be solved client-side.

When publishing a message to a stream, the client is essentially not aware whether there is an underlying stream or not. It simply publishes a standard NATS request (message with reply subject) on a stream's subject and waits for the response. If there is another non-jetstream subscription listening on this subject (and the stream does not exist), this will end with a timeout. The server is also not aware whether the request is jetstream specific or not, since this is a standard NATS message. To know for sure whether there is a stream with interest on the publish subject, we would have to send a separate JetStream request to get the streams, which is not something we want to do on each publish (or even after the timeout is hit).

My suggestion would be to not use the same subject for both stream and standard NATS subscription. If you want a separate subscription to just do the logging, I would suggest creating a simple ephemeral consumer with AckPolicy set to AckNone and possibly DeliverPolicy set to DeliverNewPolicy (you can use stream.OrderedConsumer() for simplicity).

[kung-foo]

I kinda figured.
There is the ExpectedStreamHeader option which implies that you can communicate something about JetStream even though it is "just a NATS message".
I think the docs should indicate the limitations of the error response for this API. My reading of the existence of a "retry" option is that any error during publication will trigger a retry. However, it is only an ErrNoResponders error where a retry is attempted. And since that error is somewhat unreliable, it makes it hard to reason about what will and will not happen.

[piotrpio]

Agreed, docs can be improved. ExpectedStreamHeader is there, but it's optional and is handled within the context of the stream itself, before sending ACK to the client. It is there to make sure that the subject you publish on belongs to the stream you expect. It will not be treated in a special way by the server if the stream for the given subject does not exist.

[jnmoyne]

There are 3 ways a JS publish can fail and you must catch and log them all:

1. No responders
2. Timeout
3. Error returned by the server (e.g. some limit is hit).

The first 2 cases are equivalent from the client application's perspective: either wait a few seconds and try again or give up.

Will add something about this in the docs.