Jetstream "context deadline exceeded"

[jdupont22]

Observed behavior

I want to create a KV watcher resilient to no connection or lost connection.

I've configure the Nats connection with reconnect every 2s, retry on failed connect.

Work great if connection is lost after been connected.
But I have a "context deadline exceeded" if I launch the program with Nats server stopped, and after few seconds, started

Expected behavior

Have a KV watcher available when no connection (next connection succeed) or connection lost (next restore)

Server and client version

server => 2.10.11
client => 1.34.0

Host environment

server launch with Docker Compose in my local machine
client launch in my local machine

Steps to reproduce

Launch the following code with Nats server stopped, after few seconds, start the server

Gist

[piotrpio]

Hey @jdupont22, thank you for creating the issue - we will take a look at it shortly.

[piotrpio]

@jdupont22 I could not reproduce the problem using your gist, because if you don't connect initially, you'll never even attempt to get the KV right?

	if (n.conn == nil) || !n.conn.IsConnected() {
		fmt.Println("AfterConnect | Skip NATs, not connected")
		return
	}

So there is no way to get to the panic in your gist unless we are actually connected, unless I'm not executing something correctly. However, I get what you are encountering and I have a suggestion.

You're doing the right thing re-creating the connection in ReconnectHandler. In order to achieve your goal, you can use the same logic which you have in this handler and put it in ConnectHandler as well, which would be invoked after the connection is established in Connect() (it's especially useful when combined with RetryOnFailedConnect(). So, instead of calling afterConnect() immediately after nats.Connect() (which may fail as you don't know whether the conn is established or not at this point), you could drop it and only establish connection in the two aforementioned handlers.

We recently fixed a bug where ConnectHandler would not be invoked if the initial connection failed but future attempt succeeds: #1619 - it's not released yet, but will be a part of next release.

After adjustments, this is what your main function could look like:

func main() {
	ctx, _ := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)

	n := natsBroker{
		ctx: &ctx,
	}
	connEstablishedHandler := func(nc *nats.Conn) {
		fmt.Printf("Got connected to %v!\n", nc.ConnectedUrl())
		n.afterConnect()
	}

	nc, err := nats.Connect(nats.DefaultURL, nats.RetryOnFailedConnect(true),
		nats.MaxReconnects(-1),
		nats.ReconnectHandler(connEstablishedHandler),
		nats.ConnectHandler(connEstablishedHandler),
	)
	if err != nil {
		fmt.Println("Nats err ", err)
		return
	}

	n.conn = nc

	for {
		select {
		case <-ctx.Done():
			return
		default:
			time.Sleep(5 * time.Second)
		}
	}
}

Let me know if that helps.

[wallyqs]

Do you create the KV with R=1? Those look like the errors you would get when the server with the stream is down.