Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kv watcher always fails on buckets with lots of KVs #1431

Open
HeavyHorst opened this issue Oct 5, 2023 · 1 comment · May be fixed by #1432
Open

kv watcher always fails on buckets with lots of KVs #1431

HeavyHorst opened this issue Oct 5, 2023 · 1 comment · May be fixed by #1432
Assignees
Labels
defect Suspected defect such as a bug or regression

Comments

@HeavyHorst
Copy link

What version were you using?

v1.30.2

What environment was the server running in?

Linux pop-os 6.4.6-76060406-generic #202307241739169531258622.04~4d8f3e9 SMP PREEMPT_DYNAMIC Thu S x86_64 x86_64 x86_64 GNU/Linux

Is this defect reproducible?

Create lot of Key-Value pairs.
Try to print all keys and values with an watcher.

package main

import (
	"context"
	"fmt"
	"time"

	"github.com/nats-io/nats.go"
	"github.com/nats-io/nats.go/jetstream"
)

func main() {
	nc, err := nats.Connect(nats.DefaultURL)
	if err != nil {
		panic(err)
	}

	js, _ := jetstream.New(nc)

	kv, err := js.KeyValue(context.Background(), "data")
	if err != nil {
		panic(err)
	}

	context, _ := context.WithTimeout(context.Background(), 10*time.Minute)

	start := time.Now()
	watcher, err := kv.Watch(context, ">", jetstream.IgnoreDeletes())
	if err != nil {
		panic(err)
	}
	defer watcher.Stop()

	for entry := range watcher.Updates() {
		if entry == nil {
			break
		}

		fmt.Println(string(entry.Value()))
	}
	fmt.Println(time.Since(start))
}

Here is a demo snapshot of a KV stream with ~3million kVs that can be used for testing:

nats://demo.nats.io:4222

rkaufmann@pop-os:~$ nats --context demo object ls discussions_957 
╭──────────────────────────────────────────────────────╮
│                   Bucket Contents                    │
├─────────────────┬────────┬───────────────────────────┤
│ Name            │ Size   │ Time                      │
├─────────────────┼────────┼───────────────────────────┤
│ snapshot.tar.xz │ 62 MiB │ 2023-04-17T16:14:18+02:00 │
│ demo.go         │ 611 B  │ 2023-04-17T16:17:24+02:00 │
╰─────────────────┴────────┴───────────────────────────╯

Given the capability you are leveraging, describe your expectation?

I created a context with a Timeout of 10 minutes so i should be able to create the watcher if it does not take longer then 10 minutes.

Given the expectation, what is the defect you are observing?

After 5 seconds the program stops with "context deadline exceeded".

@HeavyHorst HeavyHorst added the defect Suspected defect such as a bug or regression label Oct 5, 2023
@HeavyHorst HeavyHorst linked a pull request Oct 5, 2023 that will close this issue
@piotrpio piotrpio self-assigned this Oct 12, 2023
@piotrpio
Copy link
Collaborator

Hey @HeavyHorst thanks for the issue and the PR, I'll be reviewing it shortly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
defect Suspected defect such as a bug or regression
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants