Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow setting consumer replicas though options #1019

Merged
merged 8 commits into from Jul 29, 2022
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
5 changes: 5 additions & 0 deletions example_test.go
Expand Up @@ -625,6 +625,11 @@ func ExampleSubOpt() {
js.Subscribe("foo", func(msg *nats.Msg) {
fmt.Printf("Received a message: %s\n", string(msg.Data))
}, nats.ManualAck(), nats.MaxDeliver(2), nats.BackOff([]time.Duration{50 * time.Millisecond, 250 * time.Millisecond}))

// Set consumer replicas count for a durable while subscribing.
js.Subscribe("foo", func(msg *nats.Msg) {
fmt.Printf("Received a message: %s\n", string(msg.Data))
}, nats.Durable("FOO"), nats.ConsumerReplicas(1))
}

func ExampleMaxWait() {
Expand Down
14 changes: 14 additions & 0 deletions js.go
Expand Up @@ -1299,6 +1299,9 @@ func checkConfig(s, u *ConsumerConfig) error {
if u.Heartbeat > 0 && u.Heartbeat != s.Heartbeat {
return makeErr("heartbeat", u.Heartbeat, s.Heartbeat)
}
if u.Replicas > 0 && u.Replicas != s.Replicas {
return makeErr("replicas", u.Replicas, s.Replicas)
}
return nil
}

Expand Down Expand Up @@ -2413,6 +2416,17 @@ func InactiveThreshold(threshold time.Duration) SubOpt {
})
}

// ConsumerReplicas sets the number of replica count for a consumer.
func ConsumerReplicas(replicas int) SubOpt {
return subOptFn(func(opts *subOpts) error {
if replicas < 1 {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can also consider setting adding && replicas <=5, as that is the upper limit.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is the current server limit, but since server defined and we could change would leave that out..

return fmt.Errorf("invalid ConsumerReplicas value (%v), needs to be greater than 0", replicas)
}
opts.cfg.Replicas = replicas
return nil
})
}

func (sub *Subscription) ConsumerInfo() (*ConsumerInfo, error) {
sub.mu.Lock()
// TODO(dlc) - Better way to mark especially if we attach.
Expand Down
30 changes: 30 additions & 0 deletions js_test.go
Expand Up @@ -1026,3 +1026,33 @@ func TestJetStreamClusterPlacement(t *testing.T) {
t.Fatalf("Unexpected tag: %q", v)
}
}

func TestConsumerReplicasOption(t *testing.T) {
s := RunBasicJetStreamServer()
defer shutdownJSServerAndRemoveStorage(t, s)

nc, js := jsClient(t, s)
defer nc.Close()

if _, err := js.AddStream(&StreamConfig{Name: "CONSUMER_REPLICAS_TEST", Subjects: []string{"foo"}}); err != nil {
t.Fatalf("Error adding stream: %v", err)
}

cb := func(msg *Msg) {}
// Subscribe to the stream with a durable consumer "bar".
_, err := js.Subscribe("foo", cb, Durable("bar"), ConsumerReplicas(1))
if err != nil {
t.Fatalf("Unexpected error: %v", err)
}

// Get consumer info
consInfo, err := js.ConsumerInfo("CONSUMER_REPLICAS_TEST", "bar")
if err != nil {
t.Fatalf("Unexpected error: %v", err)
}

// Check if the number of replicas is the same as we provided.
if consInfo.Config.Replicas != 1 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be fair, this test does not prove anything since even if ConsumerReplicas() was doing nothing, this test would pass, simply because the stream itself is replicas 1, so the new consumer would inherit this value. So the test should either be moved to test/ directory where we may have some utilities to create a cluster and in that case the stream should be created with replicas 3 and consumer with 1, and in that case this test here is valid, or, before the js.Subscribe() call, create a NATS subscription on the durable create request and check that the request contains the replicas field. Check test func TestJetStreamConsumerConfigReplicasAndMemStorage() to see how I am doing this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kozlovic I totally agree on this. Will make the changes

t.Fatalf("Expected consumer replica to be %v, got %+v", 1, consInfo.Config.Replicas)
}
}