Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow setting consumer replicas though options #1019

Merged
merged 8 commits into from Jul 29, 2022

Conversation

goku321
Copy link
Contributor

@goku321 goku321 commented Jul 22, 2022

Currently, only js.AddConsumer supports configuring replicas count for a consumer. Configuring it through nats.SubOpt will allow configuring the same with js.Subscribe, js.PullSubscribe etc.

@coveralls
Copy link

coveralls commented Jul 22, 2022

Coverage Status

Coverage decreased (-0.005%) to 84.08% when pulling 1d9feeb on goku321:opt-cons-replica into f4a86f3 on nats-io:main.

@piotrpio
Copy link
Collaborator

Hey, this looks good, I just have 2 suggestions:

  1. Could you add a simple check in checkConfig() func in js.go to verify whether the value is not changed if the consumer already exists (this is done for other settable parameters as well to avoid overwriting existing config in Subscribe()
  2. It would be nice to have at least a simple test in test/js_test.go to verify whether a consumer created with Subscribe() has a proper Replicas value set in config.

// ConsumerReplicas sets the number of replica count for a consumer.
func ConsumerReplicas(replicas int) SubOpt {
return subOptFn(func(opts *subOpts) error {
if replicas < 1 {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can also consider setting adding && replicas <=5, as that is the upper limit.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is the current server limit, but since server defined and we could change would leave that out..

@kozlovic
Copy link
Member

@piotrpio @derekcollison @tbeets I may be mistaken, but the approach of the JS simplification is that we would no longer create JS consumers, so not sure if it is advisable to add those sub options that would no longer be needed after the upcoming simplification?

@piotrpio
Copy link
Collaborator

piotrpio commented Jul 22, 2022

@kozlovic Simplified JestStream API will be in it's own package, so the current implementation can be modified separately.
You're right, the simplified API will not allow creating consumers at the time of subscribing, so there will be no need for adding those options.
Here is the draft of js-simplification - #1018

Copy link
Member

@kozlovic kozlovic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@piotrpio Ok, then I concur with your ask to add the replicas check in checkConfig()

@goku321 goku321 requested a review from piotrpio July 28, 2022 12:13
js_test.go Outdated
}

// Check if the number of replicas is the same as we provided.
if consInfo.Config.Replicas != 1 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be fair, this test does not prove anything since even if ConsumerReplicas() was doing nothing, this test would pass, simply because the stream itself is replicas 1, so the new consumer would inherit this value. So the test should either be moved to test/ directory where we may have some utilities to create a cluster and in that case the stream should be created with replicas 3 and consumer with 1, and in that case this test here is valid, or, before the js.Subscribe() call, create a NATS subscription on the durable create request and check that the request contains the replicas field. Check test func TestJetStreamConsumerConfigReplicasAndMemStorage() to see how I am doing this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kozlovic I totally agree on this. Will make the changes

@goku321 goku321 requested a review from kozlovic July 29, 2022 12:19
Copy link
Collaborator

@piotrpio piotrpio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! It would be great if you could squash your commits before we'll merge.

Copy link
Member

@kozlovic kozlovic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check formatting (go fmt) since that failed the checks.

@kozlovic
Copy link
Member

@goku321 As @piotrpio suggested, it would be good if you did squash all your commits into one. Could you do that please?

@goku321
Copy link
Contributor Author

goku321 commented Jul 29, 2022

@kozlovic I merged main into my branch to resolve conflicts. Is it possible to squash all the commits now?

@kozlovic
Copy link
Member

@goku321 Wished you had done a rebase instead of a merge, would have been easier for you to squash all the commits. I guess we can use the "Squash and Merge" in the drop-down list. Not sure, @wallyqs or @piotrpio do you know if that would work well?

@wallyqs
Copy link
Member

wallyqs commented Jul 29, 2022

sounds good to squash and merge keeping the title from the first commit via GitHub UI

@kozlovic kozlovic merged commit fcc7c44 into nats-io:main Jul 29, 2022
@goku321 goku321 deleted the opt-cons-replica branch July 29, 2022 18:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants