Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

baseport for content cluster nodes doesn't work well for all node types #31145

Open
kawatea opened this issue May 8, 2024 · 0 comments
Open
Assignees
Milestone

Comments

@kawatea
Copy link

kawatea commented May 8, 2024

Describe the bug
When we set baseport for content cluster nodes, it works for storagenode but doesn't work well for distributor/searchnode.
distributor-base-port also doesn't work at all.

To Reproduce
Steps to reproduce the behavior:

  1. Deploy one content cluster with baseport
  <content id="test2" version="1.0">
    <redundancy>1</redundancy>
    <documents>
      <document type="test2" mode="index" />
    </documents>
    <nodes>
      <node hostalias="node1" distribution-key="0" baseport="20000"/>
    </nodes>
  </content>
  1. Add another content cluster before the existing cluster
  <content id="test1" version="1.0">
    <redundancy>1</redundancy>
    <documents>
      <document type="test1" mode="index" />
    </documents>
    <nodes>
      <node hostalias="node1" distribution-key="0" baseport="10000"/>
    </nodes>
  </content>

  <content id="test2" version="1.0">
    <redundancy>1</redundancy>
    <documents>
      <document type="test2" mode="index" />
    </documents>
    <nodes>
      <node hostalias="node1" distribution-key="0" baseport="20000"/>
    </nodes>
  </content>
  1. See warnings and restart messages
WARNING distributor cannot reserve port 10000 on vespa-container: Already reserved for storagenode. Using default port range from 19109
WARNING distributor cannot reserve port 10001 on vespa-container: Already reserved for storagenode. Using default port range from 19110
WARNING distributor cannot reserve port 10002 on vespa-container: Already reserved for storagenode. Using default port range from 19111
WARNING distributor2 cannot reserve port 20000 on vespa-container: Already reserved for storagenode2. Using default port range from 19120
WARNING distributor2 cannot reserve port 20001 on vespa-container: Already reserved for storagenode2. Using default port range from 19121
WARNING distributor2 cannot reserve port 20002 on vespa-container: Already reserved for storagenode2. Using default port range from 19122
WARNING Change(s) between active and new application that require restart:
In cluster 'test2' of type 'content':
    Restart services of type 'distributor' because:
        1) stor-communicationmanager.mbusport has changed from 19109 to 19120
stor-communicationmanager.rpcport has changed from 19110 to 19121
stor-status.httpport has changed from 19111 to 19122
In cluster 'test2' of type 'search':
    Restart services of type 'searchnode' because:
        1) # Port to use for the rpcserver.
proton.rpcport has changed from 19103 to 19114
# Port to use for the web server
proton.httpport has changed from 19107 to 19118
# Connect spec for transactionlog server.
# TODO Consider not using RPC at all
proton.tlsspec has changed from "tcp/vespa-container:19108" to "tcp/vespa-container:19119"
# Port number to use for listening.
translogserver.listenport has changed from 19108 to 19119

Expected behavior
baseport setting should work properly for distributor/searchnode and we should be able to avoid restarting services.

Environment (please complete the following information):

  • OS: Ubuntu
  • Infrastructure: self-hosted (observed the same behavior on Kubernetes)
  • Versions 18.04

Vespa version
8.339.15

Additional context
We want to avoid restarting unrelated distributor/storagenode/searchnode when adding/removing/updating content clusters because it causes outage for a while.
baseport setting works for storagenode but doesn't work well for distributor (distributor-base-port also doesn't work) and doesn't affect searchnode.

@kkraune kkraune added this to the later milestone May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants