Can the Key-Value Store synchronize info between the cluster and the leafnode

[ldmid666]

the Key Value Store is built on top of the JetStream persistence layer., can synchronize info between the super cluster.

But I found Key-Value Store Bucket can not synchronize info between the cluster and the leafnode .
And execute the command "nats stream info" ,can not find the stream "KV_icn-bucket".

So , the Key-Value Store can not synchronize info between the cluster and the leafnode ? Or I configured it wrong ?
And why cannot synchronize info ?

[derekcollison]

Leafnodes will need to be a separate JetStream domain generally since they separate operator and security domains. If you control both the cluster and the leafnode you could configure to extend the cluster JetStream domain via also having the leafnode connect to the system account on the cluster.

Either way you can create a mirror on the leafnode that will sync the cluster KV, you just need to override the external API using the domain name from the cluster when creating.

We plan on adding in a bit more convenience to the client apis and NATS cli to make this easier.

[ldmid666]

Thanks,but execute the command 'nats kv add my_kv --creds ./my.creds' on cluster , respond

on my leafnode
nats stream add --mirror KV_my_kv --js-domain leafnodehome --creds ./my.creds
success respond

 ? Stream Name KV_my_kv
? Storage file
? Replication 1
? Retention Policy Limits
? Discard Policy Old
? Stream Messages Limit -1
? Total Stream Size -1
? Message TTL -1
? Max Message Size -1
? Allow message Roll-ups No
? Allow message deletion Yes
? Allow purging subjects or the entire stream Yes
? Adjust mirror start No
? Import mirror from a different JetStream domain Yes
? Foreign JetStream domain name mainbj
? Delivery prefix
Stream KV_my_kv was created

Information for Stream KV_my_kv created 2022-10-15 13:51:09

             Replicas: 1
              Storage: File

Options:

            Retention: Limits
     Acknowledgements: true
       Discard Policy: Old
     Duplicate Window: 0s
    Allows Msg Delete: true
         Allows Purge: true
       Allows Rollups: false

Limits:

     Maximum Messages: unlimited
  Maximum Per Subject: unlimited
        Maximum Bytes: unlimited
          Maximum Age: unlimited
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited

Limits:

               Mirror: KV_my_kv, API Prefix: $JS.mainbj.API, Delivery Prefix:


State:

             Messages: 0
                Bytes: 0 B
             FirstSeq: 0
              LastSeq: 0
     Active Consumers: 0

I execute 'nats kv put my_kv key1 val1 --creds ./my.creds ' success on cluster
but execute 'nats kv get my_kv key1' on leafnode get errors

nats: error: nats: bucket not valid key-value store

Maybe my configuration is wrong, and what is the correct operation? Thanks for reply

[derekcollison]

Will try to take a closer look myself.

[derekcollison]

ok it works, but is hard to setup correctly. Are you using the NATS cli?

[ldmid666]

ok it works, but is hard to setup correctly. Are you using the NATS cli?

Yes，I'm verifying functionality from the NATS cli. After that, I will maybe use golang.
How do you configure it ？

[derekcollison]

I could not properly with the NATS cli either, so will be adding this capability tomorrow.

To get it to work I had to create a custom app.

So say that the KeyValue bucket name is TEST, on domain ngs, and you want to create MIRROR.

Load stream info for KV_TEST, modify the stream config as follows.

Change name to KV_MIRROR, remove Subjects property, and add in a Mirror property as follows.

	Mirror: &nats.StreamSource{
			Name: "KV_TEST",
			External: &nats.ExternalStream{
				APIPrefix: "$JS.ngs.API",
			},
		},

Or wait for the update to the NATS cli which will make this trivial.

[ldmid666]

I could not properly with the NATS cli either, so will be adding this capability tomorrow.

To get it to work I had to create a custom app.

So say that the KeyValue bucket name is TEST, on domain ngs, and you want to create MIRROR.

Load stream info for KV_TEST, modify the stream config as follows.

Change name to KV_MIRROR, remove Subjects property, and add in a Mirror property as follows.

	Mirror: &nats.StreamSource{
			Name: "KV_TEST",
			External: &nats.ExternalStream{
				APIPrefix: "$JS.ngs.API",
			},
		},

Or wait for the update to the NATS cli which will make this trivial.

OK，Thanks. It looks a little hard, I will wait for the update to the NATS cli .

[ripienaar]

For me doing nats s add KV_MIRROR --source KV_X and then later when prompted:

? Adjust mirror start No
? Import mirror from a different JetStream domain Yes
? Foreign JetStream domain name NATS_DEVELOPMENT

Just works. For NGS you'd just say ngs as domain.

[derekcollison]

Although we will be adding an easier way to the cli, @ripienaar is correct, I must have not properly placed the KV_ prefix into the cli command. Apologies for that.

[derekcollison]

The PR for the Go client has landed, next will be to get it into the CLI.

nats-io/nats.go#1112

[ldmid666]

The PR for the Go client has landed, next will be to get it into the CLI.

nats-io/nats.go#1112

nice

[bpatel85]

I tried this out but things are not working as expected. Here are my repro steps:

• Create some buckets

nats kv add us1_av_store
nats kv add us2_av_store
nats kv add us3_av_store

• Add new global_av_state KV Bucket with go client

bucketNames := []string{"us1_av_store", "us2_av_store", "us3_av_store"}
    sources := make([]*nats.StreamSource, 0)
	for _, streamName := range bucketNames {
		sources = append(sources, &nats.StreamSource{
			Name:        streamName,
			OptStartSeq: 0,
		})
	}
    // get the jsCtx 
    kv, err := jsCtx.CreateKeyValue(&nats.KeyValueConfig{
		Bucket:   "global_av_state",
		Sources:  sources,
		Storage:  nats.FileStorage,
		Replicas: 3,
	})

• Publish messages nats kv put us1_av_store foo bar. NTAS immidiately syncs and message is available in global_av_state bucket
  

But when I try to read key from global_av_state I get key not found error. I was expecting the value bar to be returned. I believe that is because the subjects are kept intact when sourcing as we can see below:

~ # nats kv get global_av_store foo
nats: error: nats: key not found

[derekcollison]

KV layer does subject mapping and so the prefix is different. We are looking into it, but if you are mirroring or sourcing into a different domain or account using the same name will help.

[derekcollison]

Closing for now but feel free to re-open as needed.