How to reduce time to migrate data from rdb file to redis cluster ?

[dhruvil-alphonso]

问题描述（Issue Description）

Looking for ways optimize the calls and reduce time it takes to migrate data from rdb file to redis cluster. Right now, it is taking about an hour to ingest a 10GB rdb file into a cluster. I have only tuned the pipeline_count_limit = 4096, and can do it further to some large number, but would like to get suggestions on how i can reduce this to a significantly smaller number. Any help would be helpful. The command I am executing is ./redis-shake shake.toml
请在这里简要描述你遇到的问题。

Please provide a brief description of the issue you encountered.
Apart from the reader/writer config, this is the rest of toml file

[advanced]
dir = "data"
ncpu = 0 # runtime.GOMAXPROCS, 0 means use runtime.NumCPU() cpu cores
pprof_port = 0 # pprof port, 0 means disable
status_port = 0 # status port, 0 means disable

log_file = "shake.log"
log_level = "info" # debug, info or warn
log_interval = 5 # in seconds
rdb_restore_command_behavior = "panic"
pipeline_count_limit = 4096
target_redis_client_max_querybuf_len = 1024_000_000
target_redis_proto_max_bulk_len = 512_000_000
aws_psync = ""
[module]
target_mbbloom_version = 20603

环境信息（Environment）

• RedisShake 版本（RedisShake Version）：4.0.2
• Redis 源端版本（Redis Source Version）：rdb file
• Redis 目的端版本（Redis Destination Version）：6.x
• Redis 部署方式（standalone/cluster/sentinel）：destination is cluster
• 是否在云服务商实例上部署（Deployed on Cloud Provider）：no

日志信息（Logs）

如果有错误日志或其他相关日志，请在这里提供。

If there are any error logs or other relevant logs, please provide them here.

其他信息（Additional Information）

请提供任何其他相关的信息，如配置文件、错误信息或截图等。

Please provide any additional information, such as configuration files, error messages, or screenshots.

[suxb201]

你是否使用 function?

[dhruvil-alphonso]

In shake.toml function is function = "" So I guess I am not using this @suxb201

[suxb201]

源端是 standalone 的情况下没有办法进一步提高速度。但一小时的同步时间远超我的测试结果，请提供一下运行日志，可以帮助分析一下性能瓶颈。

[dhruvil-alphonso]

logs from the last time I ran with pipeline_count_limit as 8196. @suxb201

RedisShake$ ./bin/redis-shake shake.toml
2023-12-07 14:26:22 INF load config from file: shake.toml
2023-12-07 14:26:22 INF log_level: [info], log_file: [/path/shake.log]
2023-12-07 14:26:22 INF changed work dir to [/path/data]
2023-12-07 14:26:22 INF GOMAXPROCS defaults to the value of runtime.NumCPU [32]
2023-12-07 14:26:22 INF not set pprof port
2023-12-07 14:26:22 INF no function script
2023-12-07 14:26:22 INF create RdbReader: /path/dump.rdb
2023-12-07 14:26:22 INF redisClusterWriter connected to redis cluster successful. addresses=[redis cluster nodes]
2023-12-07 14:26:22 INF create RedisClusterWriter:
2023-12-07 14:26:22 INF not set status port
2023-12-07 14:26:22 INF start syncing...
2023-12-07 14:26:22 INF [rdb_reader] start read
2023-12-07 14:26:27 INF read_count=[286461], read_ops=[56226.66], write_count=[286460], write_ops=[56226.66], [rdb_reader] rdb file synced: 0.17%
2023-12-07 14:26:32 INF read_count=[573385], read_ops=[56251.08], write_count=[573384], write_ops=[56251.08], [rdb_reader] rdb file synced: 0.31%
2023-12-07 14:26:37 INF read_count=[852543], read_ops=[56520.40], write_count=[852542], write_ops=[56520.40], [rdb_reader] rdb file synced: 0.47%
2023-12-07 14:26:42 INF read_count=[1131470], read_ops=[55241.30], write_count=[1131469], write_ops=[55241.30], [rdb_reader] rdb file synced: 0.64%
2023-12-07 14:26:47 INF read_count=[1414674], read_ops=[55246.97], write_count=[1414673], write_ops=[55246.97], [rdb_reader] rdb file synced: 0.81%
2023-12-07 14:26:52 INF read_count=[1696260], read_ops=[56492.99], write_count=[1696259], write_ops=[56492.99], [rdb_reader] rdb file synced: 0.97%
2023-12-07 14:26:57 INF read_count=[1980843], read_ops=[55777.04], write_count=[1980842], write_ops=[55777.04], [rdb_reader] rdb file synced: 1.14%
2023-12-07 14:27:02 INF read_count=[2261153], read_ops=[57447.67], write_count=[2261152], write_ops=[57447.67], [rdb_reader] rdb file synced: 1.31%
2023-12-07 14:27:07 INF read_count=[2545045], read_ops=[56282.31], write_count=[2545044], write_ops=[56281.31], [rdb_reader] rdb file synced: 1.48%
2023-12-07 14:27:12 INF read_count=[2829023], read_ops=[57186.09], write_count=[2829022], write_ops=[57186.09], [rdb_reader] rdb file synced: 1.64%
2023-12-07 14:27:17 INF read_count=[3108041], read_ops=[55233.53], write_count=[3108040], write_ops=[55232.53], [rdb_reader] rdb file synced: 1.81%
2023-12-07 14:27:22 INF read_count=[3384648], read_ops=[53898.91], write_count=[3384647], write_ops=[53898.91], [rdb_reader] rdb file synced: 1.98%
.....
2023-12-07 15:14:57 INF read_count=[166563747], read_ops=[58274.82], write_count=[166563746], write_ops=[58273.82], [rdb_reader] rdb file synced: 98.69%
2023-12-07 15:15:02 INF read_count=[166855772], read_ops=[57376.40], write_count=[166855771], write_ops=[57376.40], [rdb_reader] rdb file synced: 98.83%
2023-12-07 15:15:07 INF read_count=[167146004], read_ops=[58237.04], write_count=[167146003], write_ops=[58238.04], [rdb_reader] rdb file synced: 99.00%
2023-12-07 15:15:12 INF read_count=[167441062], read_ops=[60856.30], write_count=[167441061], write_ops=[60856.30], [rdb_reader] rdb file synced: 99.17%
2023-12-07 15:15:17 INF read_count=[167732878], read_ops=[57633.37], write_count=[167732877], write_ops=[57633.37], [rdb_reader] rdb file synced: 99.35%
2023-12-07 15:15:22 INF read_count=[168027157], read_ops=[60803.77], write_count=[168027156], write_ops=[60803.77], [rdb_reader] rdb file synced: 99.52%
2023-12-07 15:15:27 INF read_count=[168324522], read_ops=[61424.57], write_count=[168324521], write_ops=[61423.57], [rdb_reader] rdb file synced: 99.70%
2023-12-07 15:15:32 INF read_count=[168621520], read_ops=[59541.77], write_count=[168621519], write_ops=[59542.77], [rdb_reader] rdb file synced: 99.87%
2023-12-07 15:15:34 INF [rdb_reader] rdb file parse done
2023-12-07 15:15:34 INF all done

[dhruvil-alphonso]

@suxb201 one more thing, I have observed is sometimes when I get errors like below
2023-12-10 14:08:11 ERR [writer_208.85.5.39_7002] redisStandaloneWriter received BUSYKEY reply.
It says redisStandaloneWriter and not RedisClusterWriter. Not sure if its relevant tbh, but letting you know anyways

[xishian]

源端是 standalone 的情况下没有办法进一步提高速度。但一小时的同步时间远超我的测试结果，请提供一下运行日志，可以帮助分析一下性能瓶颈。

@suxb201
那如果源端是redis cluster有办法提高同步速度吗，我有一个64个节点的redis集群，总共16亿个key，350GB内存，我要把它同步到另一个机房的redis集群（节点数80），之前尝试同步过一次，预估耗时十多个小时，就中断了

请问有哪些办法可以帮助我提高同步速度呢，比如分布式并发同步之类的？

谢谢

[suxb201]

@dhruvil-alphonso 速度是：read_ops=[56226.66]，已经很快了，你的 key 很多是吗？

@xishian 是的，现在只有一个 client 在写，只能提供 6 万左右的写能力。我后面打算改成多个 client，并按 key 并发来提高写能力。

[dhruvil-alphonso]

Yes, I have lots of keys @suxb201, around 144M or so . Looking forward to the multiple clients support. When do you plan to release that ?

[suxb201]

@dhruvil-alphonso 需要很多改动。本周或者下周。

[suxb201]

@dhruvil-alphonso 我认为我的修改不能提高你的同步速度。需要再进一步确认你的瓶颈在哪里。可能是从盘上读取文件较慢，可能是 dump.rdb 文件解析较慢，也可能是目的端消费命令较慢。

[suxb201]

@xishian 这个 PR 对于你的场景应该有用：#738