Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

同步数据较慢 #799

Closed
cataglyphis opened this issue May 15, 2024 · 3 comments
Closed

同步数据较慢 #799

cataglyphis opened this issue May 15, 2024 · 3 comments
Labels
type: question Further information is requested

Comments

@cataglyphis
Copy link

cataglyphis commented May 15, 2024 •
edited
Loading

问题描述（Issue Description）

同步自建的 redis cluster 数据到 AWS ElastiCache，redis 是 3 主 3 从的配置。
有 2 个节点的 Key 数量在 1200W 左右，有 1 个在 100W 左右

环境信息（Environment）

  • RedisShake 版本（RedisShake Version）： 4.0.5
  • Redis 源端版本（Redis Source Version）：6.2
  • Redis 目的端版本（Redis Destination Version）：6.2
  • Redis 部署方式（standalone/cluster/sentinel）：cluster
  • 是否在云服务商实例上部署（Deployed on Cloud Provider）：否

日志信息（Logs）

2024-05-15 06:57:24 INF create RedisClusterWriter: clustercfg.redis-cluster-prod-us.xnwejy.usw1.cache.amazonaws.com:6379
2024-05-15 06:57:24 INF not set status port
2024-05-15 06:57:24 INF start syncing...
2024-05-15 06:57:29 INF read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-1, waiting bgsave
2024-05-15 06:57:34 INF read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-2, waiting bgsave
2024-05-15 06:57:39 INF read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-0, receiving rdb
2024-05-15 06:57:44 INF read_count=[49704], read_ops=[49717.07], write_count=[49703], write_ops=[49716.07], src-1, waiting bgsave
2024-05-15 06:57:49 INF read_count=[350387], read_ops=[58625.47], write_count=[350386], write_ops=[58625.47], src-2, waiting bgsave
2024-05-15 06:57:54 INF read_count=[654089], read_ops=[60445.45], write_count=[654088], write_ops=[60445.45], src-0, syncing rdb, size=[3.9 MiB/492 MiB]
2024-05-15 06:57:59 INF read_count=[959948], read_ops=[61315.85], write_count=[959947], write_ops=[61315.85], src-1, waiting bgsave
2024-05-15 06:58:04 INF read_count=[1259413], read_ops=[59568.26], write_count=[1259412], write_ops=[59569.26], src-2, waiting bgsave
2024-05-15 06:58:09 INF read_count=[1564152], read_ops=[61100.45], write_count=[1564151], write_ops=[61100.45], src-0, syncing rdb, size=[9.6 MiB/492 MiB]
2024-05-15 06:58:14 INF read_count=[1874758], read_ops=[62724.37], write_count=[1874757], write_ops=[62724.37], src-1, waiting bgsave
2024-05-15 06:58:19 INF read_count=[2177452], read_ops=[58886.78], write_count=[2177451], write_ops=[58886.78], src-2, waiting bgsave
2024-05-15 06:58:24 INF read_count=[2485751], read_ops=[63225.79], write_count=[2485750], write_ops=[63224.79], src-0, syncing rdb, size=[16 MiB/492 MiB]
2024-05-15 06:58:29 INF read_count=[2793423], read_ops=[61891.68], write_count=[2793423], write_ops=[61891.68], src-1, waiting bgsave
2024-05-15 06:58:34 INF read_count=[3099020], read_ops=[59701.81], write_count=[3099020], write_ops=[59700.81], src-2, waiting bgsave
2024-05-15 06:58:39 INF read_count=[3400906], read_ops=[58369.00], write_count=[3400905], write_ops=[58369.00], src-0, syncing rdb, size=[21 MiB/492 MiB]
2024-05-15 06:58:44 INF read_count=[3707526], read_ops=[62169.54], write_count=[3707525], write_ops=[62169.54], src-1, waiting bgsave
2024-05-15 06:58:49 INF read_count=[4012763], read_ops=[59020.97], write_count=[4012762], write_ops=[59019.97], src-2, waiting bgsave
2024-05-15 06:58:54 INF read_count=[4316515], read_ops=[60600.20], write_count=[4316514], write_ops=[60600.20], src-0, syncing rdb, size=[27 MiB/492 MiB]
2024-05-15 06:58:59 INF read_count=[4619623], read_ops=[60846.64], write_count=[4619622], write_ops=[60846.64], src-1, waiting bgsave
2024-05-15 06:59:04 INF read_count=[4919546], read_ops=[59948.93], write_count=[4919545], write_ops=[59948.93], src-2, waiting bgsave
2024-05-15 06:59:09 INF read_count=[5213744], read_ops=[61345.92], write_count=[5213743], write_ops=[61345.92], src-0, syncing rdb, size=[33 MiB/492 MiB]
2024-05-15 06:59:14 INF read_count=[5523272], read_ops=[62902.64], write_count=[5523272], write_ops=[62903.64], src-1, receiving rdb
2024-05-15 06:59:19 INF read_count=[5826432], read_ops=[59643.55], write_count=[5826431], write_ops=[59642.55], src-2, receiving rdb
2024-05-15 06:59:24 INF read_count=[6129314], read_ops=[59554.78], write_count=[6129314], write_ops=[59554.78], src-0, syncing rdb, size=[39 MiB/492 MiB]
2024-05-15 06:59:29 INF read_count=[6437981], read_ops=[62391.95], write_count=[6437981], write_ops=[62391.95], src-1, receiving rdb
2024-05-15 06:59:34 INF read_count=[6742262], read_ops=[57873.70], write_count=[6742261], write_ops=[57873.70], src-2, receiving rdb
2024-05-15 06:59:39 INF read_count=[7053437], read_ops=[64293.44], write_count=[7053436], write_ops=[64293.44], src-0, syncing rdb, size=[45 MiB/492 MiB]
2024-05-15 06:59:44 INF read_count=[7353244], read_ops=[58878.97], write_count=[7353243], write_ops=[58878.97], src-1, receiving rdb
2024-05-15 06:59:49 INF read_count=[7660840], read_ops=[61090.99], write_count=[7660839], write_ops=[61091.99], src-2, receiving rdb
2024-05-15 06:59:54 INF read_count=[7965915], read_ops=[63200.30], write_count=[7965915], write_ops=[63200.30], src-0, syncing rdb, size=[51 MiB/492 MiB]
2024-05-15 06:59:59 INF read_count=[8273314], read_ops=[63063.86], write_count=[8273314], write_ops=[63063.86], src-1, receiving rdb
2024-05-15 07:00:04 INF read_count=[8577411], read_ops=[60231.23], write_count=[8577410], write_ops=[60230.23], src-2, receiving rdb
2024-05-15 07:00:09 INF read_count=[8862772], read_ops=[63702.26], write_count=[8862771], write_ops=[63701.26], src-0, syncing rdb, size=[56 MiB/492 MiB]
2024-05-15 07:00:14 INF read_count=[9172598], read_ops=[61363.63], write_count=[9172598], write_ops=[61364.63], src-1, receiving rdb
2024-05-15 07:00:19 INF read_count=[9480032], read_ops=[62779.04], write_count=[9480031], write_ops=[62779.04], src-2, receiving rdb
2024-05-15 07:00:24 INF read_count=[9790131], read_ops=[63962.91], write_count=[9790130], write_ops=[63962.91], src-0, syncing rdb, size=[62 MiB/492 MiB]
2024-05-15 07:00:29 INF read_count=[10100890], read_ops=[62255.06], write_count=[10100889], write_ops=[62255.06], src-1, receiving rdb
2024-05-15 07:00:34 INF read_count=[10405061], read_ops=[59681.40], write_count=[10405060], write_ops=[59681.40], src-2, receiving rdb
2024-05-15 07:00:39 INF read_count=[10713382], read_ops=[61506.67], write_count=[10713381], write_ops=[61507.67], src-0, syncing rdb, size=[68 MiB/492 MiB]
2024-05-15 07:00:44 INF read_count=[11012457], read_ops=[58501.53], write_count=[11012456], write_ops=[58500.53], src-1, receiving rdb
2024-05-15 07:00:49 INF read_count=[11321879], read_ops=[61335.88], write_count=[11321878], write_ops=[61335.88], src-2, receiving rdb
2024-05-15 07:00:54 INF read_count=[11626385], read_ops=[61502.21], write_count=[11626384], write_ops=[61502.21], src-0, syncing rdb, size=[74 MiB/492 MiB]
2024-05-15 07:00:59 INF read_count=[11935964], read_ops=[61171.01], write_count=[11935963], write_ops=[61172.01], src-1, receiving rdb
2024-05-15 07:01:04 INF read_count=[12241517], read_ops=[59477.20], write_count=[12241516], write_ops=[59477.20], src-2, receiving rdb
2024-05-15 07:01:09 INF read_count=[12530486], read_ops=[56598.57], write_count=[12530485], write_ops=[56598.57], src-0, syncing rdb, size=[79 MiB/492 MiB]
2024-05-15 07:01:14 INF read_count=[12818118], read_ops=[56980.89], write_count=[12818118], write_ops=[56980.89], src-1, syncing rdb, size=[1.6 MiB/5.4 GiB]
2024-05-15 07:01:19 INF read_count=[13081142], read_ops=[48202.00], write_count=[13081141], write_ops=[48202.00], src-2, syncing rdb, size=[368 KiB/5.4 GiB]
2024-05-15 07:01:24 INF read_count=[13336656], read_ops=[52202.20], write_count=[13336655], write_ops=[52202.20], src-0, syncing rdb, size=[81 MiB/492 MiB]
2024-05-15 07:01:29 INF read_count=[13592171], read_ops=[52341.70], write_count=[13592170], write_ops=[52341.70], src-1, syncing rdb, size=[1.6 MiB/5.4 GiB]
2024-05-15 07:01:34 INF read_count=[13852056], read_ops=[52168.14], write_count=[13852055], write_ops=[52167.14], src-2, syncing rdb, size=[2.5 MiB/5.4 GiB]
2024-05-15 07:01:39 INF read_count=[14114527], read_ops=[52949.90], write_count=[14114526], write_ops=[52949.90], src-0, syncing rdb, size=[83 MiB/492 MiB]
2024-05-15 07:01:44 INF read_count=[14366021], read_ops=[49612.24], write_count=[14366021], write_ops=[49611.24], src-1, syncing rdb, size=[1.6 MiB/5.4 GiB]
2024-05-15 07:01:49 INF read_count=[14625717], read_ops=[52057.25], write_count=[14625716], write_ops=[52056.25], src-2, syncing rdb, size=[4.0 MiB/5.4 GiB]
2024-05-15 07:01:54 INF read_count=[14878159], read_ops=[50054.16], write_count=[14878158], write_ops=[50054.16], src-0, syncing rdb, size=[84 MiB/492 MiB]
2024-05-15 07:01:59 INF read_count=[15139508], read_ops=[53386.96], write_count=[15139507], write_ops=[53386.96], src-1, syncing rdb, size=[1.6 MiB/5.4 GiB]
2024-05-15 07:02:04 INF read_count=[15395287], read_ops=[51361.67], write_count=[15395286], write_ops=[51361.67], src-2, syncing rdb, size=[6.1 MiB/5.4 GiB]
2024-05-15 07:02:09 INF read_count=[15657893], read_ops=[52952.58], write_count=[15657893], write_ops=[52952.58], src-0, syncing rdb, size=[86 MiB/492 MiB]
2024-05-15 07:02:14 INF read_count=[15918024], read_ops=[51670.44], write_count=[15918023], write_ops=[51670.44], src-1, syncing rdb, size=[1.6 MiB/5.4 GiB]
2024-05-15 07:02:19 INF read_count=[16173454], read_ops=[49914.77], write_count=[16173453], write_ops=[49914.77], src-2, syncing rdb, size=[8.2 MiB/5.4 GiB]
2024-05-15 07:02:24 INF read_count=[16424770], read_ops=[45679.81], write_count=[16424769], write_ops=[45679.81], src-0, syncing rdb, size=[87 MiB/492 MiB]
2024-05-15 07:02:29 INF read_count=[16685182], read_ops=[52144.54], write_count=[16685181], write_ops=[52144.54], src-1, syncing rdb, size=[8.0 MiB/5.4 GiB]
2024-05-15 07:02:34 INF read_count=[16942793], read_ops=[52668.01], write_count=[16942792], write_ops=[52668.01], src-2, syncing rdb, size=[9.6 MiB/5.4 GiB]
2024-05-15 07:02:39 INF read_count=[17204845], read_ops=[52398.31], write_count=[17204844], write_ops=[52398.31], src-0, syncing rdb, size=[89 MiB/492 MiB]
2024-05-15 07:02:44 INF read_count=[17460081], read_ops=[49617.08], write_count=[17460080], write_ops=[49617.08], src-1, syncing rdb, size=[10 MiB/5.4 GiB]
2024-05-15 07:02:49 INF read_count=[17710277], read_ops=[49990.90], write_count=[17710276], write_ops=[49990.90], src-2, syncing rdb, size=[9.6 MiB/5.4 GiB]
2024-05-15 07:02:54 INF read_count=[17967899], read_ops=[52754.64], write_count=[17967899], write_ops=[52754.64], src-0, syncing rdb, size=[91 MiB/492 MiB]
2024-05-15 07:02:59 INF read_count=[18228283], read_ops=[52953.06], write_count=[18228282], write_ops=[52953.06], src-1, syncing rdb, size=[12 MiB/5.4 GiB]
2024-05-15 07:03:04 INF read_count=[18483977], read_ops=[50480.75], write_count=[18483976], write_ops=[50479.75], src-2, syncing rdb, size=[9.6 MiB/5.4 GiB]
2024-05-15 07:03:09 INF read_count=[18744977], read_ops=[52736.05], write_count=[18744976], write_ops=[52736.05], src-0, syncing rdb, size=[92 MiB/492 MiB]
2024-05-15 07:03:14 INF read_count=[19007607], read_ops=[52773.57], write_count=[19007607], write_ops=[52773.57], src-1, syncing rdb, size=[14 MiB/5.4 GiB]
2024-05-15 07:03:19 INF read_count=[19262889], read_ops=[50460.73], write_count=[19262888], write_ops=[50460.73], src-2, syncing rdb, size=[9.6 MiB/5.4 GiB]
2024-05-15 07:03:24 INF read_count=[19521513], read_ops=[52715.50], write_count=[19521512], write_ops=[52715.50], src-0, syncing rdb, size=[94 MiB/492 MiB]
2024-05-15 07:03:29 INF read_count=[19780249], read_ops=[52266.16], write_count=[19780248], write_ops=[52266.16], src-1, syncing rdb, size=[16 MiB/5.4 GiB]
2024-05-15 07:03:34 INF read_count=[20033352], read_ops=[49237.11], write_count=[20033351], write_ops=[49237.11], src-2, syncing rdb, size=[15 MiB/5.4 GiB]
2024-05-15 07:03:39 INF read_count=[20289052], read_ops=[52331.53], write_count=[20289051], write_ops=[52331.53], src-0, syncing rdb, size=[96 MiB/492 MiB]
2024-05-15 07:03:44 INF read_count=[20539697], read_ops=[47760.46], write_count=[20539696], write_ops=[47759.46], src-1, syncing rdb, size=[18 MiB/5.4 GiB]
2024-05-15 07:03:49 INF read_count=[20798161], read_ops=[50239.35], write_count=[20798160], write_ops=[50240.35], src-2, syncing rdb, size=[16 MiB/5.4 GiB]
2024-05-15 07:03:54 INF read_count=[21059095], read_ops=[53560.81], write_count=[21059094], write_ops=[53560.81], src-0, syncing rdb, size=[97 MiB/492 MiB]
2024-05-15 07:03:59 INF read_count=[21318595], read_ops=[52702.82], write_count=[21318594], write_ops=[52702.82], src-1, syncing rdb, size=[20 MiB/5.4 GiB]
2024-05-15 07:04:04 INF read_count=[21576099], read_ops=[52066.23], write_count=[21576098], write_ops=[52066.23], src-2, syncing rdb, size=[18 MiB/5.4 GiB]
2024-05-15 07:04:09 INF read_count=[21839243], read_ops=[53722.72], write_count=[21839242], write_ops=[53721.72], src-0, syncing rdb, size=[99 MiB/492 MiB]
2024-05-15 07:04:14 INF read_count=[22100307], read_ops=[53570.54], write_count=[22100307], write_ops=[53570.54], src-1, syncing rdb, size=[21 MiB/5.4 GiB]
2024-05-15 07:04:19 INF read_count=[22356541], read_ops=[50516.51], write_count=[22356540], write_ops=[50516.51], src-2, syncing rdb, size=[20 MiB/5.4 GiB]
2024-05-15 07:04:24 INF read_count=[22614206], read_ops=[51587.97], write_count=[22614205], write_ops=[51587.97], src-0, syncing rdb, size=[101 MiB/492 MiB]
2024-05-15 07:04:29 INF read_count=[22869818], read_ops=[52034.68], write_count=[22869817], write_ops=[52035.68], src-1, syncing rdb, size=[21 MiB/5.4 GiB]
2024-05-15 07:04:34 INF read_count=[23127535], read_ops=[51108.46], write_count=[23127534], write_ops=[51108.46], src-2, syncing rdb, size=[23 MiB/5.4 GiB]
2024-05-15 07:04:39 INF read_count=[23388860], read_ops=[51680.77], write_count=[23388859], write_ops=[51680.77], src-0, syncing rdb, size=[102 MiB/492 MiB]
2024-05-15 07:04:44 INF read_count=[23639642], read_ops=[49563.42], write_count=[23639641], write_ops=[49563.42], src-1, syncing rdb, size=[21 MiB/5.4 GiB]
2024-05-15 07:04:49 INF read_count=[23902105], read_ops=[50821.72], write_count=[23902104], write_ops=[50821.72], src-2, syncing rdb, size=[24 MiB/5.4 GiB]
2024-05-15 07:04:54 INF read_count=[24164546], read_ops=[52444.32], write_count=[24164545], write_ops=[52444.32], src-0, syncing rdb, size=[104 MiB/492 MiB]
2024-05-15 07:04:59 INF read_count=[24428660], read_ops=[52439.94], write_count=[24428660], write_ops=[52439.94], src-1, syncing rdb, size=[21 MiB/5.4 GiB]
2024-05-15 07:05:04 INF read_count=[24682924], read_ops=[49252.67], write_count=[24682923], write_ops=[49251.67], src-2, syncing rdb, size=[26 MiB/5.4 GiB]
2024-05-15 07:05:09 INF read_count=[24945077], read_ops=[53259.09], write_count=[24945076], write_ops=[53259.09], src-0, syncing rdb, size=[106 MiB/492 MiB]
2024-05-15 07:05:14 INF read_count=[25201637], read_ops=[53075.37], write_count=[25201636], write_ops=[53075.37], src-1, syncing rdb, size=[21 MiB/5.4 GiB]
2024-05-15 07:05:19 INF read_count=[25459819], read_ops=[51775.62], write_count=[25459818], write_ops=[51775.62], src-2, syncing rdb, size=[33 MiB/5.4 GiB]
2024-05-15 07:05:24 INF read_count=[25719125], read_ops=[53105.86], write_count=[25719124], write_ops=[53105.86], src-0, syncing rdb, size=[107 MiB/492 MiB]
2024-05-15 07:05:29 INF read_count=[25981309], read_ops=[53377.39], write_count=[25981308], write_ops=[53377.39], src-1, syncing rdb, size=[21 MiB/5.4 GiB]
2024-05-15 07:05:34 INF read_count=[26237376], read_ops=[51017.06], write_count=[26237375], write_ops=[51017.06], src-2, syncing rdb, size=[36 MiB/5.4 GiB]
2024-05-15 07:05:39 INF read_count=[26499263], read_ops=[52493.50], write_count=[26499262], write_ops=[52493.50], src-0, syncing rdb, size=[109 MiB/492 MiB]
2024-05-15 07:05:44 INF read_count=[26749924], read_ops=[49024.38], write_count=[26749923], write_ops=[49024.38], src-1, syncing rdb, size=[21 MiB/5.4 GiB]
2024-05-15 07:05:49 INF read_count=[27010779], read_ops=[51137.40], write_count=[27010779], write_ops=[51138.40], src-2, syncing rdb, size=[38 MiB/5.4 GiB]
2024-05-15 07:05:54 INF read_count=[27268557], read_ops=[49413.08], write_count=[27268556], write_ops=[49412.08], src-0, syncing rdb, size=[110 MiB/492 MiB]
2024-05-15 07:05:59 INF read_count=[27527170], read_ops=[52292.19], write_count=[27527169], write_ops=[52292.19], src-1, syncing rdb, size=[28 MiB/5.4 GiB]
2024-05-15 07:06:04 INF read_count=[27781327], read_ops=[51743.01], write_count=[27781326], write_ops=[51743.01], src-2, syncing rdb, size=[40 MiB/5.4 GiB]
2024-05-15 07:06:09 INF read_count=[28024480], read_ops=[52892.96], write_count=[28024480], write_ops=[52892.96], src-0, syncing rdb, size=[114 MiB/492 MiB]
2024-05-15 07:06:14 INF read_count=[28280875], read_ops=[49881.98], write_count=[28280874], write_ops=[49881.98], src-1, syncing rdb, size=[30 MiB/5.4 GiB]
2024-05-15 07:06:19 INF read_count=[28532759], read_ops=[49202.59], write_count=[28532758], write_ops=[49202.59], src-2, syncing rdb, size=[41 MiB/5.4 GiB]
2024-05-15 07:06:24 INF read_count=[28790741], read_ops=[52472.80], write_count=[28790740], write_ops=[52473.80], src-0, syncing rdb, size=[115 MiB/492 MiB]
2024-05-15 07:06:29 INF read_count=[29052131], read_ops=[53442.31], write_count=[29052130], write_ops=[53442.31], src-1, syncing rdb, size=[32 MiB/5.4 GiB]
2024-05-15 07:06:34 INF read_count=[29308580], read_ops=[49337.89], write_count=[29308580], write_ops=[49336.89], src-2, syncing rdb, size=[43 MiB/5.4 GiB]
2024-05-15 07:06:39 INF read_count=[29571116], read_ops=[52468.54], write_count=[29571115], write_ops=[52468.54], src-0, syncing rdb, size=[117 MiB/492 MiB]
2024-05-15 07:06:44 INF read_count=[29825588], read_ops=[49366.67], write_count=[29825587], write_ops=[49365.67], src-1, syncing rdb, size=[34 MiB/5.4 GiB]
2024-05-15 07:06:49 INF read_count=[30084683], read_ops=[50245.69], write_count=[30084683], write_ops=[50245.69], src-2, syncing rdb, size=[45 MiB/5.4 GiB]
2024-05-15 07:06:54 INF read_count=[30341291], read_ops=[52275.17], write_count=[30341290], write_ops=[52275.17], src-0, syncing rdb, size=[118 MiB/492 MiB]
2024-05-15 07:06:59 INF read_count=[30599904], read_ops=[50970.80], write_count=[30599903], write_ops=[50970.80], src-1, syncing rdb, size=[36 MiB/5.4 GiB]
2024-05-15 07:07:04 INF read_count=[30851763], read_ops=[45750.69], write_count=[30851762], write_ops=[45750.69], src-2, syncing rdb, size=[46 MiB/5.4 GiB]
2024-05-15 07:07:09 INF read_count=[31108368], read_ops=[52767.23], write_count=[31108367], write_ops=[52767.23], src-0, syncing rdb, size=[120 MiB/492 MiB]
2024-05-15 07:07:14 INF read_count=[31371390], read_ops=[52874.37], write_count=[31371389], write_ops=[52874.37], src-1, syncing rdb, size=[38 MiB/5.4 GiB]
2024-05-15 07:07:19 INF read_count=[31630350], read_ops=[52103.91], write_count=[31630349], write_ops=[52103.91], src-2, syncing rdb, size=[48 MiB/5.4 GiB]
2024-05-15 07:07:24 INF read_count=[31892047], read_ops=[52786.45], write_count=[31892047], write_ops=[52785.45], src-0, syncing rdb, size=[122 MiB/492 MiB]
2024-05-15 07:07:29 INF read_count=[32153894], read_ops=[53995.61], write_count=[32153893], write_ops=[53995.61], src-1, syncing rdb, size=[40 MiB/5.4 GiB]
2024-05-15 07:07:34 INF read_count=[32408574], read_ops=[49068.64], write_count=[32408573], write_ops=[49068.64], src-2, syncing rdb, size=[50 MiB/5.4 GiB]
2024-05-15 07:07:39 INF read_count=[32665123], read_ops=[51562.93], write_count=[32665122], write_ops=[51562.93], src-0, syncing rdb, size=[123 MiB/492 MiB]
2024-05-15 07:07:44 INF read_count=[32914499], read_ops=[47948.70], write_count=[32914498], write_ops=[47948.70], src-1, syncing rdb, size=[42 MiB/5.4 GiB]
2024-05-15 07:07:49 INF read_count=[33166647], read_ops=[49715.57], write_count=[33166646], write_ops=[49714.57], src-2, syncing rdb, size=[51 MiB/5.4 GiB]
2024-05-15 07:07:54 INF read_count=[33421498], read_ops=[51745.07], write_count=[33421497], write_ops=[51745.07], src-0, syncing rdb, size=[125 MiB/492 MiB]
2024-05-15 07:07:59 INF read_count=[33682866], read_ops=[54258.69], write_count=[33682866], write_ops=[54258.69], src-1, syncing rdb, size=[44 MiB/5.4 GiB]
2024-05-15 07:08:04 INF read_count=[33937440], read_ops=[49178.66], write_count=[33937439], write_ops=[49178.66], src-2, syncing rdb, size=[53 MiB/5.4 GiB]
2024-05-15 07:08:09 INF read_count=[34194516], read_ops=[52136.34], write_count=[34194516], write_ops=[52137.34], src-0, syncing rdb, size=[127 MiB/492 MiB]
2024-05-15 07:08:14 INF read_count=[34460156], read_ops=[52791.41], write_count=[34460155], write_ops=[52792.41], src-1, syncing rdb, size=[46 MiB/5.4 GiB]
2024-05-15 07:08:19 INF read_count=[34718813], read_ops=[52993.18], write_count=[34718812], write_ops=[52993.18], src-2, syncing rdb, size=[54 MiB/5.4 GiB]
2024-05-15 07:08:24 INF read_count=[34981932], read_ops=[53641.50], write_count=[34981931], write_ops=[53641.50], src-0, syncing rdb, size=[128 MiB/492 MiB]
2024-05-15 07:08:29 INF read_count=[35242936], read_ops=[52580.55], write_count=[35242936], write_ops=[52580.55], src-1, syncing rdb, size=[48 MiB/5.4 GiB]
2024-05-15 07:08:34 INF read_count=[35493305], read_ops=[48687.81], write_count=[35493304], write_ops=[48688.81], src-2, syncing rdb, size=[54 MiB/5.4 GiB]
2024-05-15 07:08:39 INF read_count=[35746383], read_ops=[52673.93], write_count=[35746382], write_ops=[52672.93], src-0, syncing rdb, size=[130 MiB/492 MiB]
2024-05-15 07:08:44 INF read_count=[35997876], read_ops=[50923.84], write_count=[35997875], write_ops=[50923.84], src-1, syncing rdb, size=[49 MiB/5.4 GiB]
2024-05-15 07:08:49 INF read_count=[36251779], read_ops=[50212.45], write_count=[36251779], write_ops=[50212.45], src-2, syncing rdb, size=[54 MiB/5.4 GiB]
2024-05-15 07:08:54 INF read_count=[36508020], read_ops=[51077.50], write_count=[36508019], write_ops=[51077.50], src-0, syncing rdb, size=[132 MiB/492 MiB]
2024-05-15 07:08:59 INF read_count=[36766406], read_ops=[51586.14], write_count=[36766406], write_ops=[51585.14], src-1, syncing rdb, size=[51 MiB/5.4 GiB]
2024-05-15 07:09:04 INF read_count=[37016192], read_ops=[50118.95], write_count=[37016191], write_ops=[50119.95], src-2, syncing rdb, size=[58 MiB/5.4 GiB]
2024-05-15 07:09:09 INF read_count=[37268578], read_ops=[51260.73], write_count=[37268577], write_ops=[51260.73], src-0, syncing rdb, size=[133 MiB/492 MiB]
2024-05-15 07:09:14 INF read_count=[37531302], read_ops=[53160.47], write_count=[37531301], write_ops=[53160.47], src-1, syncing rdb, size=[53 MiB/5.4 GiB]
2024-05-15 07:09:19 INF read_count=[37783084], read_ops=[49934.90], write_count=[37783084], write_ops=[49933.90], src-2, syncing rdb, size=[65 MiB/5.4 GiB]
2024-05-15 07:09:24 INF read_count=[38042391], read_ops=[50797.03], write_count=[38042391], write_ops=[50796.03], src-0, syncing rdb, size=[135 MiB/492 MiB]
2024-05-15 07:09:29 INF read_count=[38298443], read_ops=[52036.29], write_count=[38298443], write_ops=[52036.29], src-1, syncing rdb, size=[54 MiB/5.4 GiB]
2024-05-15 07:09:34 INF read_count=[38553375], read_ops=[51201.50], write_count=[38553374], write_ops=[51201.50], src-2, syncing rdb, size=[67 MiB/5.4 GiB]
2024-05-15 07:09:39 INF read_count=[38809552], read_ops=[50724.17], write_count=[38809552], write_ops=[50724.17], src-0, syncing rdb, size=[137 MiB/492 MiB]
2024-05-15 07:09:44 INF read_count=[39057980], read_ops=[49294.70], write_count=[39057980], write_ops=[49295.70], src-1, syncing rdb, size=[56 MiB/5.4 GiB]
2024-05-15 07:09:49 INF read_count=[39314114], read_ops=[50657.34], write_count=[39314113], write_ops=[50657.34], src-2, syncing rdb, size=[69 MiB/5.4 GiB]
2024-05-15 07:09:54 INF read_count=[39569479], read_ops=[49372.68], write_count=[39569478], write_ops=[49372.68], src-0, syncing rdb, size=[138 MiB/492 MiB]
2024-05-15 07:09:59 INF read_count=[39828890], read_ops=[51207.80], write_count=[39828889], write_ops=[51208.80], src-1, syncing rdb, size=[58 MiB/5.4 GiB]
2024-05-15 07:10:04 INF read_count=[40081031], read_ops=[50801.48], write_count=[40081031], write_ops=[50800.48], src-2, syncing rdb, size=[71 MiB/5.4 GiB]
2024-05-15 07:10:09 INF read_count=[40326906], read_ops=[48983.84], write_count=[40326906], write_ops=[48983.84], src-0, syncing rdb, size=[140 MiB/492 MiB]
2024-05-15 07:10:14 INF read_count=[40583448], read_ops=[50593.68], write_count=[40583447], write_ops=[50593.68], src-1, syncing rdb, size=[60 MiB/5.4 GiB]
2024-05-15 07:10:19 INF read_count=[40839249], read_ops=[47854.03], write_count=[40839248], write_ops=[47855.03], src-2, syncing rdb, size=[73 MiB/5.4 GiB]
2024-05-15 07:10:24 INF read_count=[41097391], read_ops=[52315.08], write_count=[41097390], write_ops=[52315.08], src-0, syncing rdb, size=[142 MiB/492 MiB]
2024-05-15 07:10:29 INF read_count=[41357067], read_ops=[52106.04], write_count=[41357066], write_ops=[52107.04], src-1, syncing rdb, size=[62 MiB/5.4 GiB]
2024-05-15 07:10:34 INF read_count=[41609520], read_ops=[47325.09], write_count=[41609519], write_ops=[47324.09], src-2, syncing rdb, size=[75 MiB/5.4 GiB]
2024-05-15 07:10:39 INF read_count=[41858437], read_ops=[50560.57], write_count=[41858436], write_ops=[50560.57], src-0, syncing rdb, size=[143 MiB/492 MiB]
2024-05-15 07:10:44 INF read_count=[42109121], read_ops=[48800.77], write_count=[42109120], write_ops=[48801.77], src-1, syncing rdb, size=[64 MiB/5.4 GiB]
2024-05-15 07:10:49 INF read_count=[42362012], read_ops=[48200.20], write_count=[42362011], write_ops=[48200.20], src-2, syncing rdb, size=[77 MiB/5.4 GiB]
2024-05-15 07:10:54 INF read_count=[42615028], read_ops=[51671.85], write_count=[42615027], write_ops=[51670.85], src-0, syncing rdb, size=[145 MiB/492 MiB]
2024-05-15 07:10:59 INF read_count=[42870740], read_ops=[51850.35], write_count=[42870739], write_ops=[51850.35], src-1, syncing rdb, size=[66 MiB/5.4 GiB]
2024-05-15 07:11:04 INF read_count=[43126378], read_ops=[50690.87], write_count=[43126377], write_ops=[50690.87], src-2, syncing rdb, size=[79 MiB/5.4 GiB]
2024-05-15 07:11:09 INF read_count=[43387713], read_ops=[52168.15], write_count=[43387712], write_ops=[52168.15], src-0, syncing rdb, size=[146 MiB/492 MiB]
2024-05-15 07:11:14 INF read_count=[43645924], read_ops=[51450.49], write_count=[43645923], write_ops=[51450.49], src-1, syncing rdb, size=[67 MiB/5.4 GiB]
2024-05-15 07:11:19 INF read_count=[43904825], read_ops=[52454.65], write_count=[43904824], write_ops=[52454.65], src-2, syncing rdb, size=[80 MiB/5.4 GiB]
2024-05-15 07:11:24 INF read_count=[44153751], read_ops=[50459.40], write_count=[44153750], write_ops=[50459.40], src-0, syncing rdb, size=[148 MiB/492 MiB]
2024-05-15 07:11:29 INF read_count=[44421938], read_ops=[54786.57], write_count=[44421937], write_ops=[54785.57], src-1, syncing rdb, size=[67 MiB/5.4 GiB]
2024-05-15 07:11:34 INF read_count=[44685727], read_ops=[52085.57], write_count=[44685726], write_ops=[52085.57], src-2, syncing rdb, size=[82 MiB/5.4 GiB]
2024-05-15 07:11:39 INF read_count=[44948625], read_ops=[52032.06], write_count=[44948624], write_ops=[52031.06], src-0, syncing rdb, size=[149 MiB/492 MiB]
2024-05-15 07:11:44 INF read_count=[45203314], read_ops=[51795.69], write_count=[45203314], write_ops=[51794.69], src-1, syncing rdb, size=[67 MiB/5.4 GiB]
2024-05-15 07:11:49 INF read_count=[45469362], read_ops=[53209.14], write_count=[45469361], write_ops=[53210.14], src-2, syncing rdb, size=[84 MiB/5.4 GiB]
2024-05-15 07:11:54 INF read_count=[45736162], read_ops=[52843.07], write_count=[45736161], write_ops=[52843.07], src-0, syncing rdb, size=[151 MiB/492 MiB]
2024-05-15 07:11:59 INF read_count=[46004678], read_ops=[54048.87], write_count=[46004677], write_ops=[54048.87], src-1, syncing rdb, size=[67 MiB/5.4 GiB]
2024-05-15 07:12:04 INF read_count=[46265822], read_ops=[51151.59], write_count=[46265821], write_ops=[51151.59], src-2, syncing rdb, size=[86 MiB/5.4 GiB]
2024-05-15 07:12:09 INF read_count=[46533600], read_ops=[54016.67], write_count=[46533599], write_ops=[54017.67], src-0, syncing rdb, size=[153 MiB/492 MiB]
2024-05-15 07:12:14 INF read_count=[46797499], read_ops=[53226.84], write_count=[46797498], write_ops=[53225.84], src-1, syncing rdb, size=[67 MiB/5.4 GiB]
2024-05-15 07:12:19 INF read_count=[47055220], read_ops=[50802.31], write_count=[47055219], write_ops=[50803.31], src-2, syncing rdb, size=[89 MiB/5.4 GiB]
2024-05-15 07:12:24 INF read_count=[47322553], read_ops=[54955.67], write_count=[47322552], write_ops=[54956.67], src-0, syncing rdb, size=[154 MiB/492 MiB]
2024-05-15 07:12:29 INF read_count=[47590010], read_ops=[54010.15], write_count=[47590009], write_ops=[54010.15], src-1, syncing rdb, size=[67 MiB/5.4 GiB]
2024-05-15 07:12:34 INF read_count=[47852892], read_ops=[52383.05], write_count=[47852891], write_ops=[52383.05], src-2, syncing rdb, size=[91 MiB/5.4 GiB]
2024-05-15 07:12:39 INF read_count=[48119623], read_ops=[53094.47], write_count=[48119622], write_ops=[53094.47], src-0, syncing rdb, size=[156 MiB/492 MiB]
2024-05-15 07:12:44 INF read_count=[48377279], read_ops=[53721.96], write_count=[48377278], write_ops=[53721.96], src-1, syncing rdb, size=[67 MiB/5.4 GiB]
2024-05-15 07:12:49 INF read_count=[48637262], read_ops=[49875.25], write_count=[48637261], write_ops=[49874.25], src-2, syncing rdb, size=[92 MiB/5.4 GiB]
2024-05-15 07:12:54 INF read_count=[48902793], read_ops=[53527.54], write_count=[48902792], write_ops=[53528.54], src-0, syncing rdb, size=[158 MiB/492 MiB]
2024-05-15 07:12:59 INF read_count=[49168303], read_ops=[53397.66], write_count=[49168302], write_ops=[53397.66], src-1, syncing rdb, size=[75 MiB/5.4 GiB]
2024-05-15 07:13:04 INF read_count=[49432497], read_ops=[50506.17], write_count=[49432496], write_ops=[50506.17], src-2, syncing rdb, size=[93 MiB/5.4 GiB]
2024-05-15 07:13:09 INF read_count=[49696189], read_ops=[54002.72], write_count=[49696188], write_ops=[54002.72], src-0, syncing rdb, size=[160 MiB/492 MiB]
2024-05-15 07:13:14 INF read_count=[49962493], read_ops=[54349.86], write_count=[49962493], write_ops=[54349.86], src-1, syncing rdb, size=[78 MiB/5.4 GiB]
2024-05-15 07:13:19 INF read_count=[50224960], read_ops=[50650.85], write_count=[50224959], write_ops=[50650.85], src-2, syncing rdb, size=[96 MiB/5.4 GiB]
2024-05-15 07:13:24 INF read_count=[50491048], read_ops=[53372.25], write_count=[50491047], write_ops=[53372.25], src-0, syncing rdb, size=[161 MiB/492 MiB]
2024-05-15 07:13:29 INF read_count=[50759051], read_ops=[53960.81], write_count=[50759051], write_ops=[53959.81], src-1, syncing rdb, size=[78 MiB/5.4 GiB]
2024-05-15 07:13:34 INF read_count=[51023338], read_ops=[51610.73], write_count=[51023337], write_ops=[51610.73], src-2, syncing rdb, size=[98 MiB/5.4 GiB]
2024-05-15 07:13:39 INF read_count=[51290582], read_ops=[54275.55], write_count=[51290581], write_ops=[54274.55], src-0, syncing rdb, size=[163 MiB/492 MiB]
2024-05-15 07:13:44 INF read_count=[51543883], read_ops=[49627.03], write_count=[51543882], write_ops=[49628.03], src-1, syncing rdb, size=[78 MiB/5.4 GiB]
2024-05-15 07:13:49 INF read_count=[51805970], read_ops=[51732.96], write_count=[51805970], write_ops=[51732.96], src-2, syncing rdb, size=[100 MiB/5.4 GiB]
2024-05-15 07:13:54 INF read_count=[52068968], read_ops=[51080.60], write_count=[52068967], write_ops=[51080.60], src-0, syncing rdb, size=[165 MiB/492 MiB]
2024-05-15 07:13:59 INF read_count=[52335847], read_ops=[54490.77], write_count=[52335846], write_ops=[54490.77], src-1, syncing rdb, size=[78 MiB/5.4 GiB]
2024-05-15 07:14:04 INF read_count=[52593364], read_ops=[49469.58], write_count=[52593364], write_ops=[49470.58], src-2, syncing rdb, size=[102 MiB/5.4 GiB]

其他信息（Additional Information）

sync_reader 与 redis_writer 的配置如下，其他配置项采用默认参数

[sync_reader]
cluster = true
address = "redis-master.prod.svc:6379"
username = ""
password = "xxxxxx"
tls = false
sync_rdb = true
sync_aof = true
prefer_replica = true

[redis_writer]
cluster = true
address = "xxx.usw1.cache.amazonaws.com:6379"
username = ""
password = "xxx"
tls = true
off_reply = false
@cataglyphis cataglyphis added the type: question Further information is requested label May 15, 2024
@suxb201
Copy link
Member

suxb201 commented May 20, 2024

目前版本确实性能存在问题：#738
对于集群实例，建议开为每个分片单独启动 redis-shake 来提高性能。

@cataglyphis
Copy link
Author

目前版本确实性能存在问题：#738 对于集群实例，建议开为每个分片单独启动 redis-shake 来提高性能。

@suxb201 相当于起多个 redis shake 去分别同步集群模式的各个分片，比如 3 主 3 从，就起 3 个 redis shake，每个 redis shake 只同步一个从分片，并把所同步的分片认为是 standalone 模式？

@suxb201
Copy link
Member

suxb201 commented May 22, 2024

@cataglyphis 是的，可以这样设置。

@suxb201 suxb201 closed this as completed May 22, 2024
@suxb201 suxb201 mentioned this issue Oct 17, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type: question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cataglyphis @suxb201