一、困難點

建立topic的時候，可以通過指定參數 –replication-factor 設置

二、解決辦法

實際上，我們可以考慮一種 “另類” 的辦法：可以利用 kafka-reassign-partitions.sh 命令對所有分區進行重新分布，在做分區重新分布的時候，通過增加每個分區的replica備份數量來達到目的。

本文將介紹如何利用 kafka-reassign-partitions.sh 命令增加topic的備份數量。

注意：以下命令使用到的topic名稱、zookeeper的ip和port，需要讀者替換成為實際集群的參數。

(假設kafka集群有4個broker，id分別為：1001,1002,1003,1004)

2.1、獲取當前topic的所有分區分布在broker的情況

[root@tbds bin]# ./kafka-topics.sh --zookeeper 172.16.32.13:2181 --topic ranger_audits --describe

Topic:ranger_audits PartitionCount:10 ReplicationFactor:1 Configs:

Topic: ranger_audits Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001

Topic: ranger_audits Partition: 1 Leader: 1002 Replicas: 1002 Isr: 1002

Topic: ranger_audits Partition: 2 Leader: 1001 Replicas: 1001 Isr: 1001

Topic: ranger_audits Partition: 3 Leader: 1002 Replicas: 1002 Isr: 1002

Topic: ranger_audits Partition: 4 Leader: 1001 Replicas: 1001 Isr: 1001

Topic: ranger_audits Partition: 5 Leader: 1002 Replicas: 1002 Isr: 1002

Topic: ranger_audits Partition: 6 Leader: 1001 Replicas: 1001 Isr: 1001

Topic: ranger_audits Partition: 7 Leader: 1002 Replicas: 1002 Isr: 1002

Topic: ranger_audits Partition: 8 Leader: 1001 Replicas: 1001 Isr: 1001

Topic: ranger_audits Partition: 9 Leader: 1002 Replicas: 1002 Isr: 1002

可以看出，ranger_audits 這個topic有10個分區，每個分區只有一個feplica備份，分布在1001和1002兩臺broker上面。

下面我們需要將ranger_audits 的每個分區數據都增加到2個replica備份，且分布到4個broker上面。

2.2、創建增加replica備份數量的配置文件

(注意：盡量保持topic的原有每個分區的主備份不變化。因此，配置文件的每個分區的第一個broker保持不變。)

[root@tbds bin]# vim ../config/increase-replication-factor.json

{"version":1,

"partitions":[

{"topic":"ranger_audits","partition":0,"replicas":[1001,1003]},

{"topic":"ranger_audits","partition":1,"replicas":[1002,1004]},

{"topic":"ranger_audits","partition":2,"replicas":[1001,1003]},

{"topic":"ranger_audits","partition":3,"replicas":[1002,1004]},

{"topic":"ranger_audits","partition":4,"replicas":[1001,1003]},

{"topic":"ranger_audits","partition":5,"replicas":[1002,1004]},

{"topic":"ranger_audits","partition":6,"replicas":[1001,1003]},

{"topic":"ranger_audits","partition":7,"replicas":[1002,1004]},

{"topic":"ranger_audits","partition":8,"replicas":[1001,1003]},

{"topic":"ranger_audits","partition":9,"replicas":[1002,1004]}

]}

上面的配置文件說明，我們將topic的每個分區都增加了一個replica，且保持每個分區原有的主備份所在broker不變化，將每個分區新增的replica備份數據放到到1003和1004兩個broker上面。

2.3、開始執行增加分區

[root@tbds bin]# ./kafka-reassign-partitions.sh -zookeeper 172.16.32.13:2181 --reassignment-json-file ../config/increase-replication-factor.json --execute

Current partition replica assignment

{"version":1,"partitions":[{"topic":"ranger_audits","partition":3,"replicas":[1002]},{"topic":"ranger_audits","partition":9,"replicas":[1002]},{"topic":"ranger_audits","partition":8,"replicas":[1001]},{"topic":"ranger_audits","partition":1,"replicas":[1002]},{"topic":"ranger_audits","partition":4,"replicas":[1001]},{"topic":"ranger_audits","partition":2,"replicas":[1001]},{"topic":"ranger_audits","partition":5,"replicas":[1002]},{"topic":"ranger_audits","partition":0,"replicas":[1001]},{"topic":"ranger_audits","partition":6,"replicas":[1001]},{"topic":"ranger_audits","partition":7,"replicas":[1002]}]}

Save this to use as the --reassignment-json-file option during rollback

Successfully started reassignment of partitions

{"version":1,"partitions":[{"topic":"ranger_audits","partition":0,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":8,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":5,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":2,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":9,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":1,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":3,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":4,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":7,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":6,"replicas":[1001,1003]}]}

2.4、查看執行進度

[root@tbds bin]# ./kafka-reassign-partitions.sh -zookeeper 172.16.32.13:2181 --reassignment-json-file ../config/increase-replication-factor.json --verify

Status of partition reassignment:

Reassignment of partition [ranger_audits,0] completed successfully

Reassignment of partition [ranger_audits,8] completed successfully

Reassignment of partition [ranger_audits,5] completed successfully

Reassignment of partition [ranger_audits,2] completed successfully

Reassignment of partition [ranger_audits,9] completed successfully

Reassignment of partition [ranger_audits,1] completed successfully

Reassignment of partition [ranger_audits,3] completed successfully

Reassignment of partition [ranger_audits,4] completed successfully

Reassignment of partition [ranger_audits,7] completed successfully

Reassignment of partition [ranger_audits,6] completed successfully

上面顯示增加分區操作成功

2.5、再次查看topic的情況

[root@tbds bin]# ./kafka-topics.sh --zookeeper 172.16.32.13:2181 --topic ranger_audits --describe

Topic:ranger_audits PartitionCount:10 ReplicationFactor:2 Configs:

Topic: ranger_audits Partition: 0 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003

Topic: ranger_audits Partition: 1 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004

Topic: ranger_audits Partition: 2 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003

Topic: ranger_audits Partition: 3 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004

Topic: ranger_audits Partition: 4 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003

Topic: ranger_audits Partition: 5 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004

Topic: ranger_audits Partition: 6 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003

Topic: ranger_audits Partition: 7 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004

Topic: ranger_audits Partition: 8 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003

Topic: ranger_audits Partition: 9 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004

從上面可以看出，備份數量增加成功

三、進一步思考

利用上述介紹的辦法，除了可以用來增加topic的備份數量之外，還能夠處理以下幾個場景：

1、對topic的所有分區數據進行整體遷移。怎么理解呢？假如集群有N個broker，后來新擴容M個broker。由于新擴容的broker磁盤都是空的，原有的broker磁盤占用都很滿。那么我們可以利用上述方法，將存儲在原有N個broker的某些topic整體搬遷到新擴容的M個broker，進而實現kafka集群的整體數據均衡。

具體使用方法就是：通過編寫2.2章節的配置文件，將topic的所有分區都配置到新的M個broker上面去，再執行excute，即可完成topic的所有分區數據整體遷移到新擴容的M個broker節點。

2、broker壞掉的情況。導致某些topic的某些分區的replica數量減少，可以利用kafka-reassign-partitions.sh增加replica；

3、kafka 某些broker磁盤占用很滿，某些磁盤占用又很少?？梢岳胟afka-reassign-partitions.sh遷移某些topic的分區數據到磁盤占用少的broker，實現數據均衡；

4、kafka集群擴容。需要把原來broker的topic數據整體遷移到新的broker，合理利用新擴容的broker，實現負載均衡。

注明：本文來自投稿，不代表服務器文檔網立場，如若轉載，請注明出處：https://www.fwqwd.com/12212.html

總結

以上是生活随笔為你收集整理的kafka增加服务器,kafka增加topic的备份数量的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。