配置機架感知

core-site.xml

cat $HADOOP_HOME/etc/hadoop/core-site.xml

<property>

????? <name>net.topology.script.file.name</name>

????? <value>pathdir/RackAware.py</value>

</property>

?

RackAware.py

cat etc/hadoop/RackAware.py????????????

#!/usr/bin/python?

#-*-coding:UTF-8 -*-?

import sys?

?

rack = {"bs022.zx.nicx.cn":"rack1",?

??????? "bs035.zx.nicx.cn":"rack2",?

??????? "bs038.zx.nicx.cn":"rack2",?

??????? "bs042.zx.nicx.cn":"rack3",

??????? "192.168.1.22":"rack1",?

??????? "192.168.1.35":"rack2",?

??????? "192.168.1.38":"rack2",?

??????? "192.168.1.42":"rack3",

??????? }?

? ??

if __name__=="__main__":?

print "/" + rack.get(sys.argv[1])

?

機架感知驗證

bin/hdfs dfsadmin -printTopology

Rack: /rack1

? ?192.168.1.22:50010 (bs022.zx.nicx.cn)

Rack: /rack2

? ?192.168.1.35:50010 (bs035.zx.nicx.cn)

? ?192.168.1.38:50010 (bs038.zx.nicx.cn)

Rack: /rack3

? ?192.168.1.42:50010 (bs042.zx.nicx.cn)

?

機架感知配置完成

?

注：

???????? 在python腳本中配置的時候，需要兼容主機名和IP地址

???????? RackAware.py需要有可執行權限

???????? Namenode的日志中含有：

????????

2016-07-04 14:42:22,198 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack0/218.241.108.38:50010
……
2016-07-04 14:42:22,232 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack0/218.241.108.35:50010
……
2016-07-04 14:42:22,263 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack0/218.241.108.42:50010
……
2016-07-04 14:42:22,295 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack0/218.241.108.22:50010
……
2016-07-04 14:42:52,503 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 1 racks and 4 datanodes

總結

以上是生活随笔為你收集整理的Hadoop配置机架感知的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。