系統(tǒng)拓撲

角色	?ip地址	hdfs	yarn
Master	10.0.0.201	NameNode	ResourceManager
slave	10.0.0.202	DataNode	NodeManager
slave	10.0.0.203	DataNode	NodeManager

?

前提條件

安裝jdk

關(guān)閉selinux

關(guān)閉防火墻或者配置端口

設(shè)置ssh免密登錄

在所有節(jié)點生RSA密鑰對, 不要設(shè)置私鑰密碼。

ssh-keygen

把所有節(jié)點的公鑰加入authorized_keys文件。

cat id_rsa.pub >> authorized_keys

cat id_rsa.pub.202 >> authorized_keys

cat id_rsa.pub.203 >> authorized_keys

?

scp authorized_keys root@10.0.0.202:/root/.ssh/authorized_keys

scp authorized_keys root@10.0.0.203:/root/.ssh/authorized_keys

?

ssh node1

ssh node2

ssh node3

設(shè)置/etc/hosts

scp /etc/hosts root@10.0.0.202:/etc/hosts

scp /etc/hosts root@10.0.0.203:/etc/hosts

設(shè)置環(huán)境變量

/etc/profile

?

配置Hadoop

解壓hadoop

并創(chuàng)建目錄

mkdir –p dfs/name

mkdir –p dfs/data

vim /etc/profile

exportHADOOP_HOME=/usr/local/src/Hadoop-2.7.0

?

vim ./etc/Hadoop/hadoop-env.sh

export JAVA_HOME=/usr/local/src/jdk1.7.0_60

core-site.xml

?

hdfs-site.xml

?

mapred-site.xml

yarn-site.xml

yarn.nodemanager.resource.memory-mb必須大于等于2048

圖中有誤，改為yarn.nodemanager.aux-services.mapreduce_shuffle.class

修改slaves

?

分發(fā)hadoop

scp -r ./hadoop-2.7.0 root@10.0.0.202:/usr/local/src

scp -r ./hadoop-2.7.0 root@10.0.0.203:/usr/local/src

?

運行hadoop

運行HDFS

bin/hdfs namenode –format (運行一次)

?

sbin/start-dfs.sh

運行yarn

sbin/yarn-daemon.sh? --config ./etc/hadoop start proxyserver

sbin/start-yarn.sh

運行mapreduce JobHistory Server

sbin/mr-jobhistory-daemon.sh –config ./etc/hadoopstart historyserver

查看進程

201

?

202

?

203

測試hadoop

hdfs namenode

http://10.0.0.201:50070

secondary namenode

yarn ResourceManager

http://10.0.0.201:8088

MapReduce JobHistory Server

http://10.0.0.201:19888

?

?

運行mapreduce worldcount

bin/hadoop jar./share/hadoop/mapreduce/hadoop-mapred????????????????? ???????????????????????????????????????????????????????????????????????????????????????????uce-examples-2.7.0.jarwordcount /example/input /example/output

停止hadoop

停止hdfs

sbin/stop-dfs.sh

停止yarn

sbin/stop-yarn.sh

?

停止WebAppProxy

sbin/yarn-daemon.sh –config ./etc/hadoopstop proxyserver

?

停止MapReduce JobHistory Server

sbin/mr-jobhistory-daemon.sh –config ./etc/hadoopstop historyserver

總結(jié)

以上是生活随笔為你收集整理的Hadoop 2.7集群环境搭建的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。